Mixed Reality

As of 2020, consumers have already purchased more than 2 billion Augmented Reality (AR) capable smartphones. By 2022, that number will be 3.4 billion. This huge installed base presents news organizations with the best opportunity to explore spatial 3D media with their audiences.

In this project, Knight Lab students will make exploring mobile AR easier for small teams and individuals, by developing and publishing a series of guides on how to make AR features using open source and low cost software.

Students will learn by doing, developing skills along the whole production pipeline, from asset creation, to development and design, to deployment. We will produce at least three AR projects over the course of the quarter. Each project will contain at least one 3d model of a food item produced by photogrammetry, the process of constructing a 3d model from 2d photos.

We chose food as the theme because it taps into an existing culture of sharing tabletop photos on social media and requires a lot of detail in order to be captured in a compelling way. We’re also interested in the potential mobile AR has to bring food journalism into our audience’s homes in a much more literal way. Are you more likely to feel confident approaching a recipe if you see the ingredients laid out on your own countertop beforehand? Can a 3d model illustrate the difference between two kinds of tomatoes more effectively than a flat photograph? Would you try more unusual foods if you could inspect them close up in 3d on your phone first?

This is a great project for students with experience and interest in photography, 3D modeling, writing, graphic and interaction design. Students coming from a software background will have the opportunity to learn about graphics optimization, how AR technologies work, working with SDKs and deploying projects to the web. Tools we’ll explore will include Figma, Meshroom, Blender, Torch and the Spark AR Studio for creating Facebook and Instagram filters.

Automated Photogrammetry. As AR and VR increasingly becomes the focus as we move away from smartphones, media organizations will have to find ways to produce content native to those mediums. These same organizations have a wealth of stories that continue to have value over time. Flat video and photo will become less desirable as we continue to move into these new spaces. Previous projects in the lab have surfaced an opportunity to take existing video and process it using photogrammetry software to produce a 3D model.

An experimental design project that explores visualizing data in three dimensions for augmented reality. Visualizations that can be examined and inspected by physically getting closer or understood by walking around them, open up exciting possibilities for how we communicate complex ideas and data that reveals hidden truths.

An experimental design project, in which we students are analyzing methods for communicating data visually and exploring how those principles might be transferred and transformed in a 360 environment.

An experimental design project that explores an emerging concept of information spaces. This concept is behind Microsoft’s pivot to create future Windows operating systems that exist in mixed reality headsets. Students will explore the concept of a news/information app that exists in AR and/or VR. For example, a political news feed might exist on a wall in your living room, and breaking news would appear on a coffee table, twitter reactions surround the coffee table on the floor. Students will also explore the same concept in a virtual environment.

3d technology gives us the opportunity to tell the story of an object from all angles. With 3d scans we can examine tiny butterfly wings, visit distant archaeological sites and get up close to objects too fragile to be examined in real life. While 3d gives the viewer the ability to see the whole story, it’s often difficult to know just what to look at and how to construct meaning from this wealth of information.

Augmented Reality allows creators to tell stories in the places where they actually happened. For the purpose of this project, we’ll be working with the concept of physical accessibility on campus. We will use the Unity game engine and a framework developed at Knight Lab so that we can quickly experiment and refine our ideas. We’ll spend some time early in the quarter looking at AR experiences from museums and tourist attractions as well as games like Ingress and Pokemon Go. From there, we’ll build an app and start testing it with users, refining ideas as we go along. We’ll consider the differences between gamified apps that include scavenger hunts or challenges and AR applications that lay out information in a more traditional way.

Though many in journalism are excited about VR, few are addressing real issues with making it attractive and interactive for their audience. This story team will explore the idea of making multiple three-dimensional VR photos around a scene and linking them together so that the user can navigate it. They’ll be exploring complex VR design challenges, such as how to move around space without disorienting the user and how to easily author interactive environments.

Advancements in neuroscience and immersive technologies offer mechanisms for engineering an entirely new mode of performance art one that engages audiences to unprecedented degree. Using the latest VR production techniques, students used motion capture and machine learning to teach a computer to improvise a performance, creating an engaging VR experience.

In this project, students will use modern approaches to making 3D images both with hardware and software processing.

Imagine that any wall, building floor or doorway could come alive and tell stories. Using projection mapping, many artists, advertisers and industries are already doing it. In this project, students will explore ways in which projection mapping is currently being used and adapt them for journalistic purposes. Students will build and prototype their adaptations.

This Knight Lab project explores the power of Augmented Reality (AR) for public storytelling. Participating students will collaborate with conceptual artist, Bradley McCallum, to create an innovative exhibition that honors the life of freelance war correspondent James Wright Foley.

In this project, students will film 360 video and explore the best ways to add on that additional layer; students will finish the quarter with two videos, and will document their findings to make their storytelling methods more accessible to others.