Virtual reality has been called an internet-sized opportunity. But the Internet only attracted the billions of users it has today once every user was empowered to create, contribute and share content, whether they could code or not.
Similarly, making something today for VR generally requires coding skills. However, there may be an easy solution from the surveying industry: Photogrammetry.
Photogrammetry is any process that uses photos to make measurements. Initially, surveyors used it to create flat maps. Google used it to create its 3D-map Google Earth from high-resolution aerial photography. But specialized software has made the technology very easy to use, especially when each of us have a high-resolution smartphone camera in our pockets.
However, one of the limitations of photogrammetry is that the end-product doesn’t move. You can move it, but it isn’t alive. So whatever you make needs to be captivating while motionless.
We weren’t the first to this idea, nor to find an ideal use. To demonstrate the HTC Vive and welcome new users to VR, Valve used photogrammetry to create a breathtaking mountain vista in “The Lab.”
So at our lab, we set-out to understand how journalists could use photogrammetry for improved storytelling on the web and VR.
Ideation
One of the first things to acknowledge is the “if it’s not broke, don’t fix it” dilemma of new technology.
Not every story actually benefits with 3D models made using photogrammetry, so to decide when photogrammetry may be valuable we developed three questions:
- Will exploration be a key aspect of the story?
- Will the model have a visceral, visual impact when it is motionless?
- Will you have the control to get your subject to remain still long enough to take all of the photos?
Locations could easily pass all three questions, but, as practitioners, we knew most journalism is people-driven. So we found ourselves with a new question:
When are people visually explorable, viscerally impressive, and incredibly still, without coercion?
A portrait.
But why should a portrait be 3D?
We didn’t immediately have an answer to this question—but a use-case solution fell into our laps when we realized that an international cosplay competition would be in town in 2 weeks.
The contestants would have rich stories of why they put hundreds of hours of labor into costumes with stunning details from head-to-toe, front-and-back.
Capture Technique
Over a short period, we recognized three things we needed to understand and practice as we planned the C2E2 story.
Can we optimize our technique so that a cosplayer only needs to stand motionless for a few seconds instead of minutes?
Photogrammetry was designed to sync still photos, however, taking hundreds of individual still photos takes a few minutes if you’re moving quickly. And asking someone to stay perfectly motionless for minutes when they had other plans, and aren’t being paid, is completely unreasonable.
But if you’re shooting video, you can create 300 still images in just 10 seconds. Asking someone if they’re willing to stand still for 10-15 seconds is a much easier sell than 2-3 minutes.
But that means that you need to know how to get every angle you need in a quick and steady 15-second motion, with known ways to compensate for whatever pose the subject may take.
We tried a number of movements, but the one that worked the most consistently was the simplest: a 270-degree orbit of the subject, starting behind their left foot, crossing in front of them and ending behind their right foot, filming along a flat-plane.
How do we capture large poses?
Capturing large poses was a new challenge, but not a daunting one.
Before preparing for the C2E2 story, we had been mostly modeling objects and environments– and the few people we had rendered were closeups only.
We immediately began practicing capture with a wider arc to frame entire people, and then more complicatedly, people who were posing (as we expected would occur at C2E2).
As we began to step back to capture larger poses, we saw more and more strange blobs/artifacts in the models, and we quickly realized that these were related to the environment where we were capturing.
What kind of environments will yield us consistently acceptable results?
Initially we chose controlled environments that had no movement in sight, but that often left us in cramped hallways with little room to walk and lighting that cast hard shadows. We needed much larger spaces to walk around our subject if we wanted to capture from head-to-toe. And we needed much better lighting.
Over time we realized that we only need a controlled environment for the space between our camera and our subject. This meant we could position our subject in large, well-lit rooms, even when there were lots of people around, so long as we could keep the crowds of people from walking through our capture radius for 15 seconds.
Once we knew how to position ourselves and capture our posing subjects quickly, we were as ready as we could be. We went to C2E2 for the whole weekend and captured 41 different cosplayers. You can find our full collection of cosplay models here.
Software
Once we returned from the convention with a lot of cosplayer videos, we needed to turn them into 3D models.
The first step was to take each video, and extract the individual frames into separate .tiff format picture files. We used Adobe Premiere to do this.
We explored two different pieces of photogrammetry software: Reality Capture and PhotoScan.
After some comparative testing, we found that we were getting better models, in much less time, using Reality Capture, so that’s what we used to produce everything in our C2E2 article and seen above. (Read more about our testing in our photogrammetry tools review.)
Finally, we hosted all of our finished models on the website sketchfab so that we could easily embed them into our articles.
Presentation
Going in, We knew that this article wasn’t going to be an all-encompassing virtual narrative, so we made sure to quickly interview each person that we modeled. Because we interviewed each person, and kept ourselves open to what we heard, we could consider the quality of each model, as well as the quotes that we’d received, when determining how to structure our narrative arc.
We knew that this story was going to remain about the cosplayers and their passion. We decided to open with High School student Morgan because, while her finished model wasn’t the best, she had a great, relatable story for why she was there. And as a young amateur, she was a great foundation to arc from youth to amateur, competitor, champion, to professional who wants to inspire more youth.
We wanted to focus on show, don’t tell, so we kept the rest of the story minimal.
Current Limitations
One disappointing limitation was difficulty incorporating audio into the story. We wanted you to hear from each cosplayer when you saw them, but that functionality wasn’t readily available.
Additionally, lots of very interesting cosplayers simply didn’t reconstruct as good 3D models a variety of reasons.
You can find more of what works in photogrammetry today and what doesn’t in our full photogrammetry guides below.
Future Applications
Photogrammetry is going to become much easier, and over time as the VR industry grows, may become ubiquitous as how we share 3D moments.
Additionally, as we’ve explored using extracting frames of a video to capture still moments in time, we acknowledge that it may be possible to sync multiple videos to create moving photogrammetry models, although that technology is still probably a few years away. As technology progresses, in the future, photogrammetry could be one way that we create holograms of people in the future.
Regardless, photogrammetry can be, and should be, used to create compelling and adventurous stories, and to empathize with people and their environments.
Resources
- How to use Photogrammetry for Journalism - Our comprehensive guide for all lessons learned
- Our 3D Photo Technology Review - A breakdown of tools and software we tried
- The Art of 3D capture - A supplementary article detailing things to consider when modeling people and objects
- Valve Destinations tutorial - The first tutorial we used on photogrammetry
- Reality Capture tutorial by Alex Cheparev - Helped us learn Reality Capture
