3D Modeling in the field

Capturing People for Photogrammetry With a Smartphone

Virtual reality has been called an internet-sized opportunity. But the Internet only attracted the billions of users it has today once every user was empowered to create, contribute and share content, whether they could code or not.

Similarly, making something today for VR generally requires coding skills. However, there may be an easy solution from the surveying industry: Photogrammetry.

Photogrammetry is any process that uses photos to make measurements. Initially, surveyors used it to create flat maps. Google used it to create its 3D-map Google Earth from high-resolution aerial photography. But specialized software has made the technology very easy to use, especially when each of us have a high-resolution smartphone camera in our pockets.

Screenshot of Valve's The Lab

However, one of the limitations of photogrammetry is that the end-product doesn’t move. You can move it, but it isn’t alive. So whatever you make needs to be captivating while motionless.

We weren’t the first to this idea, nor to find an ideal use. To demonstrate the HTC Vive and welcome new users to VR, Valve used photogrammetry to create a breathtaking mountain vista in “The Lab.”

So at our lab, we set-out to understand how journalists could use photogrammetry for improved storytelling on the web and VR.

Ideation

One of the first things to acknowledge is the “if it’s not broke, don’t fix it” dilemma of new technology.

Not every story actually benefits with 3D models made using photogrammetry, so to decide when photogrammetry may be valuable we developed three questions:

  1. Will exploration be a key aspect of the story?
  2. Will the model have a visceral, visual impact when it is motionless?
  3. Will you have the control to get your subject to remain still long enough to take all of the photos?

Locations could easily pass all three questions, but, as practitioners, we knew most journalism is people-driven. So we found ourselves with a new question:

When are people visually explorable, viscerally impressive, and incredibly still, without coercion?

A portrait.

But why should a portrait be 3D?

We didn’t immediately have an answer to this question—but a use-case solution fell into our laps when we realized that an international cosplay competition would be in town in 2 weeks.

The contestants would have rich stories of why they put hundreds of hours of labor into costumes with stunning details from head-to-toe, front-and-back.

Capture Technique

Over a short period, we recognized three things we needed to understand and practice as we planned the C2E2 story.

Can we optimize our technique so that a cosplayer only needs to stand motionless for a few seconds instead of minutes?

Photogrammetry was designed to sync still photos, however, taking hundreds of individual still photos takes a few minutes if you’re moving quickly. And asking someone to stay perfectly motionless for minutes when they had other plans, and aren’t being paid, is completely unreasonable.

But if you’re shooting video, you can create 300 still images in just 10 seconds. Asking someone if they’re willing to stand still for 10-15 seconds is a much easier sell than 2-3 minutes.

But that means that you need to know how to get every angle you need in a quick and steady 15-second motion, with known ways to compensate for whatever pose the subject may take.

We tried a number of movements, but the one that worked the most consistently was the simplest: a 270-degree orbit of the subject, starting behind their left foot, crossing in front of them and ending behind their right foot, filming along a flat-plane.

How do we capture large poses?

Capturing large poses was a new challenge, but not a daunting one.

Before preparing for the C2E2 story, we had been mostly modeling objects and environments– and the few people we had rendered were closeups only.

We immediately began practicing capture with a wider arc to frame entire people, and then more complicatedly, people who were posing (as we expected would occur at C2E2).

As we began to step back to capture larger poses, we saw more and more strange blobs/artifacts in the models, and we quickly realized that these were related to the environment where we were capturing.

What kind of environments will yield us consistently acceptable results?

Initially we chose controlled environments that had no movement in sight, but that often left us in cramped hallways with little room to walk and lighting that cast hard shadows. We needed much larger spaces to walk around our subject if we wanted to capture from head-to-toe. And we needed much better lighting.

Over time we realized that we only need a controlled environment for the space between our camera and our subject. This meant we could position our subject in large, well-lit rooms, even when there were lots of people around, so long as we could keep the crowds of people from walking through our capture radius for 15 seconds.

Once we knew how to position ourselves and capture our posing subjects quickly, we were as ready as we could be. We went to C2E2 for the whole weekend and captured 41 different cosplayers. You can find our full collection of cosplay models here.

Software

Once we returned from the convention with a lot of cosplayer videos, we needed to turn them into 3D models.

The first step was to take each video, and extract the individual frames into separate .tiff format picture files. We used Adobe Premiere to do this.

We explored two different pieces of photogrammetry software: Reality Capture and PhotoScan.

After some comparative testing, we found that we were getting better models, in much less time, using Reality Capture, so that’s what we used to produce everything in our C2E2 article and seen above. (Read more about our testing in our photogrammetry tools review.)

Finally, we hosted all of our finished models on the website sketchfab so that we could easily embed them into our articles.

Presentation

Going in, We knew that this article wasn’t going to be an all-encompassing virtual narrative, so we made sure to quickly interview each person that we modeled. Because we interviewed each person, and kept ourselves open to what we heard, we could consider the quality of each model, as well as the quotes that we’d received, when determining how to structure our narrative arc.

We knew that this story was going to remain about the cosplayers and their passion. We decided to open with High School student Morgan because, while her finished model wasn’t the best, she had a great, relatable story for why she was there. And as a young amateur, she was a great foundation to arc from youth to amateur, competitor, champion, to professional who wants to inspire more youth.

We wanted to focus on show, don’t tell, so we kept the rest of the story minimal.

Current Limitations

One disappointing limitation was difficulty incorporating audio into the story. We wanted you to hear from each cosplayer when you saw them, but that functionality wasn’t readily available.

Additionally, lots of very interesting cosplayers simply didn’t reconstruct as good 3D models a variety of reasons.

You can find more of what works in photogrammetry today and what doesn’t in our full photogrammetry guides below.

Future Applications

Photogrammetry is going to become much easier, and over time as the VR industry grows, may become ubiquitous as how we share 3D moments.

Additionally, as we’ve explored using extracting frames of a video to capture still moments in time, we acknowledge that it may be possible to sync multiple videos to create moving photogrammetry models, although that technology is still probably a few years away. As technology progresses, in the future, photogrammetry could be one way that we create holograms of people in the future.

Regardless, photogrammetry can be, and should be, used to create compelling and adventurous stories, and to empathize with people and their environments.

Resources

About the authors

Henry Keyser

VR Journalism Student; focusing on media innovation, product management, TV Producing. Also a relapsing playwright.

Theodore Chryssos

Lifestyle journalist studying marketing communications, emerging-media geek, VR acolyte, amateur videographer, stationery enthusiast.

Jessica Buchleitner

Author of the 50 Women anthology series and Co-lead Editor/Producer Stories from Girls and Women of Mogadishu. United Nations delegate now obsessed with committing acts of journalism using virtual and augmented reality.

About the project

Photojournalism in 3D for VR and Beyond

In this project, students will use modern approaches to making 3D images both with hardware and software processing.

More results from Photojournalism in 3D for VR and Beyond

  • A New Look-Around Cosplay

    Photogrammetry 3D models from the floor of Chicago's C2E2 Comic Convention

    12-feet of dark feathers, a noir-detective's trench coat, blue tie and black boots was a sight to see on a 5-foot stature. Truly appreciating her costume required seeing it from all angles—exactly the reason we went to make 3D models of cosplayers at the Chicago Comic and Entertainment Expo. I really just wanted an excuse to make wings Morgan She spent a lot of late nights on this high school science fair project—but even when...

    Continue Reading

  • Photogrammetry for Journalism

    Documenting reality in 3D

    The ability to recreate an object in 3D to be viewed and handled in virtual reality seems like something you would read in a science fiction novel. In fact, as early as 1950, Ray Bradbury entertained the idea of a virtual reality nursery in "[The Veldt](https://en.wikipedia.org/wiki/The_Veldt_(short_story))" - his short story about two children who solve their anger with their parents by escaping to a simulated grassland. Photogrammetry is the science of deriving measurements from photographs....

    Continue Reading

  • Photogrammetry Tools Review

    The winner is clear, but expensive.

    Journalists know the maxim, "show, don't tell" creating engaging stories. We spent ten weeks exploring tools for creating 3D photogrammetry models which can be used to show story subjects in an entirely new way. Here we present a rundown of what we learned. For any journalist interested in creating 3D photogrammetry models there are two key steps: capture and processing. The basic gist is this: a journalist shoots about 100 images during the capture process,...

    Continue Reading

  • The 4 Key Components to Photogrammetry Capture

    Location, stillness, camera settings and camera movement

    Whether you are in a controlled or uncontrolled environment, there are four key components for successful photogrammetry capture: [location](#location), [stillness](#stillness), [camera settings](#camera-settings) and [camera movement](#camera-movement). Paying close attention to these will help reduce tiling, blurring and other issues when scanning photogrammetry objects. Location lighting, shadows, refraction and reflections During the post-processing of a photogrammetry model, most of the cleanup necessary is due to bad lighting conditions present while shooting in the field. Strange white or...

    Continue Reading