Transforming Stereographs into Point Clouds

Virtual and augmented reality, though often used for gaming purposes, may be turned to a more academic or journalistic purpose. Throughout the fall, we explored ways to make historical footage, such as videos and pictures, available in a more interactive space.

One method we developed allowed us to create 3D point clouds from stereographs, (two photos taken an ocular distance apart which, when viewed through a stereoscope, appeared 3D). In the simplest of terms, we cropped stereographs that were readily available, sent them through a program which rectified them, ran another program on them, which produced a depth map, then ran the results through one more program which created a point cloud. Below, we will explain this method in greater detail.

Software Requirements:

Some of the specific image processing tools we experimented with only run on Microsoft Windows. Outside of that software, you will need to install Python with Pillow on your computer. (We used Ubuntu with the Windows Subsystem for Linux to run Python.) You will also need a photo editing program such as Photoshop or Gimp that will allow you to crop photos as well as change the color mode of a photo to grayscale and a program that can read .rar files.

You may also want to download MeshLab to look at the point clouds quickly, though you can also upload the files to Sketchfab and view them there.

Method

Download the software to rectify images and generate depth maps.

We found both of these programs here, but we will only be using ER9B and DMAG5.

Download the two .rar files above from Dropbox onto your computer. Create a new file folder, we will call it Stereo Folder for clarity here, and extract all the files from both .rar files into this folder. Depending on your program, you may have to manually move the files into your new folder.

Edit the rectifying software for multiple uses.

Open er9b.bat with a text editor.

er9b_input.txt after renaming the file names

Return to Stereo Folder and find er9b.exe. Left click, open properties, and copy the location. Paste this line into er9b.bat, replacing the current path, (C:\msys64\home\Hugues\other\stuff9\er9b.exe) with your path, adding \er9b.exe at the end if necessary.
Save the file.
Return to er9b_test.
Open er9b_input.txt. The first four lines will be four file names. Change them to inputLeft.png, inputRight.png, outputLeft.png and outputRight.png respectively. (It should look exactly like the file on the right.)
Save the file.
Navigate back to Stereo Folder.

Edit the depth map software for multiple uses.

Open the folder called dmag5_test. Find the files dmag5.bat and dmag_input.txt.
Open dmag5.bat with a text editor. Return to Stereo Folder and find dmag5.exe. Left click, open properties, and copy the location. Paste this line into dmag5.bat, replacing the current path, (C:\msys64\home\Hugues\other\stuff5\dmag5.exe) with your path, adding \dmag5.exe at the end if necessary.
Save the file.

dmag5_input.txt after renaming the file names

Return to dmag5_test.
Open dmag5_input.txt. The first two lines will be file names. Change the file names to outputLeft.png and outputRight.png respectively. (These are the names of the adjusted photos you will be getting from running the er9b program.)
The next two lines are numbers. Ignore those for now.
There will be four more file names below on the next four lines. Change them to depthLeft.png, depthRight.png, occlusionLeft.pngand occlusionRight.png respectively. (It should look exactly like the file on the left.)
Save the file.
Navigate back to Stereo Folder.

You can use any file or folder names you prefer, but you will want them to be consistent across both input files and easy for you to distinguish from each other.

Create a copyable folder for multiple iterations.

In Stereo Folder create a new folder and name it. (We will call it Initial Folder for our purposes here).
Reopen the file folder er9b_test. Find er9b.bat and er9b_input.txt.
Copy the er9b_inpt.txt file and the er9b.bat file.
Navigate back to Stereo Folder.
Paste er9b_inpt.txt and er9b.bat into Initial Folder.
Reopen the file folder dmag5 test.
Find dmag5.bat and dmag_input.txt.
Copy the dmag5.bat file and dmag_input.txt file, and paste them into Initial Folder.

Add the depth map to point cloud program.

We originally found this program here, but we have since made modifications to it, so you can download our file here: Download File
Add this file to Initial Folder.

What we just set up is a folder that you can copy each time you use this process. This will make finding each iteration easier, and it will reduce the amount of work you have to do each time.

Now that we have all the software installed, we will explain the process of getting from a stereograph to a point cloud.

First, make a copy of Initial Folder in Stereo Folder and rename it. (For this example we will call it First Stereograph.)

Pick a stereograph.

Find a stereograph that appears to have good depth information that you would like to model.
Good resources for historical stereograph include the Library of Congress and the American Antiquarian Society.
For this example we used a stereograph from the American Antiquarian Society.
Save it into First Stereograph for easy access.

Seperate the two images:

Open the stereograph in a photo editing program, such as Photoshop.
First, crop the edges so that both pictures remain, but both have the same standard height.

Cropping the stereograph to an even height

Select the left photo and crop it. Save it as a .png into First Stereograph with the name inputLeft (inputLeft.png).

Select the right photo and crop it. Save it as a .png into First Stereograph with the name inputRight (inputRight.png).

Cropping the other side using the same size selection

First Stereograph after saving the cropped stereograph photos.

Rectify the images.

Open the file folder First Stereograph.
Double click on er9b.bat.
It will create multiple files including outputLeft.png, outputRight.png and disp_range.txt.

Left: First Stereograph after running er9b. Right: Terminal code after running er9b.

Create a depth map.

In First Stereograph find and open disp_range.txt and dmag5_input.txt.
There will be two numbers in disp_range.txt. They are the minimum and maximum disparity between the two images respectively.
Replace the number on the third line of dmag_input.txtwith the smaller number from disp_range.txt and the number on the fourth line with the larger number from disp_range.txt.
Save the file.
Double click on dmag5.bat.
You should get four new files: depthLeft.png, depthRight.png, occlusionLeft.png and occlusionRight.png.

Left: First Stereograph after running dmag5. Right: Terminal code after running dmag5.

Prepare to create a point cloud.

Open depthLeft.png in a photo editing program, such as Photoshop.
If it looks similar to the original picture with no obvious flaws, it will probably make a decent point cloud.
Change the file mode so it is no longer in an rgb format but in grayscale.
Save the file.

Create a point cloud.

Open command line or a command line shell such as Ubuntu and navigate to the file folder First Stereograph.
Run python pythonToPointCloud.py outputLeft.png depthLeft.png firstStereograph.ply
If this runs correctly, you will get a .ply file which you can view in Mesh Lab or Sketchfab.

Troubleshooting

Rectifying Images:

If the program does not create disp_range.txt, outputLeft.png and outputRight.png: it was unable to rectify the images. Re-separate the images, being careful to crop the images as precisely as possible, or try a different stereograph.

Creating a Depth Map:

If depthLeft.png and depthRight.png are not recognizable, check outputLeft.png and outputRight.png. They may not have rectified properly. If the photos are severely cropped or highly tilted, they will not make a good depth map. Try recropping the photos and running the programs again, or using a different stereograph.

Depth Map to Point Cloud:

If you get an error message reading:

 File "pythonToPointCloud.py", line 110, in <module> generate_pointcloud(args.rgb_file,args.depth_file,args.ply_file) File "pythonToPointCloud.py", line 79, in generate_pointcloud Z = depth.getpixel((u,v)) / scalingFactor TypeError unsupported operand type(s) for /: 'tuple' and 'float

check to make sure your depth map is an 8 bit grayscale png.