Reality Capture: Getting great results with Photogrammetry

This week on Journey to VR, I interview Jon Tojek on the subject of Reality Capture. Jon shares his best tips and tricks for getting quality results when doing Photogrammetry.

For a roundup of the best practices he shares in the video, see

Reality Capture: A roundup of best practices for shooting great environments.

VIDEO TRANSCRIPT

Daryl: Jon, let's start with the basics here. What are you thinking about when someone comes to you and says, "Reality Capture, I want to get something done." What's the first thing that comes to your mind?

Jon: Well the first issue is, are we capturing a small object, or a human body, or do we want an environment? You know, small objects like a shoe, or a rock, or a tree, those are the common ones; those are easy. Humans are pretty tricky, but environments are pretty much the worst. There's just a lot of stuff.

Daryl: I know that when we were thinking about the environment [for my VR project], you had some advice for me. We thought indoor/outdoor; we thought about your apartment, which is behind you; the parking garage in your building; some exterior shots. What are you looking for in an environment that's going to give you success?

Jon: Once [you decide] you're going outdoors, you can find a space like an alley that has two clear walls, and maybe a third building, so you can have three walls. At least you need three walls, and then the sky. That works for an environment that's outside. I personally like outdoor environments. The thing is, you got to capture the sky, so along with your photogrammetry, you got to shoot a panoramic which would get all those things in the distance, and the sun, and the clouds, and the current lighting setup.

Then you have interiors: the best are ruins or things that are deserted, very detailed and decayed with tons of things on the walls and the floor. The more tracking markers from natural decay you have, the better photogrammetry result you'll get. Some things you don't want are modern, clean apartments... reflective surfaces are very poor. Photogrammetry on a car that's very reflective is difficult. My apartment did not work out that well because I have blank walls that are one color, just a beige, and if there's not a lot for the camera to pick up and track, it does make it difficult to get a good photogrammetry solution.

Daryl: Right, we ultimately ended up on deciding to use a staircase inside of your building, which had concrete walls, lots of texture in there, and pretty defined light sources, so it seems like our early tests are positive. It looks like the geometry is going to end up looking really sweet when it's done. So, once you've scouted your location, you know what you want to do, and what you're trying to accomplish, the next step is photographing the environment. Do you have any tips about what type of gear you're using? What's the process of doing the photographic work for the acquisition of the photos, and how many photos do you need? 

Jon: It's always more photos is better, and the most important thing is sharpness to the photos. You want complete clarity, no blurring of any sort. You do everything you can to make your camera shoot very sharp photos. The first thing and most obvious is stability, so you want a tripod. Right after that stability, you want a remote control so you're never touching the camera. You don't want any camera motion, and some people do this handheld. Sometimes you don't have a choice if you're outside. If you have to do it handheld, you brace yourself and you use the timer, two-second timer, so you can take a breath and hold the camera still. Those are all the physical things that help your camera get a good clean photo.

After that, there are all the settings of the camera, and of course the physical camera. I got a Sony Alpha 7, so a reasonable full-frame sensor, and I got a good lens, a Zeiss lens. It's a variable lens between 24 and 70 millimeters. That's not the most ideal. The best would be a fixed lens.

Daryl: Like a Prime?

Jon: Just a fixed 24 millimeter. Mine's variable, but I locked it to 24 millimeters. The reason I want that, I want the widest angle possible so I can capture as much of the space as possible. So a fairly wide lens. You could do a 35 millimeter fixed. That'd be fine. Then you have to do all your settings in the camera. You want a very low ISO to have the least possible point.

You want a fixed aperture and a fixed focus, so the aperture point of your lens where it is the sharpest. Obviously, you want a smaller aperture, a smaller gap, to have no blurriness in your photos, but your smallest gap is not the sharpest point of the lens. Every lens is a little bit different. A general rule, it's usually in the middle about one over 11, one over 12. Those are the normal points. You can even do a focus test yourself by taking a wedge of photos of one particular object with lots of detail and looking at them closely to find your best point, or you can look on the internet. There are a lot of websites that give you a chart for every single lens, and then they chart, "Where is the sharpest point?"

I personally did both on mine, and I found F over 11 to be the best aperture point for sharpness. Once I found that I locked that, and then you also want to lock your exposure time. Now, the exposure time is the one thing that can change over the whole shoot, but you want to keep it a consistent change. You want to change incrementally because sometimes you're shooting a light source, sometimes you're shooting a dark spot. You'll adjust your exposure time to get the most consistent exposure over the whole set of 500 photos.

Daryl: Is the number around 500 to start with? That's the ideal spot to get to, or...

Jon: It's a reasonable number. You need at least 200 to get a room that's just a square room with some stuff. 200 will get you a full scan. 400 will get it much better. You will get less bubbling, less bad data in your scan.

Daryl: When you're taking the photographs, what's your strategy for moving through the environment or moving through the room? How do you approach that?

Jon: Software wants a consistent horizontal motion, so if it's four walls and I'm trying to scan wall number one, I want to be perpendicular to that wall, and I'm going to move the camera one inch every shot so that it's getting it at a 90-degree angle. I'm gonna stand as far away from that wall as possible, and get the far wall, then I'm gonna walk around the edge of the room always pointing to the other far wall.

There is a distance limit here. A good limit is about three meters away. If the far wall is much farther away from that if you're in like a big warehouse, then you're gonna want to get the full distant wall, then you're gonna want to get closer to it because any more than three meters isn't going to give you detail. 

The other key thing is, it doesn't do you any good to turn the camera, especially around the noble point of the lens. That doesn't help the software at all to figure it out, so turning like this does no good at all. You want to do a move with a turn, a move with a turn, that kind of stuff.

Corners are always a little difficult. One other thing is that I mount the camera to be in portrait mode so it's taller, and I do that so I can get a lot of the floor and the roof in my shot, and then my side to side, I have full control over because I'm moving this thing.

Daryl: The next question I have is, you scan your room. What about making a light probe? What's your strategy for doing a light probe?

Jon: That one's kind of standard. You just need a Nodal Ninja or any other nodal head. I use a different camera for the light probe [and] I always got to use a color chart.

The color chart helps me get the two cameras to be closer and get a white balance for each one. For the panos, I use just an older Nikon with a fisheye lens, and then the important part is the Promote remote control. This allows you to do a bracket of nine photos of different exposures, and I do ten photos around, usually, nine exposures each, depending on if it's indoor or outdoor, how much light there is. The key way of shooting panos is you want to focus on that light source, and you want your darkest shot to be all black with a tiny bit of light source. If you're doing the sun, if you're doing outside in the sun, you want a black photo with a tiny dot as your sun, for the lowest exposure. 

Daryl: How do you get that? Are you putting an ND lens on the camera or something? How do you how do you clamp it down?

Jon: Yes, I do have an ND lens on there. That's especially important for the sun, and for this one, I did a polarizer lens to help get rid of reflections...or at least lower the reflections and specularity on anything that might be shiny.

Daryl: You're putting filters on your lenses, is it part of the strategy there?

Jon: Yes, unfortunately, that darkens the shots a little bit, but with your tripod and a little bit of exposure time, it doesn't matter. You can still capture decent light.

Daryl: Any other tips or tricks for when you're gonna go out, you're gonna try to attempt to do in your first photogrammetry or your first pano. What are the things that got you the first time?

Well, you do have to start small. Your first one, it's good to be in a controlled environment, you know? If you do it at home, where things are not gonna change quickly, it helps. A controlled environment in a studio or an office. Once you go outside and you start doing things like, "Oh, I found a deserted skate pool," or, "I found an old hotel with graffiti," or, "an alley with graffiti," once you're in public, it gets a little riskier, because people get involved, cars get involved. You have some equipment and people can grab your stuff. I've had people interfere with my shoots many times. A controlled environment is the first thing to do.

After that, take notes before you start, because there are a lot of details, and have a small checklist that says,'"I cleaned my lens. I'm using these settings. I'm for sure in RAW.' Sometimes you could shoot it in a JPEG format, and you just had a bad shoot, because all of your photos will have less color range, you'll have a little bit additional noise because of the compression. You just have to check all your camera settings very carefully before you start. There's a list, you know? Every software company has a list of these tips and tricks. They're all similar, but they're all a little bit different. I've compiled one myself, and you just have to go down that list. You gotta remember to do it in a consistent way every time.

Daryl: After you've got your photos, do you then bring them into Lightroom and balance them, color balance them, all to each other? What's the first step of the processing? Do you go right into the photogrammetry software, like Reality Capture, or what do you-

Jon: I use this color checker, this Macbeth color chart from X-Rite. Once I take a photo of this thing, this goes through a little piece of software from X-Rite and it pops out a camera profile, a color profile. Then I go all my photos into Lightroom, and I only do two things. I do white balance, and I give it the correct color profile. You can do a tiny bit of exposure adjustment, but you shouldn't really. You can't do sharpening. You don't need to do lens distortion reversal. You shouldn't do anything to modify the image. Just a little bit of color and white balance. Ideally, they're in RAW, and then you write them out as TIFF or another good format, PNGs, and you bring them into Reality Capture.

Daryl: Then process away, and you end up with some data that's gonna be usable, or not usable? Where does the data dump you?

Jon: This is the interesting thing: The process does take a long time. Say I got 500 photos. They're 6,000 x 4,000. They're pretty large. That can take 20 hours of processing, and once you get it out, you get 50 million polygons. Then you've got a texture. You can make one 8K texture. You can make 10 8K textures. You can do almost anything with the textures. It depends on how much process time you put into it. Then that model will have a great texture, that is very together. I mean, it can look photorealistic if you have really flat lighting and nothing happening. You bring that model into Maya and you just look at it, it can render out to look like a bunch of photographs, but if you look at it in Hardware Render, you can see it'll have bumpiness, it'll have some messiness on it. On the back parts, you do have to do some polygon rebuilding, and that's quite a bit of process, quite a bit of work, but you can rebuild the polygons, and then re-project the same textures on there to get cleaner results on the roughest parts of your model.

Which is exactly what we're gonna be doing for our piece, and we've got the raw data. I shared that to our viewers last week, and obviously, now sort of the cleanup process begins from here. That's where we'll end up kind of refining this over the Journey to VR and getting to that really tight asset that's gonna look great in the game engine, and in renders, too, actually. That's the end goal for us.

Well, that's really cool, Jon. Thanks so much for taking the time to go through these tips and tricks...You can always go to the Journey to VR blog and look at the written piece that we put together that will highlight a few of these best practices.