Our brains are incredible at perceiving depth. With two eyes, we effortlessly see the world in three dimensions. But how can a computer do the same thing with a series of flat, 2D photographs? This is the challenge solved by a technique called Structure from Motion, or SfM.

SfM is a clever process that reconstructs a 3D scene from a collection of 2D images taken from different viewpoints. It simultaneously figures out two things: the “structure” of the scene (where objects are in 3D space) and the “motion” of the camera (where each picture was taken from). This allows us to create detailed 3D models of everything from small objects to entire landscapes, just by walking around and taking pictures.

The secret behind SfM is a principle you experience every day: motion parallax. Think about looking out the window of a moving car. The nearby guardrail zips by in a blur, while the distant mountains seem to barely move at all. Your brain instinctively uses this difference in apparent speed to understand which objects are close and which are far away.

SfM algorithms do the exact same thing. By comparing two or more photos of the same scene, the software tracks how different points have shifted. A point that moves a lot between frames is inferred to be close to the camera. A point that moves only slightly is understood to be far away. By analyzing thousands of such points, the system can build a surprisingly accurate 3D map of the scene.

For motion parallax to work, the computer needs a reliable way to identify the same physical point across multiple photos. It does this by looking for unique “features” in the images. A feature is simply a distinctive area that is easy to recognize again, like the corner of a window, a speck of dirt on a rock, or the unique texture of a patch of wood.

The first step is feature detection, where the algorithm scans each image for thousands of these stable, high-contrast points. Next comes feature tracking or matching, which is the crucial step of finding correspondences. The algorithm compares the features in one image to the features in the next, figuring out which ones represent the same point in the real world.

Once these features are tracked across several images, the SfM process has what it needs. It uses the pixel shifts of each matched feature, powered by the logic of motion parallax, to compute the 3D coordinates of those points. The result is a sparse “point cloud,” which serves as the skeleton of the final 3D model.

Before we finish, let's test your understanding of these core concepts.

Structure from Motion is a powerful tool that turns simple photo collections into immersive 3D worlds, all by cleverly applying principles of how we see.