An autonomous agent, like a self-driving car or a game-playing AI, doesn't see the world in pixels. Raw visual data is overwhelming. A single 64x64 pixel color image contains over 12,000 individual data points. Processing this firehose of information for every decision is computationally expensive and inefficient. The agent needs a way to distill the essence of what it's seeing into a compact, meaningful summary.

This is the job of the 'Vision' module in a World Model, a process called spatial compression. Its goal is to take a high-dimensional observation, like a frame from a camera, and shrink it into a low-dimensional latent vector, often labeled z. This vector z is a compressed representation of the world's current state, capturing only the most important features for the agent to make decisions.

The tool for this job is the Variational Autoencoder (VAE). A VAE is a type of generative neural network with two main parts: an encoder and a decoder. The encoder takes the input image and maps it to a latent space, producing the vector z. The decoder then tries to reconstruct the original image using only the information in z. The magic happens in how the VAE learns to create this compressed representation.

The encoder's job is to create an information bottleneck. It forces all the data from the input image through a tiny opening, the latent space. To be successful, the encoder must learn to discard noise and irrelevant details while preserving the crucial features. For a car, this might mean keeping the position of other cars and pedestrians but ignoring the clouds in the sky.

The decoder's job is to reverse the process. It takes the compressed vector z and attempts to recreate the original image. The quality of the reconstruction is a direct measure of how well the latent vector z captured the important information. If the decoder can produce a faithful copy of the input, it means the latent space contains a good summary of the world.

Unlike a standard autoencoder which maps the input to a single point in the latent space, a VAE maps the input to a probability distribution. The encoder doesn't output one vector z; it outputs two vectors: a mean ($μ$) and a standard deviation ($σ$). The latent vector z is then sampled from this distribution. This is a crucial difference. It ensures the latent space is continuous and smooth, which prevents the model from simply memorizing specific points for each input. This property allows the agent to generalize to new, unseen situations.

Training a VAE involves a delicate balance between two competing goals, which are combined in its loss function. The total loss is the sum of two separate terms: the reconstruction loss and the Kullback-Leibler (KL) divergence.

The reconstruction loss pushes the model to encode as much information as possible into z to create perfect copies of the input. If this were the only objective, the VAE would learn to cheat by hiding information in obscure corners of the latent space.

The KL divergence pulls in the opposite direction. It forces the encoded distributions to be simple and well-behaved, similar to a standard bell curve. This encourages the model to use the latent space efficiently and prevents it from overfitting. The trade-off is key: we want a representation that is rich enough to reconstruct the scene but simple enough to be useful for planning and prediction.

This compressed vector z becomes the agent's perception of the world at a single moment. Instead of a mess of pixels, the agent gets a clean, low-dimensional summary. All subsequent parts of the World Model, such as predicting future states and deciding on actions, will operate not on the raw images, but on this elegant, compressed representation. The quality of this initial spatial compression step is fundamental to the agent's ability to understand and navigate its environment.

Check your understanding of how VAEs enable latent perception.

By mastering spatial compression with VAEs, we've given our agent a way to see. Next, we'll explore how it learns to dream about the future.