Taking a powerful text-only model like Llama 3.1 and enabling it to understand images is not as simple as just adding a camera. The core architectural challenge is teaching the model to see the world in a way that aligns with how it understands language. This is achieved by bolting on a specialized module called a vision encoder.

For Llama 3.2, the team chose a Vision Transformer, or ViT, to be its eyes. Specifically, it uses a variant of the vision encoder from CLIP, a model renowned for its ability to connect images and text with impressive accuracy. This ViT acts as a translator, converting the raw pixel data of an image into a sequence of numerical representations, or tokens, that the language model can comprehend. Think of it as describing a picture using a vocabulary the language model already knows.

Modern digital images are often high-resolution, containing millions of pixels. Feeding all this data into a model at once is computationally expensive and inefficient. Llama 3.2 solves this with a tile-based approach. It slices a high-resolution image into a grid of smaller, manageable patches or tiles.

The vision encoder processes each of these tiles independently to generate a corresponding visual token. This method allows the model to handle very large images without being overwhelmed, preserving fine details that would be lost if the image were simply shrunk down.

Once the image is converted into a series of visual tokens, the final step is to combine them with the text tokens from the user's prompt. The model doesn't use a complex cross-attention mechanism where text and image tokens constantly query each other. Instead, it employs a simpler and more efficient method: interleaving.

The visual tokens are simply inserted into the sequence of text tokens. For example, a prompt like "What is happening in this image? [image]" becomes a single, unified sequence for the model: [text_tokens] [visual_tokens]. This combined stream is then fed directly into the Llama 3.1 language decoder.

The model processes this interleaved sequence using the same [{] mechanism it uses for text. This means that when the model is predicting the next token, it can only "look back" at the tokens that came before it in the sequence. A text token can pay attention to all preceding image and text tokens, and an image token can pay attention to all preceding image tokens. This simple, elegant design allows the powerful, pre-trained language part of the model to apply its pattern-recognition skills to a new domain without major architectural changes.

Llama 3.2 Vision comes in two primary sizes, 11B and 90B, each designed with specific hardware capabilities in mind. While both models share the same fundamental architecture of a ViT paired with a Llama 3.1 text decoder, their internal dimensions and resource requirements are quite different.

The 11B model is a significant achievement, packaging powerful vision-language capabilities into a footprint that can run more efficiently. It's built upon the Llama 3.1 8B text model, with the additional parameters coming from the vision encoder and the components that connect it to the language decoder. The 90B model provides state-of-the-art performance by scaling up this same design, using the much larger Llama 3.1 70B model as its reasoning engine.

Let's review the key architectural concepts we've covered.

Understanding this architecture reveals how modern AI combines different types of information into a single, cohesive understanding.