Linear Algebra for Large Language Models
High Dimensional Latent Spaces
The Geometry of Meaning
When we talk about language models understanding words, what we're really talking about is geometry. Models like Llama 3 or GPT-4 don't see words; they see coordinates. Every word or token is mapped to a vector, which is just a point in a vast, multi-dimensional space. Think of it as a highly complex star chart where concepts are constellations.
But this space isn't just 2D or 3D. Modern LLMs operate in incredibly high-dimensional spaces. For instance, some Llama 3 models use an embedding space with 8,192 dimensions. Each word is a point defined by 8,192 separate coordinates. This is what we call a high-dimensional latent space.
The magic is in how these points are arranged. Words with similar meanings are placed close to each other. The vector for "king" is near the vector for "queen," and the vector for "walk" is near "stroll." The distance and direction between vectors encode semantic relationships. This is the geometric interpretation of meaning. The relationship between "king" and "queen" might be captured by a specific vector that represents the concept of gender. You could theoretically take the vector for "king," add this "female" vector, and land very close to the vector for "queen."
Semantic similarity becomes spatial proximity. Relationships become directions.
Directions as Concepts
This leads to a powerful idea known as the Linear Representation Hypothesis It suggests that abstract concepts like 'gender,' 'tense,' or even 'sentiment' aren't just fuzzy clouds of points. Instead, they are represented as specific directions, or 1D subspaces, within the high-dimensional embedding space. A single, consistent vector direction can represent the concept of 'plurality.' You could add this vector to "cat" to get closer to "cats," or to "dog" to get closer to "dogs."
This is incredibly efficient. Instead of learning about plurals for every single noun from scratch, the model learns a general 'plurality' direction. This allows it to generalize to new words it encounters. If it learns the vector for a new noun, it can apply the 'plurality' vector to infer its plural form.
The hypothesis suggests that many concepts we find intuitive can be unpacked into these linear components. This is what allows models to perform analogies like "king is to queen as man is to woman." Mathematically, this looks something like:
Packing Features with Superposition
A model with 8,192 dimensions seems to have a lot of space. But the number of concepts, features, and nuances in human language is virtually infinite. How does an LLM with a finite number of dimensions handle this? The answer lies in a phenomenon called where a single neuron or dimension can represent multiple, unrelated concepts simultaneously.
This works because of a peculiar property of high-dimensional spaces: vectors can be "almost orthogonal." In 3D space, you can only have three perfectly orthogonal (90-degree) vectors. But in 8,192-dimensional space, you can have a massive number of vectors that are very close to orthogonal. The model leverages this by assigning different concepts to these near-orthogonal directions within the same dimension. When the model needs to retrieve one concept, it can effectively ignore the others because their vector projections are close to zero.
This is a clever compression scheme. However, it comes with a trade-off. Because the vectors aren't perfectly orthogonal, there's always a small amount of interference. Activating one feature might faintly activate another, unrelated feature that shares the same dimension. This can lead to unexpected model behaviors or subtle inaccuracies. The model is constantly balancing the need to represent more features (capacity) against the errors created by their overlap (interference). This delicate geometric balancing act is at the heart of how LLMs manage to encode the world's knowledge into a finite set of vectors.
Time to see how well you've grasped these geometric ideas.
In the context of LLMs, what does it mean for words to be represented geometrically?
The Linear Representation Hypothesis suggests that abstract concepts like 'plurality' or 'gender' are represented as specific directions within the embedding space.
Understanding the geometry of LLMs is crucial for interpretability and for building more powerful and reliable models. It transforms our view of AI from abstract computation to a tangible, spatial arrangement of concepts.