Large Language Model Architecture
Mathematical Embeddings
From Words to Vectors
Large language models don't understand words like humans do. For an LLM, language is math. Specifically, it's geometry in a very, very large space. Every piece of text the model processes, from a single word to an entire document, is converted into a numerical representation called an embedding. This embedding is a vector—a list of numbers—that pinpoints the text's location in a created by the model.
Think of it like a map. On a 2D map, Chicago might be at coordinate (41.88, -87.63). In an LLM, the word 'king' might be a vector with 768 coordinates, each a floating-point number. This isn't just a random list of numbers; it's a dense representation. Early NLP methods used sparse representations, like one-hot encoding, where a vocabulary of 50,000 words would mean each word was a vector of 50,000 zeros and a single one. This was inefficient and captured no semantic meaning.
| Representation | Characteristics | Use Case |
|---|---|---|
| Sparse (One-Hot) | Mostly zeros; dimension equals vocabulary size. | Early NLP, categorical data. |
| Dense (Embedding) | Continuous values; fixed, lower dimension. | Modern LLMs, semantic representation. |
Dense embeddings, by contrast, pack meaning into every number. These vectors are learned during the model's training process. Words with similar meanings are pushed closer together in this vast geometric space. 'Dog' and 'puppy' will have vectors that are near each other, while 'dog' and 'car' will be far apart.
The Algebra of Meaning
This geometric arrangement is what allows LLMs to capture nuanced relationships. The distance and direction between vectors encode meaning. This leads to one of the most famous examples of vector arithmetic: vector('King') - vector('Man') + vector('Woman') results in a vector that is remarkably close to vector('Queen'). This isn't a programmed rule; it's an emergent property of the model learning statistical patterns from text.
But how do we measure 'closeness' in a space with thousands of dimensions? While we could use Euclidean distance (the straight-line path between two points), it becomes less meaningful in high dimensions. Instead, models primarily use cosine similarity which measures the angle between two vectors.
A cosine similarity of 1 means the vectors point in the exact same direction, indicating very similar meaning. A value of -1 means they point in opposite directions. A value of 0 means they are orthogonal, or conceptually unrelated. This focus on direction rather than magnitude makes cosine similarity a robust measure of semantic relatedness.
The Embedding Matrix
So where do these vectors come from? They are stored in a giant lookup table called an This matrix has one row for every token in the model's vocabulary and one column for each dimension of the embedding space. When the model processes text, it converts each token into its corresponding integer index, and then simply retrieves the appropriate row (vector) from this matrix.
This process transforms a discrete sequence of token IDs into a continuous, semantically rich representation. This matrix multiplication is the very first step in how an LLM 'sees' text. It converts the symbolic nature of language into a geometric one, setting the stage for all the complex computations that follow in the transformer's attention layers.
Let's test your understanding of how language becomes math.
What is the primary way a large language model represents a piece of text, like a word or sentence?
In the vector space of an LLM, the words 'dog' and 'puppy' have vectors that are very close to each other. What does this proximity signify?
By representing words as vectors in a high-dimensional space, LLMs can perform powerful reasoning and analogy tasks that go far beyond simple keyword matching.