LLM Architecture and Training Mechanics
From States to Attention
From States to Attention
You know how Markov chains work: they predict the next step based only on the present state. It's like walking a path where your next footstep only depends on where your last one landed. This is powerful for modeling systems with short-term memory, but it falls short when dealing with the complexities of human language.
Consider the sentence: "The directors who founded the company in 2010 finally sold it." To understand what "it" refers to, you need to look back past "sold" and "finally" all the way to "company." A simple Markov chain, looking only at the word "sold," would have no clue. It lacks the long-range memory required to grasp context.
This is the core problem that older language models faced. They were good at predicting the next word based on the last few words, but terrible at understanding the broader story of a sentence or paragraph.
Self-Attention: A Wider View
Transformer models solve this memory problem with a mechanism called self-attentions. Instead of just looking at the previous word (or state), self-attention allows the model to look at every other word in its context window simultaneously. It weighs the importance of each word relative to the current word it's processing. It asks, "To understand this word, which other words in this sentence are the most important clues?"
This process involves creating three vectors for each word: a Query, a Key, and a Value.
- Query (Q): Represents the current word asking for information.
- Key (K): Acts like a label for all other words in the sequence, announcing their potential relevance.
- Value (V): Contains the actual substance or meaning of each word.
The model matches the Query of the current word with the Keys of all other words. A strong match results in a high attention score, meaning the Value of that other word is given more weight in understanding the current word.
A New Kind of Markov Chain
We can reframe self-attention using a concept called a Context-Conditioned Markov Chain (CCMC). It's a way to bridge the gap between our old and new models. In a standard Markov chain, the probability of transitioning to the next state depends only on the current state. The transition matrix is fixed.
In a CCMC, the "state" is not just the last token. Instead, the transition to the next state depends on a weighted representation of the entire sequence within the context window. Self-attention dynamically calculates the transition probabilities for each new word based on the full context. The state itself is conditioned by context.
This is why Transformers excel. They aren't bound by a fixed, local memory. They can dynamically form long-range connections between words, capturing the nuanced dependencies that give language its rich meaning.
| Feature | Markov Model | Transformer (Self-Attention) |
|---|---|---|
| Memory | Fixed, based on current state only | Dynamic, based on entire context window |
| Context | Local (only immediate past) | Global (entire sequence) |
| Dependencies | Struggles with long-range connections | Excels at capturing long-range connections |
| Transitions | Static transition probabilities | Context-dependent transition probabilities |
This leap from fixed states to dynamic, context-aware attention was necessary to move beyond simple statistical patterns. It allows models to understand grammar, resolve ambiguity, and generate coherent, long-form text in a way that was previously impossible.