LLM Architecture and Mechanics
Self-Attention Mechanics
The Engine of Understanding
At the heart of any Large Language Model (LLM) lies a mechanism that allows it to grasp context, nuance, and the intricate relationships between words in a sentence. This mechanism, called self-attention, is what separates modern Transformer architectures from their sequential predecessors like Recurrent Neural Networks (RNNs). While RNNs process a sentence word by word, struggling to remember connections over long distances, self-attention looks at the entire sentence at once. This parallel processing allows it to weigh the importance of every word in relation to every other word, creating a rich, context-aware understanding.
To achieve this, the model creates three distinct representations for each input token: a Query (Q), a Key (K), and a Value (V). These are generated by multiplying the token's initial embedding by three separate, learned weight matrices. You can think of it like a library search. The is your search term—the word you're currently focused on. The is like the index card for every other word in the sentence; your query compares itself to each key to find a match or measure relevance. The Value is the actual content on the page that the index card points to. By matching your query to the keys, you decide which values are most important to retrieve.
Calculating Attention
The core of this process is a calculation called Scaled Dot-Product Attention. It's a series of matrix operations that turns the Q, K, and V vectors into a single, contextually-informed output vector. This process happens in a few distinct steps.
- Score: The model calculates a score between the Query vector of the current word and the Key vector of every other word in the sentence. This is done using a dot product. A higher score means higher relevance.
- Scale: The scores are scaled down by dividing them by the square root of the dimension of the key vectors (). This prevents the dot product results from becoming too large, which helps stabilize the learning process.
- Weight: A [{
}] function is applied to the scaled scores. This turns the scores into positive numbers that sum to 1, effectively creating a probability distribution. These are the "attention weights," indicating how much focus to place on each word. - Aggregate: Finally, the attention weights are multiplied by the Value vectors of each word and summed up. This produces the final output vector for the current word, which is a weighted blend of all other words' values, informed by their relevance.
The entire operation can be expressed in a single, compact formula.
Multiple Perspectives
A single self-attention calculation allows a model to focus on one type of relationship. For example, it might learn to connect pronouns to the nouns they refer to. But language is complex. Words can relate through syntax, semantics, and other subtle patterns. To capture this richness, Transformers use (MHA).
MHA works by running the Scaled Dot-Product Attention mechanism multiple times in parallel. Each of these parallel runs is called an "attention head." Before each head runs, the original Q, K, and V matrices are projected through their own unique set of weight matrices. This allows each head to learn to focus on a different aspect of the sentence. One head might focus on grammatical structure, while another tracks semantic similarity.
After each attention head computes its output vector, the results are concatenated together and passed through a final linear layer. This process allows the model to simultaneously draw on different kinds of information from different parts of the sentence. The ability to calculate these heads in parallel, combined with the parallel nature of self-attention itself, is what makes the Transformer architecture so powerful and efficient, marking a significant leap forward from the sequential nature of older models.
What is the primary advantage of the self-attention mechanism in Transformers compared to sequential models like RNNs?
In the library search analogy for self-attention, what does the "Key" (K) vector represent?