Before Transformers, models like Recurrent Neural Networks (RNNs) processed text sequentially, word by word. This was like reading a sentence one word at a time, trying to remember everything that came before. It was slow and struggled with long-term dependencies—the relationships between words far apart in a text.

The Transformer architecture, introduced in a 2017 paper titled Attention Is All You Need, broke this mold. Instead of sequential processing, it handles all words in a sentence at once. This parallelism is the key to its power and scalability. But if you look at all the words simultaneously, how do you know which ones are important? The answer is a mechanism called self-attention.

Self-attention allows a model to weigh the importance of different words in a sequence when processing a specific word. Think of it as creating a dynamic, context-aware link between every word and every other word in the input.

For each word, the model generates three vectors: a Query (Q), a Key (K), and a Value (V). These vectors are created by multiplying the word's embedding (its numerical representation) by three distinct weight matrices that are learned during training.

• Query (Q): Represents the current word's request for information. It's asking, "What other words are relevant to me?"
• Key (K): Acts as a label or identifier for each word in the sequence. It responds to the Query, saying, "Here's the kind of information I hold."
• Value (V): Contains the actual substance or meaning of the word. It's the information that gets passed on if the Query and Key match well.

To calculate the attention score, the model computes the dot product of the Query vector for the current word with the Key vector for every other word in the sequence. This score determines how much focus to place on each word. These scores are then scaled, passed through a softmax function to turn them into probabilities, and used to create a weighted sum of all the Value vectors. The result is a new vector that represents the original word, but now enriched with context from the entire sequence.

This process happens for every single word in the input, all at the same time. The result is a set of new vectors, one for each word, that are deeply aware of their context.

A single self-attention mechanism might focus on one type of relationship, like how verbs relate to subjects. But language is complex; words can relate in many ways (syntactically, semantically, etc.). This is where Multi-Head Attention comes in.

Instead of just one set of Q, K, and V weight matrices, Multi-Head Attention uses multiple sets in parallel. Each of these “heads” learns to focus on a different aspect of the language. For an 8-headed attention block, the model would have 8 independent sets of Q, K, and V matrices. Each head produces its own output vector. These separate outputs are then concatenated and passed through a final linear layer to produce the final output for that block.

This is like having a team of specialists analyze a sentence. One might focus on grammar, another on semantic meaning, and a third on pronoun references. By combining their insights, the model gets a much richer and more nuanced understanding of the text.

After attention is calculated, the output for each word passes through a simple Feed-Forward Network. This network is applied independently to each position, adding further processing and transformation. Following both the attention and feed-forward blocks, a technique called Layer Normalization is used. It standardizes the inputs to each layer, which helps stabilize the training process and allows the model to train faster and more reliably.

Because the Transformer processes all words simultaneously, it has no inherent sense of word order. To the self-attention mechanism, "The dog bit the man" and "The man bit the dog" look the same without additional information. This is a big problem for understanding language.

The solution is Positional Encoding. Before the word embeddings are fed into the first Transformer block, a vector representing the word's position is added to it. This gives the model a signal about the order of the sequence.

One common method uses sine and cosine functions of different frequencies:

Some models also use learned positional encodings, where the positional vectors are treated as parameters that are updated during training, just like the other weights in the model.

The components we've discussed—self-attention, feed-forward networks, and positional encodings—are assembled into blocks. These blocks are then stacked to form two main components: an Encoder and a Decoder.

An Encoder block takes an input sequence and generates a rich numerical representation. A Decoder block takes the encoder's output (and its own previous output) to generate a new sequence. Different models use these components in different ways.

Each architecture is optimized for its task. Encoder-only models build a deep, bidirectional understanding of the input text. Decoder-only models are masters of generation, predicting the next word based on the sequence so far. Encoder-decoder models excel at transforming one sequence into another.

Time to test what you've learned about the mechanics of Transformers.

These core mechanics are the building blocks that enable the remarkable capabilities of modern large language models.