Transformers don't process text sequentially like humans do. Instead, they look at an entire sequence at once. To understand the relationships between words, they use a mechanism called self-attention. Think of it as the model asking, for each word, "Which other words in this sentence are most important to understanding me?"

To do this, the model creates three distinct vectors for each input token (which is basically a word or part of a word). These vectors are the Query (Q), the Key (K), and the Value (V). They are generated by multiplying the token's embedding—its numerical representation—by three separate weight matrices that are learned during training.

Here’s the intuition:

• Query: This vector is like a question. It represents the current word that is trying to understand its context. It asks, "What am I looking for?"
• Key: This vector is like a label or a signpost on all the other words in the sentence. It says, "Here’s what I have to offer."
• Value: This vector contains the actual meaning or substance of a word. If the Key is the signpost, the Value is the destination itself.

The model calculates an attention score by taking the dot product of the Query vector of the current word with the Key vector of every other word in the sequence. A high score means the two words are highly relevant to each other. These scores are then scaled down for numerical stability, passed through a softmax function to turn them into probabilities, and finally used to create a weighted sum of all the Value vectors. The result is a new vector for the current word that is rich with context from the entire sentence.

A single self-attention mechanism might learn to focus on one type of relationship, like how verbs relate to their subjects. But language is complex; there are grammatical, semantic, and other types of relationships happening all at once. This is where multi-head attention comes in.

Instead of just one set of Q, K, and V weight matrices, multi-head attention uses multiple sets—or "heads." Each head gets its own set of learned matrices. The input embeddings are fed into each head, which independently performs the self-attention calculation. This allows the model to learn different types of relationships in parallel.

The outputs from all the heads are concatenated and passed through another learned linear transformation to produce the final output of the layer. This combined output captures a much richer understanding of the text by integrating different contextual perspectives simultaneously.

Because self-attention looks at all words at once, the model has no inherent sense of word order. To a basic Transformer, "the cat chased the dog" and "the dog chased the cat" would look the same. To fix this, Transformers use positional encodings.

These are vectors that are added to the input embeddings. Each encoding provides information about the position of a word in the sequence. It's a signal that gives each token a unique

It's a signal that gives each token a unique "address" in the sequence, allowing the model to understand word order and learn patterns that depend on it.

These components—self-attention and positional encodings—are bundled together into blocks, which are then stacked to form the complete Transformer. A standard architecture consists of an encoder stack and a decoder stack.

The encoder's job is to read the input sequence and build a rich numerical representation of it. The decoder's job is to take that representation and generate an output sequence, one token at a time. However, not all models use both.

Encoder-only models, like BERT (Bidirectional Encoder Representations from Transformers), are great at tasks that require a deep understanding of context, such as sentiment analysis or question answering. They see the entire input at once.

Decoder-only models, like GPT (Generative Pre-trained Transformer), are designed for generation. They are auto-regressive, meaning they predict the next token in a sequence based on the ones that came before. This makes them ideal for tasks like writing articles, code, or chatbot responses.

Let's check your understanding of these core components.

Understanding these internal mechanics is key to grasping how modern LLMs process language and generate human-like text.