At the heart of most modern Large Language Models (LLMs) is an architecture called the Transformer. Before it came along, models read text sequentially, word by word, much like we do. This was slow and made it difficult to capture long-range relationships in the text.

The Transformer changed the game by processing all the words in a sentence at the same time. This parallel processing is faster and, thanks to a clever mechanism, much better at understanding how words relate to each other, no matter how far apart they are. This core mechanism is called self-attention.

When you read the sentence, "The bee landed on the flower because it wanted nectar," your brain instantly knows that "it" refers to the bee, not the flower. Self-attention gives a model this same ability to link words and understand context.

To do this, the model creates three special vectors for each word in the input sentence:

• Query (Q): A vector representing the current word, asking a question like, "Who or what am I related to in this sentence?"
• Key (K): A vector that acts like a label for every other word. It says, "Here’s what I am and what I represent."
• Value (V): A vector containing the actual meaning or substance of each word.

The model calculates an "attention score" for the current word by comparing its Query vector with the Key vector of every other word in the sentence. A high score means the words are highly relevant to each other. These scores are then used as weights to create a weighted average of all the Value vectors. The result is a new, context-rich representation for the current word, infused with information from the words it’s most related to.

This process is repeated for every single word in the sentence, giving each one a new representation that is deeply aware of its surroundings. But a single perspective on these relationships might not be enough to capture all the nuances of language.

This is where multi-head attention comes in. Instead of calculating attention just once, the Transformer does it multiple times in parallel. Each of these parallel processes is called an "attention head."

Think of it like a panel of experts reading the same sentence. One expert might focus on grammatical relationships (subject-verb-object), another might look at causal relationships (cause and effect), and a third might track pronoun references. Each head learns to pay attention to different types of relationships in the text by using its own unique set of Query, Key, and Value transformations.

After each head has produced its own context-rich output vector, all these outputs are concatenated together and passed through one final linear layer. This combines all the different learned perspectives into a single, comprehensive representation that captures a much richer understanding of the sentence.

Self-attention has a blind spot: because it looks at all words at once, it has no inherent sense of word order. To the model, "The cat chased the dog" and "The dog chased the cat" would appear the same. To fix this, the Transformer adds Positional Encodings to the input. This is simply a vector of numbers that gives each word a unique signal about its position in the sequence.

After multi-head attention has done its job, the resulting vectors are passed to a Feed-Forward Neural Network (FFN). This network processes each word's vector independently. Its job is to perform further transformations on these representations, helping the model learn more complex patterns from the context provided by the attention layers. Every Transformer block contains both a multi-head attention layer and a feed-forward network.

These components, stacked in layers, form the Transformer architecture that has become the foundation for today's most powerful language models.