Large Language Models (LLMs) seem to understand and generate human-like text with incredible fluency. But how do they actually process language? The magic isn't in a giant dictionary or a set of grammar rules. It's in an architecture called the Transformer, introduced in a 2017 paper titled "Attention Is All You Need."

Instead of reading a sentence word by word sequentially, like earlier models, Transformers process all words at once. Their key innovation is a mechanism that allows them to weigh the importance of every other word in the sequence, creating a rich understanding of context. This mechanism is called self-attention.

Imagine you're reading the sentence: "The cat sat on the mat, and it purred." To understand what "it" refers to, you intuitively link it back to "the cat." Self-attention is the computational equivalent of this intuition. It allows the model, when processing one word, to look at all other words in the input and determine which ones are most relevant.

To do this, the model creates three vectors for each word: a Query (Q), a Key (K), and a Value (V). You can think of them like this:

• Query: A question about the word's role and context. For the word "it," the query might be, "What noun could I be referring to?"
• Key: A label or identity for each word. The Key for "cat" might effectively say, "I am a singular noun that can perform actions."
• Value: The actual substance or meaning of the word.

By comparing the Query vector of one word to the Key vectors of all other words, the model calculates an "attention score." A high score means the words are highly relevant to each other. These scores are then used as weights to create a new representation for the word, built from a weighted sum of all other words' Value vectors.

This process allows the model to dynamically build context. For the word "it," the Key for "cat" would generate a high attention score, so the final representation of "it" becomes heavily infused with the meaning of "cat."

A single self-attention mechanism might learn to focus on one type of linguistic relationship, like linking pronouns to nouns. But language is complex. Words can relate to each other in many ways simultaneously: subject-verb, adjective-noun, cause-effect, and so on.

This is where Multi-Head Attention comes in. Instead of performing self-attention once, the model runs several attention mechanisms in parallel. Each of these "heads" has its own set of learned Q, K, and V weight matrices, allowing it to focus on a different kind of relationship within the sentence.

For example, in the sentence "The smart dog quickly chased the red ball," one attention head might link "quickly" to "chased" (how the action was done), while another links "smart" to "dog" and "red" to "ball" (descriptive properties). The outputs of all these heads are then concatenated and processed through a final linear layer, providing a comprehensive contextual representation for the next stage of the network.

Because Transformers process all words in a sequence at the same time for efficiency, the model has no inherent sense of word order. To the self-attention mechanism, "The dog chased the cat" and "The cat chased the dog" would look identical. This is a problem, as word order is fundamental to meaning.

The solution is Positional Encodings. Before the input words are fed into the first Transformer layer, a vector representing each word's position in the sequence is added to its initial embedding. This encoding gives the model a sense of sequence, allowing it to distinguish between words at different positions.

With positional information baked into the word representations, the model can learn patterns that depend on order, such as the grammatical importance of a word appearing at the beginning of a sentence.

A full Transformer model is built by stacking these components. The two main types of architectures are the Encoder and the Decoder.

Encoder-only models, like BERT, are designed to build a deep understanding of text. They use self-attention to look at the entire sentence in both directions (left and right) to create rich contextual representations. This makes them excellent for tasks like sentiment analysis, text classification, and named entity recognition, where a full understanding of the input is key.

Decoder-only models, like the GPT series, are built for generation. Their self-attention mechanism is "masked," meaning that when processing a word, they can only look at the words that came before it. This auto-regressive property makes them ideal for predicting the next word in a sequence, which is the basis for text generation.

Both architectures also include Feed-Forward Networks and Layer Normalization within each block. After the attention mechanism processes the inputs, the output passes through a simple but crucial feed-forward network for further transformation. Layer normalization is applied at different steps to stabilize the network and ensure smooth training.

These components—self-attention, multi-head attention, positional encodings, and the encoder/decoder structure—form the building blocks of modern LLMs. By combining them, Transformers can process language with a level of contextual awareness that was previously out of reach, powering the generative AI tools we use today.