At the heart of nearly every modern Large Language Model (LLM) is an architecture called the Transformer. Before it came along, models like Recurrent Neural Networks (RNNs) processed text sequentially, word by word. This was slow and made it difficult for the model to remember connections between distant words in a long text.

The Transformer changed everything by processing all words in a sequence at the same time. Its core innovation is a mechanism that allows it to weigh the importance of every other word when interpreting a single word. This is called attention.

The specific mechanism used in Transformers is called self-attention. It helps the model understand context. For instance, in the sentence "The bee landed on the flower because it wanted nectar," the word "it" refers to the "bee." Self-attention allows the model to computationally determine this link by creating a pathway between "it" and "bee," assigning a higher weight or "attention score" to that connection.

To do this, the model creates three vectors for each word: a Query (Q), a Key (K), and a Value (V). The Query is like a question the current word asks: "Who am I related to?" The Key from every other word acts as a label, saying, "This is what I am." The model compares the Query of one word to the Key of all other words to find matches. When a strong match is found, the Value of that matching word is passed along, influencing the final representation of the original word. This happens for every word in the sequence simultaneously.

A single self-attention calculation gives the model one perspective on the text's relationships. To get a richer understanding, Transformers use Multi-Head Self-Attention. This means the model runs the Q, K, and V process multiple times in parallel, with each "head" learning to focus on different types of relationships. One head might focus on grammatical links (subject-verb), another on semantic relationships (synonyms), and another on positional proximity. The outputs from all heads are then combined, providing a more comprehensive and nuanced representation of the sequence.

Since self-attention processes all words at once, the model loses the original word order. To fix this, Transformers inject information about each word's position into its input embedding. This is called Positional Encodings. It’s like adding a unique numerical signature to each word that tells the model whether it’s the first, second, or tenth word in the sequence. This way, the model knows that "The dog chased the cat" is different from "The cat chased the dog."

Inside each Transformer block, two other crucial components ensure the model trains effectively: Layer Normalization and Residual Connections. Layer Normalization standardizes the outputs of a layer to have a mean of zero and a standard deviation of one, which prevents the numbers from becoming too large or small and helps stabilize the training process.

Residual Connections, also called skip connections, create a shortcut. They take the input of a layer and add it to the output of that layer, allowing the network to easily learn an identity function (i.e., just pass the input through unchanged). This helps combat the vanishing gradient problem in very deep networks and makes it easier for information to flow through the model.

Transformer models are built from stacks of these building blocks, but they aren't all the same. They generally fall into three categories.

In Encoder-Decoder models, a special type of attention called cross-attention is used. While the encoder uses self-attention to understand the input text, and the decoder uses self-attention to understand the text it has generated so far, cross-attention connects the two. At each step, the decoder's cross-attention layer looks back at the encoder's output. Its Query vectors come from the decoder, while its Key and Value vectors come from the encoder. This allows the decoder to focus on the most relevant parts of the original input sentence as it generates each new word in the output.

Understanding these core components—attention, positional encodings, and the different architectural patterns—is key to grasping how LLMs work and why certain models are better suited for specific tasks.