Most modern Large Language Models (LLMs) are built on an architecture called the Transformer. Before its introduction in 2017, models processed text sequentially, like reading a sentence one word at a time. This created a bottleneck, making it difficult to capture long-range dependencies in text.

The Transformer changed everything by processing all input tokens simultaneously. It achieves this through a clever mechanism called attention, which allows the model to weigh the importance of every other word in the input when considering a specific word.

The core of the Transformer is self-attention. Imagine you're reading the sentence: "The robot picked up the ball because it was heavy." To understand what "it" refers to, your brain instinctively weighs the importance of other words. "Robot" and "ball" are strong candidates; "picked" and "heavy" are less likely.

Self-attention mimics this process mathematically. For each input token (word), the model generates three vectors: a Query (Q), a Key (K), and a Value (V). Think of them this way:

• Query: A question about the current word. "I am the word 'it'; what am I referring to?"
• Key: A label for every other word in the sentence. "I am the word 'ball'; I am a physical object."
• Value: The actual substance of each word.

The model then compares the Query vector of the current word with the Key vector of every other word to determine how much attention it should pay to them.

This comparison is done using a dot product between the Query vector of the current word and the Key vector of another word, which produces a raw attention score. These scores are then passed through a softmax function, which converts them into positive numbers that sum to 1. These normalized scores act as weights.

Finally, each word's Value vector is multiplied by its corresponding weight. The weighted Value vectors are then summed up to produce the final output for the current word. This output vector is a new representation of the word, enriched with contextual information from the entire sentence.

A single self-attention mechanism is powerful, but it's like asking someone to analyze a sentence while only focusing on one type of relationship, such as pronoun-antecedent links. What about syntactic structure or semantic similarity?

This is where multi-head attention comes in. Instead of performing self-attention once, the model does it multiple times in parallel. Each parallel run is called an "attention head."

Before calculating attention, the model projects the initial Q, K, and V vectors into different, lower-dimensional representation subspaces for each head. This allows each head to learn and focus on a different aspect of the language. For example, one head might learn to track verbs and their subjects, while another tracks adjectives modifying nouns.

After each attention head produces its output vector, these vectors are concatenated together and passed through a final linear layer. The result is a single, rich vector that combines insights from all the different attention heads.

The original Transformer model had two main parts: an encoder and a decoder. The encoder's job was to read and understand the input text (e.g., a sentence in English). The decoder's job was to generate the output text (e.g., the same sentence translated into French), using the encoder's understanding as context.

Each encoder and decoder is made up of a stack of identical layers. A standard encoder layer contains a multi-head attention block followed by a simple, position-wise feed-forward network. A decoder layer has a similar structure but includes an additional multi-head attention block that pays attention to the output of the encoder.

One crucial problem remains. Since the Transformer processes all words in parallel, how does it know the order of the words? Without this information, "the dog chased the cat" is identical to "the cat chased the dog."

The solution is positional encodings. Before the input tokens are fed into the first encoder layer, a vector containing information about each token's position in the sequence is added to its embedding. This injects the vital word-order information that the self-attention mechanism can then use.

Together, these components—self-attention, multi-head attention, positional encodings, and the encoder-decoder structure—form the foundation of the Transformer architecture that has enabled the recent leap forward in AI capabilities.