At the heart of the Transformer is the self-attention mechanism. Unlike recurrent neural networks (RNNs) that process text sequentially, self-attention allows a model to look at all the words in a sequence at once. It weighs the importance of every other word as it processes a single word, creating a rich, context-aware representation.

For example, in the sentence "The cat chased the mouse because it was scared," the word "it" is ambiguous. Is the cat scared, or is the mouse scared? Self-attention allows the model to calculate a relationship, or an "attention score," between "it" and every other word. It would likely assign a high score to the connection between "it" and "mouse," clarifying the meaning.

To achieve this, the model creates three vectors for each input token (word): a Query, a Key, and a Value. You can think of them like this:

• Query (Q): A representation of the current word, asking a question about its context. It's looking for other words that are relevant to it.
• Key (K): A label or an identifier for each word in the sequence. It's what other words present to be matched against a query.
• Value (V): The actual content or meaning of each word. This is what gets passed along if a key matches a query.

The model learns to create these vectors during training. The core operation involves matching the Query vector of one word with the Key vectors of all other words in the sequence to determine how much attention to pay. The result is a weighted sum of the Value vectors.

The calculation itself is called Scaled Dot-Product Attention. It's an efficient matrix operation that computes the attention scores for all words in parallel.

While powerful, a single attention calculation might only capture one type of relationship. To understand language, a model needs to consider syntax, semantics, and other nuances simultaneously. This is where multi-head attention comes in.

Instead of using one set of Q, K, and V matrices, multi-head attention uses multiple sets, or "heads." Each head operates in a different "representation subspace," meaning it learns to pay attention to different kinds of relationships. For instance, one head might focus on which verb an adjective modifies, while another tracks the subject of the sentence.

The inputs are first passed through separate linear layers to project them into lower-dimensional Q, K, and V spaces for each head. Attention is calculated for each head in parallel. Finally, the outputs from all heads are concatenated and passed through another linear layer to produce the final output.

A key detail of the self-attention mechanism is that it is permutation-invariant. It treats the input as an unordered set of tokens, meaning "the cat sat on the mat" and "the mat sat on the cat" would produce similar internal representations if not for an explicit mechanism to encode word order. The model has no inherent sense of sequence.

To solve this, Transformers inject information about each token's position into the input embeddings. Early models used absolute positional encodings, where a unique vector was added to each token's embedding based on its absolute position (1st, 2nd, 3rd, etc.). While effective, this approach can struggle with very long sequences and has limited generalization to lengths not seen during training.

Within a Transformer block, the multi-head attention sub-layer is followed by a feed-forward neural network. To help with training these deep networks, two other crucial components are used: residual connections and layer normalization.

• Residual Connections: Also known as skip connections, these paths allow the input of a layer to be added directly to its output. This helps prevent the vanishing gradient problem in deep networks, allowing information from earlier layers to propagate more easily to later ones.

• Layer Normalization: This step normalizes the activations within a single layer. It helps stabilize the learning process and often speeds up training.

There are two common arrangements: Post-LN, where normalization is applied after the residual connection, and Pre-LN, where it's applied before the sub-layer's operation. While Post-LN was used in the original Transformer paper, Pre-LN has since been found to provide more stable training for very deep models and is common in many modern LLMs.

These components—multi-head self-attention, positional encodings, residual connections, and layer normalization—form the building blocks of the Transformer. By stacking these blocks, models can build up increasingly complex and abstract representations of language.