Transformer Mechanics

The Attention Revolution

In 2017, a paper titled "Attention Is All You Need" introduced a new architecture that changed everything: the Transformer. Before Transformers, models processed sequences like sentences one word at a time, which was slow and made it hard to connect words that were far apart. Transformers, however, can look at the entire sequence at once. This parallel processing is made possible by a mechanism that allows the model to weigh the importance of every word in relation to every other word.

The Transformer's core innovation lies in its self-attention mechanism, which allows the model to weigh the importance of different parts of the input data relative to each other, enabling parallel processing and capturing long-range dependencies more effectively than previous architectures like recurrent neural networks (RNNs).

This core idea, called the Self-Attention mechanism, is the engine behind most modern AI, from language models like GPT to image generation tools. It gives the model a deep, contextual understanding of the data it's processing.

How Self-Attention Works

Imagine you're in a library trying to answer a question. You have your question (a Query), and the library has books, each with a title on its spine (a Key) and its contents (a Value). You'd scan the titles (Keys) to see which ones are most relevant to your question (Query). Once you find the best matches, you pull those books and read their contents (Values).

Self-attention works in a similar way for every single token (word or part of a word) in a sequence. For each token, the model generates three vectors:

Query (Q): Represents the current token's question, asking, "Who in this sentence is relevant to me?"
Key (K): Represents the token's own label or identity, saying, "Here's what I am."
Value (V): Represents the token's actual meaning or content.

The model calculates an "attention score" by taking the dot product of the current token's Query vector with the Key vectors of all other tokens in the sequence. A high score means a strong relationship. These scores are then scaled and passed through a softmax function to turn them into weights that sum to 1. Finally, these weights are multiplied by each token's Value vector and summed up to produce the final output for the current token—a new representation that's rich with context from the entire sentence.

Attention(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

This entire process happens simultaneously for every token in the sequence, allowing the model to build a deep understanding of grammatical structure, context, and relationships, no matter how far apart the words are.

Multiple Perspectives

One attention calculation might focus on who is doing what. Another might focus on how adjectives modify nouns. Instead of relying on a single self-attention calculation, Transformers use a technique called Multi-Head Attention. This means the model runs the Q, K, V process multiple times in parallel, with each "head" learning to focus on a different type of relationship within the data.

Each head produces its own output vector. These are then concatenated and passed through a linear layer to produce the final output. This allows the model to attend to information from different representation subspaces simultaneously, capturing a much richer and more nuanced understanding of the input sequence. It's like having a team of experts analyze a sentence, with each expert focusing on a different aspect of its structure and meaning.

Two other crucial components help stabilize the training process: and layer normalization. A residual connection is a kind of shortcut that allows the input of a layer to be added directly to its output. This helps prevent the problem of vanishing gradients in deep networks. Layer normalization then rescales the output of the layer to have a mean of 0 and a standard deviation of 1, which ensures the training process remains stable.

Order in the Network

Because self-attention processes all tokens at once, it loses the inherent sense of order present in the original sequence. The model sees the words as a "bag" of tokens, not a sequence. To fix this, Transformers inject information about the position of each token. This is done using Positional Encodings.

Positional encodings are vectors that are added to each token's input embedding. These vectors are generated using a combination of sine and cosine functions of different frequencies. This method has a neat property: it allows the model to easily learn relative positions. The positional encoding for position + k can be represented as a linear function of the encoding for position, making it simple for the model to understand how far apart tokens are.

\begin{align*} PE_{(pos, 2i)} &= \sin(pos / 10000^{2i/d_{\text{model}}}) \\ PE_{(pos, 2i+1)} &= \cos(pos / 10000^{2i/d_{\text{model}}}) \end{align*}

Architectural Blueprints

The original Transformer had two main parts: an Encoder and a Decoder. However, modern models often use just one of these components, depending on their primary task.

Architecture	Description	Primary Use Case	Example Models
Encoder-Only	Processes the entire input sequence at once to build a rich contextual understanding. Each token can attend to all other tokens.	Analysis and understanding tasks like text classification, sentiment analysis, and named entity recognition.	BERT, RoBERTa
Decoder-Only	Generates an output sequence one token at a time. Each token can only attend to the preceding tokens in the sequence (causal attention).	Generative tasks like text completion, chatbots, and creative writing.	GPT series, LLaMA
Encoder-Decoder	Maps an input sequence to an output sequence. The encoder creates a representation of the input, and the decoder generates the output based on that representation.	Sequence-to-sequence tasks like machine translation, text summarization, and question answering.	T5, BART

Encoder-only models excel at tasks requiring a deep understanding of the entire input. Decoder-only models are masters of generation, predicting the next word based on what came before. Encoder-decoder models are best when you need to transform an input sequence into a completely new output sequence.

Quiz Questions 1/6

What fundamental problem in sequential models did the Transformer's self-attention mechanism solve?

Quiz Questions 2/6

In the self-attention mechanism, the model generates three vectors for each token. Match the vector to its purpose.

Mastering Generative AI Systems