In traditional sequence models like RNNs and LSTMs, information is processed one step at a time, like reading a book word by word. This sequential nature creates a bottleneck. Information from early in the sequence can get lost or diluted by the time the model reaches the end, making it difficult to capture long-range dependencies. The Transformer architecture, introduced in the paper 'Attention Is All You Need', broke this paradigm.

Instead of processing sequentially, Transformers look at all the data in a sequence at once. The core mechanism that enables this is called self-attention. It allows the model to weigh the importance of different words in the input sequence relative to each other, creating a rich, context-aware representation of each word. This parallel approach is not only more efficient but also far more effective at understanding the complex relationships within data.

Think of self-attention as a way for each word in a sentence to look at all the other words to better understand its own meaning in context. For the sentence "The cat sat on the mat," the word "sat" needs to know it's related to "cat" (who sat) and "mat" (where it sat).

To do this, the model creates three vectors for each input token: a Query (Q), a Key (K), and a Value (V). You can think of them this way:

• Query: A question from the current word, asking, "Who is relevant to me?"
• Key: A label from every other word, saying, "This is what I am."
• Value: The actual content or meaning of each word.

The model compares the Query vector of the current word with the Key vectors of all other words. The similarity between a Q and a K determines an "attention score," which is a weight. A high score means the pair of words is highly relevant. These scores are then used to create a weighted sum of all the Value vectors, producing a new representation for the current word that is infused with context from the entire sequence.

This process is formalized as Scaled Dot-Product Attention.

A single self-attention calculation is useful, but the model can learn more if it can focus on different types of relationships simultaneously. This is where Multi-Head Attention comes in.

Instead of performing one attention calculation, Multi-Head Attention runs the process in parallel multiple times (e.g., 8 or 12 "heads"). Each head has its own set of Q, K, and V weight matrices, allowing it to learn different aspects of the language. For example, one head might learn to track syntactic relationships, while another focuses on semantic similarity. The outputs from all heads are concatenated and passed through a final linear layer, combining all the learned information into a single, powerful representation.

Since the attention mechanism processes all tokens at once, it has no inherent sense of word order. To fix this, Transformers use Positional Encoding. These are vectors added to the input embeddings to give the model information about the position of each token in the sequence. The original paper used sine and cosine functions of different frequencies, creating a unique positional signature for each token.

Each Transformer block also contains a simple, position-wise Feed-Forward Network (FFN) and incorporates Layer Normalization. The FFN is applied independently to each position's representation after the attention step, providing additional computational depth. Layer Normalization helps stabilize the training of these deep networks by normalizing the inputs to each sub-layer. This prevents the values from exploding or vanishing, leading to faster and more reliable training.

These blocks are stacked on top of each other to form the full Transformer. Different arrangements of these blocks lead to different types of models.

The original Transformer had an encoder-decoder structure, but many modern models use only one part.

Encoder-only models (BERT): These models are designed to build a deep, bidirectional understanding of text. They use a stack of encoder blocks to create a rich contextual representation of the input. They are ideal for tasks that require understanding the whole sentence, such as sentiment analysis, text classification, and named entity recognition. BERT (Bidirectional Encoder Representations from Transformers) is the classic example.

Decoder-only models (GPT): These models are autoregressive, meaning they generate text one token at a time, based on the tokens generated previously. They use a stack of decoder blocks, which feature a modified self-attention mechanism that prevents them from "peeking" at future tokens during training. This makes them perfect for generative tasks like text completion, creative writing, and chatbots. The GPT (Generative Pre-trained Transformer) series popularized this architecture.

Encoder-Decoder models (T5): These models use both stacks. The encoder processes the input sequence, and its output is fed to the decoder, which then generates the output sequence. This structure excels at sequence-to-sequence tasks, such as machine translation, summarization, and question answering. The T5 (Text-to-Text Transfer Transformer) model frames every NLP problem as a text-to-text task, leveraging this architecture effectively.

Understanding these architectural differences is key to selecting the right model for a given task and is the foundation for almost all modern work in generative AI.