Before transformers, models like Recurrent Neural Networks (RNNs) processed text sequentially, word by word. This created an information bottleneck. The model's understanding of a sentence was compressed into a single, evolving state that had to carry the meaning of all previous words forward. This made it difficult to capture relationships between words that were far apart.

Transformers changed the game by looking at the entire sentence at once. The core innovation is the self-attention mechanism, which allows the model to weigh the importance of every other word in the input when processing a specific word. It learns the context not by order, but by relevance.

Imagine you're at a library searching for information. For any given book you're holding (a token), you create a query. You then compare this query against the title on the spine of every other book on the shelf (the keys). When you find a book with a relevant title, you pull it out and read its contents (the value). Self-attention works on a similar principle, but with vectors and matrix multiplication.

For every input token (typically a word or part of a word), the model generates three distinct vectors:

• Query (Q): Represents the current word's request for information. It's asking, "What other words are relevant to me?"
• Key (K): Acts as a label for a word, describing what it has to offer. It responds to queries, saying, "This is the kind of information I provide."
• Value (V): Contains the actual content or meaning of a word. This is what gets passed along if the key matches a query.

To calculate the attention score for a word, its Query vector is multiplied by the Key vector of every other word in the sequence. This dot product produces a raw score indicating relevance. These scores are then scaled and passed through a softmax function, which converts them into a set of weights that sum to 1. Finally, each word's Value vector is multiplied by its corresponding weight, and the results are summed up to create a new, context-aware representation of that word.

This process happens for every word in parallel. To improve this, transformers use Multi-head Attention a technique where the attention process is run multiple times simultaneously in different "representation subspaces." Each "head" learns different types of relationships. One head might focus on grammatical links, another on semantic similarity, and so on. The outputs from all heads are then concatenated and processed through another linear layer, creating a richer, more nuanced understanding of the text.

Because self-attention processes all tokens at once, it loses the original sequence order. The model sees a "bag of words" and has no inherent sense of which came first. To fix this, transformers use Positional Encoding vectors are added to the input embeddings at the beginning of the process. These vectors provide information about the position of each word in the sequence.

Early models used a clever trick with sine and cosine functions of different frequencies. This method has a nice property: for any position, the encoding is unique, and the model can learn to understand relative positions because there's a consistent linear relationship between the encodings of nearby words. More recent models often use learnable positional embeddings, where the model learns the best vector for each position during training.

After the attention mechanism has done its work, the resulting vectors are passed through a Feed-Forward Network (FFN). This is a standard component of neural networks, consisting of a couple of linear layers with a non-linear activation function in between (usually ReLU or GeLU). Each token's vector is processed independently by the same FFN. Its job is to add further processing and complexity, transforming the context-rich vectors into a format that's useful for the next layer or the final output.

By stacking these layers of multi-head attention and feed-forward networks, transformer models can build incredibly sophisticated representations of language. This architecture, with its ability to parallelise computation and capture long-range dependencies, is the engine behind the remarkable capabilities of modern generative AI.