Before 2017, models for understanding language, like Recurrent Neural Networks (RNNs), processed text one word at a time, in order. This sequential approach made it difficult to grasp the relationships between distant words in a long sentence. It was also slow.

The Transformer architecture, introduced in a paper titled "Attention Is All You Need," changed everything. Instead of processing words sequentially, it looks at the entire sentence at once, weighing the importance of every word in relation to every other word. This parallel processing is faster and much better at capturing context.

Self-attention is the mechanism that allows the model to determine how much focus to place on other words in the input sequence when processing a specific word. For the sentence "The cat sat on the mat and it purred," self-attention helps the model understand that "it" refers to the "cat," not the "mat."

To do this, the model creates three special vectors for each word (or more accurately, for each token's embedding): a Query, a Key, and a Value. Think of it like searching in a digital library:

• Query (Q): This is your search query. It represents the current word you're focusing on and what information it's looking for.
• Key (K): These are like the keywords or titles of all the books in the library. Each word in the sentence has a Key vector that can be compared against the Query.
• Value (V): This is the actual content of the book. It's the information that the word actually holds.

The model learns to create these Q, K, and V vectors by training three separate weight matrices. These matrices transform the initial word embeddings into the specialised vectors needed for the attention calculation.

The calculation works like this:

1. Score: The model calculates a score between the Query vector of the current word and the Key vector of every other word in the sentence. This is done using a dot product. A higher score means the words are more relevant to each other.
2. Scale and Normalise: These scores are scaled down to prevent them from becoming too large, then passed through a softmax function. Softmax converts the scores into probabilities that all add up to 1. These are the attention weights.
3. Sum: Each Value vector is multiplied by its corresponding attention weight. The weighted Value vectors are then summed up to produce the final output for the current word. This output is a new representation of the word that is rich with context from the rest of the sentence.

A single self-attention mechanism might focus on one type of relationship, like how pronouns refer to nouns. But language is complex. Words can be related in many ways—syntactically, semantically, and more.

This is where Multi-Head Attention comes in. Instead of doing the attention calculation just once, it does it multiple times in parallel. Each parallel run is called an "attention head."

Each head has its own set of Q, K, and V weight matrices. This allows each head to learn to focus on different kinds of relationships in the text. For example, one head might learn to track grammatical structure, while another tracks semantic meaning.

After each head calculates its attention output, the results are concatenated together and passed through another linear layer. This final step combines the insights from all the different heads into a single, unified output vector. This parallelisation allows the model to capture a much richer and more nuanced understanding of the text.

While the attention mechanism is the core, Transformers come in different architectural configurations, each suited for different tasks. Before we look at them, let's quickly recap the first steps of the process: Tokenisation and Embedding.

• Tokenisation: Raw text is broken down into smaller pieces called tokens. A token can be a word, part of a word, or even a single character.
• Embedding: Each token is mapped to a high-dimensional vector of numbers. This vector, or embedding, captures the token's semantic meaning.

Finally, because the model processes all tokens at once and has no inherent sense of order, we add Positional Encodings to the embeddings. These are vectors that give the model information about the position of each token in the sequence. Now, on to the architectures.

The choice of architecture depends entirely on the goal. For tasks that require understanding the full context of a sentence, like sentiment analysis, an encoder-only model like BERT is ideal. For generating new text, where the model needs to predict the next word based on what it's already written, a decoder-only model like GPT excels. And for tasks that transform an input sequence into a new output sequence, like translation, the combined power of an encoder and a decoder is the best fit.