The core of the Transformer architecture is its attention mechanism. Instead of processing data sequentially like an RNN, attention allows the model to weigh the importance of all other words in a sequence when encoding a specific word. This is accomplished through three key vectors for each input token: the Query, the Key, and the Value.

For every input token, we create a Query vector (Q), a Key vector (K), and a Value vector (V). These are generated by multiplying the token's embedding by three distinct weight matrices ($W^Q$, $W^K$, $W^V$) that are learned during training. These matrices project the input embeddings into different representation subspaces, allowing the model to calculate attention from different perspectives.

The model calculates an attention score by taking the dot product of a word's Query vector with the Key vectors of all other words in the sequence. A higher dot product signifies greater relevance between the query word and a key word.

This process produces raw attention scores. To stabilize the gradients during training, these scores are scaled down by dividing by the square root of the dimension of the key vectors, $d_k$. This scaling is crucial. Without it, for large values of $d_k$, the dot products can grow very large, pushing the softmax function into regions where it has extremely small gradients. This makes learning difficult.

After scaling, a softmax function is applied to the scores. This converts the scores into a probability distribution, where all values are between 0 and 1 and sum to 1. Each value represents the weight of attention one word should pay to another. Finally, these attention weights are multiplied by the Value vectors of each word in the sequence and summed up to produce the final output for the query word. This output is a weighted blend of all other words' values, with more relevant words contributing more to the final representation.

A single attention mechanism can only focus on one type of relationship between words at a time. To overcome this limitation, Transformers use Multi-Head Attention. Instead of performing a single attention function, this mechanism runs Scaled Dot-Product Attention multiple times in parallel.

The input Q, K, and V matrices are linearly projected into different, smaller-dimensional subspaces for each "head." Attention is then calculated for each head independently. This allows the model to jointly attend to information from different representation subspaces at different positions. For example, one head might learn to track syntactic relationships, while another tracks semantic ones.

After each head produces its output, the outputs are concatenated back together and passed through a final linear projection. This combines the knowledge learned from the different attention heads into a single, unified representation.

Let's review the core concepts we've covered.

Now, check your understanding of these mechanisms.

By running multiple attention mechanisms in parallel, Multi-Head Attention gives the Transformer model a more nuanced understanding of language, forming the foundation for its powerful capabilities.