Unlike recurrent neural networks (RNNs) that process text one word at a time, Transformers look at all words in a sequence simultaneously. This parallel processing is made possible by a mechanism called attention, which allows every word to assess its relationship with every other word in the input.

The core of this mechanism is called Scaled Dot-Product Attention. To understand it, we first need to see how input embeddings are transformed.

For each word's embedding vector in our input sequence, we generate three new, distinct vectors: a Query, a Key, and a Value. Think of this process like a library retrieval system:

• Query (Q): Represents a specific word's request for information. It's the question this word is asking.
• Key (K): Acts like a label for every word in the sequence. It describes what kind of information a word can offer.
• Value (V): Is the actual content or information that a word holds.

The Query vector of one word is compared against the Key vectors of all other words to determine relevance. The Value vectors of the most relevant words are then used to create a new, context-aware representation for the original word.

These vectors are created by multiplying the input embedding matrix, $X$, by three separate weight matrices ($W^Q$, $W^K$, and $W^V$) that are learned during training.

With these three matrices, we can now calculate the attention scores and produce a final output vector.

The process unfolds in a few distinct steps: scoring, scaling, applying softmax, and weighting the values.

This entire operation is captured in a single, elegant formula.

The scaling factor is a small but crucial detail. As the dimension of the key vectors ($d_k$) increases, the magnitude of the dot products ($QK^T$) can become very large. When fed into a softmax function, large inputs can push the function into regions with extremely small gradients. This is known as the vanishing gradient problem, which makes the model difficult to train.

By scaling the scores down, we ensure the inputs to softmax have a variance of 1, which leads to more stable gradients and smoother training.

The result of this calculation is a new matrix where each row is a contextually enriched vector. Each output vector is a blend of all the input Value vectors, weighted by how relevant each was to the word's Query. This process allows the model to capture complex relationships and long-range dependencies across the entire sequence, all at once.