When we read a sentence, we instinctively understand how words relate to each other, even if they're far apart. For example, in the sentence, "The robot picked up the screwdriver because it was heavy," we know "it" refers to the screwdriver, not the robot.

But how does a machine learn this? Older models struggled with these long-range connections. The breakthrough came with a mechanism called self-attention. It allows a model to weigh the importance of different words in a sequence when processing any single word.

This process is at the heart of powerful foundation models like BERT and GPT, enabling them to grasp nuance, resolve ambiguity, and generate coherent text.

To figure out which words to focus on, the self-attention mechanism uses an analogy similar to a library's search system. For every word in the input, the model creates three distinct vectors: a Query, a Key, and a Value.

• Query (Q): This is the current word we're focusing on. Think of it as your question to the librarian: "What am I looking for?"
• Key (K): This vector is like the label on a file folder or a book's spine. Every word in the sentence generates a Key, which says, "This is the kind of information I contain."
• Value (V): This is the actual content of the word, the information itself. If the Key is the label, the Value is the document inside the folder.

The model calculates an attention score by comparing the Query of one word with the Key of every other word in the sequence. If a Query and a Key are similar, it means the two words are highly relevant to each other.

This scoring happens through a process called scaled dot-product attention. The model takes the dot product of the Query vector ($Q$) and the transpose of the Key vector ($K^T$). This result is then scaled down to prevent the values from becoming too large, which helps with training stability. The scaling factor is the square root of the dimension of the key vectors ($d_k$).

The softmax function is crucial here. It takes the raw attention scores and transforms them into a set of positive numbers that add up to one. Each number represents an "attention weight," signifying how much focus the current word should place on every other word in the sentence, including itself.

A high weight means a strong connection. The final output for the word is a weighted sum of all the Value vectors in the sentence, colored by these attention weights. This output is a new, context-aware representation of the original word, ready for the next layer of the network.