Imagine reading a sentence one word at a time, through a tiny slot that only reveals a single word. By the time you reach the end, you might forget the beginning. Early language models, like Recurrent Neural Networks (RNNs), worked a bit like this. They processed text sequentially, which made it hard for them to remember connections between distant words in a long sentence.

Then, in 2017, a paper titled "Attention Is All You Need" introduced a new architecture that changed everything: the Transformer. Instead of reading word by word, the Transformer processes the entire sentence at once. This parallel approach allows it to see the whole picture from the start, making it incredibly powerful at understanding context.

At its core, the Transformer model has two main parts. An encoder stack works to understand the meaning and context of the input text. A decoder stack then takes that understanding and generates an output, like a translation or an answer to a question. The secret sauce that powers both parts is a mechanism called self-attention.

Self-attention is what allows the model to weigh the importance of different words in a sentence when processing any single word. It helps the model create connections and understand relationships, no matter how far apart the words are.

Consider this sentence: "The bee landed on the flower because it wanted nectar." What does "it" refer to? For a human, the answer is obvious: the bee. Self-attention allows the model to make this same connection by linking "it" back to "bee," giving it a higher importance score in this context.

So how does self-attention work? It uses three special vectors for each word: a Query, a Key, and a Value.

You can think of it like searching in a library. You have a research topic (the Query). You scan the titles on the spines of the books (the Keys) to see how relevant they are to your topic. When you find a good match, you pull out the book to read its contents (the Value).

In a Transformer, every word generates a Query vector to ask for information. It then compares its Query against the Key vector of every other word in the sentence. This comparison produces an "attention score" that determines how much focus to place on each of those other words. Finally, the model takes a weighted sum of all the Value vectors, with the attention scores determining the weights. The result is a new, context-rich representation for the original word.

This process happens for every single word in the input, all at the same time. A word like "it" sends out its Query, finds a strong match with the Key for "bee," and then heavily weights the Value of "bee" when creating its own new meaning. This is how the model builds a deep, interconnected understanding of language.

This is the formula that calculates the attention scores. The Query (Q) is multiplied by the Key (K), scaled down, and then put through a softmax function to turn the scores into probabilities. These probabilities are then used to create a weighted sum of the Values (V).

By mastering this flow of queries, keys, and values, Transformers can capture complex relationships in text, forming the foundation of today's most advanced AI models.