Older models for understanding language, like Recurrent Neural Networks (RNNs), had a specific way of working. They read sentences one word at a time, from left to right. This is similar to how we read, but it has a major drawback for computers. By the time an RNN reaches the end of a long paragraph, it might have forgotten what was said at the beginning. This makes it hard to grasp the full context.

This sequential process is also slow. A sentence can't be processed until the word before it has been fully analyzed. Transformers changed all of this by looking at every word in a sentence at the same time. This parallel approach is not only faster but also much better at understanding how words relate to each other, no matter how far apart they are.

At its core, the original Transformer model has two main parts: an encoder and a decoder. Think of them as two specialists working together on a task like language translation.

The encoder's job is to read the entire input sentence and build a rich, numerical representation of it. It figures out the meaning and context of each word. This representation, sometimes called a "context vector," is then passed over to the decoder.

The decoder's job is to take that context vector and generate the output, one word at a time. For translation, it would produce the translated sentence. For a chatbot, it would generate a reply. Critically, as the decoder generates each new word, it also looks at the words it has already produced, ensuring the output is coherent.

The secret ingredient that makes Transformers so effective is a mechanism called self-attention. It allows the model to weigh the importance of different words in a sentence when processing a specific word. Instead of just looking at the words immediately next to it, a word can "pay attention" to any other word in the sentence to better understand its context.

This works by assigning three vectors to each word in the input: a Query, a Key, and a Value.

• Query (Q): This is the current word, asking a question like, "What should I pay attention to?"
• Key (K): This is a label for all the other words in the sentence. It's used to be "found" by the Query.
• Value (V): This is the actual substance or meaning of each word.

To calculate the attention for a given word, its Query vector is compared with the Key vector of every other word. The result is an attention score, which determines how much focus to place on each of those other words. These scores are then used to create a weighted sum of all the Value vectors, producing a new representation of the original word that is rich with context from the entire sentence.

By doing this for every word in parallel, the Transformer builds a deep understanding of the sentence's internal structure and dependencies. This ability to see the whole picture at once is what makes it so powerful and efficient compared to older models.

Let's check your understanding of these core ideas.

These foundational concepts are the building blocks for nearly all modern language AI.