The Transformer architecture is the engine behind most modern LLMs. Unlike older models that processed text word-by-word, Transformers look at an entire sequence at once. This parallel processing is faster, but it creates a challenge: how does the model know the order of the words? It also needs a way to figure out which words are most important to each other, no matter how far apart they are in a sentence. The two key solutions are positional encoding and the self-attention mechanism.

The core innovation of the Transformer is the self-attention mechanism, a process that allows the model to weigh the importance of every other word in the input text for any given word. It's like a networking event for words. As the model processes the word "bank" in the sentence "I sat on the river bank," self-attention helps it listen to the conversation, notice that "river" is a key conversational partner, and assign less weight to more distant words like "I."

This is done mathematically by creating three vectors for each word: a Query (Q), a Key (K), and a Value (V). The Query is like a word asking, "Who in this sentence is relevant to me?" The Key is like a label on every other word, announcing, "Here's what I'm about." The model compares the Query of one word to the Key of every other word to generate a score. Words with higher scores are more relevant.

These scores are then used to create a weighted sum of all the Value vectors in the sentence. The Value vector contains the actual meaning or content of a word. So, a word's final representation is a blend of all other words' values, weighted by their relevance. This lets the model build a rich, context-aware understanding of the text.

But one attention calculation isn't enough. A sentence can have multiple layers of meaning. For example, in "The board approved the plan, which the CEO signed," the word "board" relates to "approved" (an action) and "CEO" (a person). To capture these different relationships, Transformers use multi-head attention.

This involves running the self-attention process multiple times in parallel, each with different, independently learned Q, K, and V matrices. Each of these "heads" can focus on a different aspect of the sentence. One head might learn to track subjects and verbs, while another tracks pronoun references. The outputs from all heads are then combined, giving the model a much more nuanced and complete understanding of the text's structure and meaning.

Because self-attention processes all words simultaneously, it loses the original sequence. To fix this, the model adds a piece of information to each word's embedding: its positional encoding. This is a vector that represents the word's position in the sequence. It's calculated using sine and cosine functions of different frequencies. This method has a neat property: it allows the model to easily learn the relative positions of words. The model doesn't just know a word is at position 5; it learns how position 5 relates to position 2 or position 8, making it effective for sequences of varying lengths.

The core Transformer components, attention and positional encoding, are assembled into different architectures for different tasks. There are three main types: encoder-only, decoder-only, and encoder-decoder.

Encoder-only models, like BERT, are designed to build a deep understanding of text. The encoder's self-attention mechanism can look at the entire sentence, both words that come before and after the current word (bidirectional context). This makes them excellent for tasks like sentiment analysis, document classification, or named entity recognition, where a complete understanding of the input is crucial.

Decoder-only models, like the GPT series and Llama, are built for text generation. Their self-attention mechanism is masked, meaning it can only look at the words that came before it (unidirectional context). This is essential for generating coherent text one word at a time, as the model can't peek ahead at the words it hasn't produced yet. These are the go-to models for chatbots, content creation, and summarisation.

Encoder-decoder models, such as T5 and the original Transformer, are best for sequence-to-sequence tasks. The encoder processes the source text (e.g., a sentence in English), and the decoder generates the target text (e.g., the same sentence in German). The decoder pays attention not only to the words it has already generated but also to the encoder's output, ensuring the translation is faithful to the original input. This makes them ideal for machine translation and complex question-answering.

Understanding these architectural differences is key to selecting the right tool for a specific business problem. You wouldn't use a generation-focused decoder model for a classification task that requires deep contextual analysis, just as you wouldn't use an encoder-only model to write a marketing email. The architecture dictates the model's strengths and limitations.

With these building blocks, you can begin to see how LLMs move beyond simple pattern matching to develop a sophisticated grasp of language, context, and structure.