At the heart of nearly every modern large language model lies the Transformer architecture. Its defining feature is the self-attention mechanism, a clever process that allows the model to weigh the importance of different words in a sequence. Think of how you read a sentence: your brain doesn't give equal weight to every word. When you read, "The cat, which was surprisingly agile, jumped onto the table," your mind links "jumped" back to "cat," not "table."

Self-attention does something similar. For each word it processes, it generates three vectors: a Query, a Key, and a Value. The Query is like a question: "What context is relevant to me?" The Key from every other word in the sentence acts like a label, saying, "This is the kind of information I represent." The model compares the Query of the current word with the Key of every other word to calculate an attention score. A high score means a strong relevance.

These scores are then used to create a weighted sum of the Value vectors. Words with higher scores contribute more to the final representation of the current word. This allows the model to build a rich, context-aware understanding of the text, knowing which words are most influential to the meaning of any other word in the sentence.

The core calculation for this process is called Scaled Dot-Product Attention.

Models don't see words; they see numbers. The first step in processing text is tokenization, which breaks down a raw string of text into smaller chunks called tokens. These can be whole words, like "cat," or sub-words, like "un-", "believ,", and "-able." Sub-word tokenization is powerful because it allows the model to handle words it has never seen before by breaking them into familiar parts. It also keeps the model's vocabulary size manageable.

Once the text is tokenized, each token is converted into a high-dimensional vector called an embedding. This isn't just a random ID. An embedding is a rich numerical representation that captures the token's semantic meaning. These vectors are organized in a vast multi-dimensional space where related concepts are located close to one another. This is how a model "understands" that "king" and "queen" are related in a way similar to "man" and "woman."

Not all Transformer models are built the same way. Their architecture is often tailored to specific tasks.

An encoder-decoder model is ideal for transforming an input sequence into a different output sequence, like translating French to English. The encoder "understands" the French sentence, and the decoder generates the English equivalent.

A decoder-only model is a pure next-word predictor. Given a prompt, it generates the most probable next token, appends it to the sequence, and repeats the process. This makes it a natural fit for open-ended text generation.

Finally, an encoder-only model excels at understanding text. By looking at a sentence from both left-to-right and right-to-left, it builds a deep understanding necessary for tasks like determining if a movie review is positive or negative.

As models grew to hundreds of billions of parameters, training and running them became incredibly expensive. A key challenge is that during any given inference pass, all of a model's parameters are activated, even if only a fraction of them are relevant to the task. This is known as dense activation.

A more efficient approach is a Mixture of Experts (MoE) architecture. An MoE model consists of multiple smaller "expert" networks and a router or gating network. For any given input, the router decides which one or two experts are best suited for the task and only activates them. This means you can have a model with a massive number of total parameters, but the computational cost for any single forward pass is much lower because you are only using a fraction of the model. It's like having a large committee of specialists, but you only call on the two who know the most about the current problem instead of convening the entire group for every decision.

Finally, the concept of a context window is crucial. This refers to the maximum number of tokens the model can consider at one time. A model with a 4,096-token context window can't "remember" the 4,097th token from the beginning of a document. Expanding this window is a major area of research, as larger context windows allow models to process entire books, codebases, or lengthy conversations, leading to better coherence and understanding over long stretches of text.

Understanding these core mechanics reveals that LLMs are not magical black boxes. They are complex but logical systems built on clever architectural principles that allow them to process language with remarkable nuance.