The original Transformer model had two distinct parts: an encoder to digest input text and a decoder to generate output. This encoder-decoder structure is great for tasks like translation, where you transform one full sequence into another. But for generating text, a different approach has proven more efficient.

Most modern LLMs, like the GPT series, are decoder-only. They discard the encoder and use a single, unified block to both process the input and generate the next word. This simplifies the architecture and training process. The model learns to predict the next token in a sequence, a task that inherently requires it to understand the preceding context. By focusing on this single, powerful objective, decoder-only models excel at open-ended generation, conversation, and instruction following.

As models grew, a brute-force approach of simply adding more layers became inefficient. Training costs skyrocketed, and returns diminished. This led to a clever architectural shift: the Mixture-of-Experts (MoE) model.

An MoE model works the same way. It replaces some of its standard feed-forward layers with a set of smaller "expert" networks. For any given input token, a router network activates only a few of these experts. This means a model can have a massive number of parameters in total, but only a fraction are used for any single computation. The result is a model that trains much faster and can be larger than a traditional "dense" model for the same computational budget.

So, how do you decide how big to make a model and how much data to train it on? For a long time, the strategy was to prioritize model size. But research from DeepMind produced a set of guidelines known as the Chinchilla scaling laws that changed everything. The key finding was that for a given compute budget, most existing LLMs were too large for the amount of data they were trained on.

The laws suggest that for optimal performance, model size and training data size should be scaled in roughly equal proportion. If you double the number of parameters, you should also double the number of training tokens. This insight means that a smaller model trained on more data can outperform a larger model trained on less.

A powerful model is useless if it's too slow to generate responses. The autoregressive nature of decoder-only models—generating one token at a time—creates a major bottleneck. At each step, the model has to re-process the entire sequence of previously generated tokens just to produce the next one.

To solve this, we use a Key-Value cache, or KV Cache. The attention mechanism in a Transformer computes three matrices from the input: Query (Q), Key (K), and Value (V). The Q is specific to the current token, but the K and V matrices for all previous tokens remain the same. The KV cache stores these K and V tensors after they're computed for the first time. In subsequent steps, the model only needs to compute the Q, K, and V for the new token and can reuse the cached values for all previous ones.

While the KV Cache reduces computation, it introduces a new problem: memory bandwidth. The size of the cache grows with the length of the sequence, and constantly reading from this large cache can max out the memory speed of the GPU. This is where FlashAttention comes in.

FlashAttention is an algorithm that reorders the attention computation to avoid the need to read and write the full attention matrix to the GPU's main memory. It computes attention in smaller blocks, using the GPU's faster on-chip SRAM. This reduces the amount of memory traffic, making attention significantly faster and more memory-efficient, which in turn allows for much longer context windows.

Time to check your understanding of these core architectural concepts.

Understanding these architectural trade-offs is key to building and deploying LLMs effectively. Decisions about using a decoder-only model, implementing MoE, or using FlashAttention directly impact a model's performance, speed, and cost.