When a Large Language Model (LLM) generates a response, it's a two-stage process. First comes the prefill stage, where the model processes your input prompt all at once. This part is compute-intensive, making full use of the GPU's parallel processing power. But the second stage, decoding, is a different beast entirely. Here, the model generates the response one token at a time, autoregressively. Each new token depends on all the ones that came before it.

This step-by-step generation shifts the bottleneck from computation to memory. For each new token, the model must read the entire Key-Value (KV) cache, which stores the attention data for the preceding tokens. The speed of this process isn't limited by how fast the GPU can perform calculations, but by how fast it can read from its own memory. This makes the decoding phase memory-bound—performance is dictated by memory bandwidth, not raw processing power.

To manage the KV cache, standard inference systems take a simple but rigid approach: they allocate a single, contiguous block of memory for each sequence in a batch. The size of this block isn't based on the actual length of the request, but on the maximum possible sequence length the model supports. Whether your prompt is ten words or a thousand, the system reserves memory for the maximum, just in case.

This pre-allocation strategy creates significant problems. Because the memory blocks must be contiguous, finding space for new requests becomes difficult as the GPU memory gets occupied. It's an inefficient system that leads to a surprising amount of waste.

The waste from pre-allocation comes in three main forms.

Internal Fragmentation: This occurs within a single allocated block. If you pre-allocate space for 2048 tokens but the model only generates 500, the memory for the remaining 1548 tokens sits empty. It’s reserved and cannot be used by any other request.

External Fragmentation: As requests of different sizes are processed and their memory is freed, the GPU memory space becomes broken up. This can leave many small, non-contiguous gaps of free memory. Even if there's enough total free memory to handle a new request, if it can't be allocated as a single continuous block, it goes to waste. It’s like having ten empty parking spots, but none are next to each other, so you can't park a bus.

Reserved for Future Tokens: This is a specific form of internal fragmentation worth calling out. For any given request, the full memory block is reserved upfront. But during the decoding process, tokens are generated one by one. At any given moment, most of the allocated space is reserved for tokens that haven't even been generated yet. This memory is locked down and unusable by the rest of the system.

When you add up these inefficiencies, the results are stark. In many production systems, only 20-40% of the allocated KV cache is actually used to store token states. The remaining 60-80% is wasted, severely limiting the number of requests that can be processed concurrently.

This extreme inefficiency directly impacts throughput. With so much memory tied up, the system can't fit as many requests into a batch, leaving the powerful GPU underutilized. This sets the stage for a more intelligent approach to memory management, one that can reclaim this wasted space and unlock the true potential of the hardware.