vLLM's prefix caching algorithm
Welcome! I am excited to dive into the architecture of vLLM and how its prefix caching algorithm makes serving Large Language Models incredibly efficient.
Before we jump into the mechanics of prefix caching, it helps to know where you are starting from. Are you already familiar with how Key-Value (KV) caching works in Transformer models, or would you like us to do a quick, intuitive refresher on that first?
I'm familiar with KV caching, let's skip the refresher
Got it! Since you are already comfortable with the basics of KV caching, we can skip right past the introductory mechanics and focus on the engine designed to make it run at scale.
At the center of high-performance open-source inference is vLLM, which stands for Virtual Large Language Model. Originally developed by UC Berkeley's Sky Computing Lab, is a high-throughput, memory-efficient serving engine designed to tackle the biggest bottleneck in deploying Large Language Models: GPU memory management.
?
To see why we need prefix caching, let's look at the hidden tax of LLM conversations: redundant computations.
Every time you chat with an LLM in a multi-turn conversation, or send different queries about the same long document, you are likely sending the exact same context over and over. In a standard setup, the engine has to run the expensive to compute the KV cache for that shared history every single time a new message is sent. This wastes massive amounts of GPU compute and memory.
This is where Prefix Caching comes in. Instead of throwing away the KV cache of your prompt after a turn, vLLM retains those computed key-value blocks in memory. If a new incoming prompt shares the exact same starting text—like a system prompt, a long PDF, or the chat history—vLLM grabs the existing KV cache blocks directly from memory. It skips the prefill math entirely for that shared section and starts generating the response immediately.