Large language models are powerful, but they are also computationally expensive. Getting a response from an LLM isn't like retrieving data from a database; it's a complex process of generation that requires a massive amount of processing power, usually from specialized hardware like GPUs.

When you build an application that relies on an LLM, users expect quick, almost instant, responses. This creates a fundamental tension. On one hand, you have a slow, resource-hungry model. On the other, you have users who demand speed. Serving a single user is manageable, but serving thousands or millions of concurrent users turns this tension into a major engineering challenge.

Think of an LLM running on a GPU like a master chef in a high-end kitchen. This chef can cook incredible dishes, but they work meticulously and take their time. If you only send one order to the kitchen at a time, the chef will spend a lot of time waiting for the next ticket, even though they're capable of working on several dishes at once. Your expensive kitchen—and your master chef—will be mostly idle. This is the essence of underutilization.

A GPU is most efficient when it's processing a large batch of data simultaneously. Sending a single user's request to a powerful GPU is like using a massive freight train to deliver a single package. The train has the capacity for much more, and running it for just one small item is wasteful. The key to efficiency is keeping the GPU busy with as much parallel work as possible.

This is directly related to latency, which is the delay a user experiences while waiting for a response. In the context of LLMs, we often think about two kinds of latency:

1. Time to First Token: How long it takes for the first word of the model's response to appear. A long delay here makes an application feel unresponsive.
2. Time per Output Token: How quickly the rest of the words are generated after the first one. Slow generation here makes the model feel sluggish, as if it's struggling to think.

The challenge is that the most obvious way to reduce latency—processing one request at a time as fast as possible—leads to poor GPU utilization. Conversely, waiting to collect a large batch of requests to improve GPU utilization can increase the waiting time for individual users. Optimizing LLM inference is all about navigating this trade-off.

To solve these problems, engineers use a few core strategies. We won't dive deep into the mechanics here, but it's important to understand the role each one plays.

Returning to our chef analogy, batching is like giving the chef all the steak orders at once. They can sear them all on the grill at the same time, rather than cooking one, cleaning the grill, and then starting the next.

These three strategies—batching, routing, and caching—form the foundation of LLM inference optimization. By cleverly combining them, it's possible to build services that are both fast for the user and efficient for the business.