In a multi-tenant GPU cluster, predicting performance under load isn't just about raw hardware power. It's a problem of stochastic processes. We can model the system using queuing theory, starting with a foundational principle: Little's Law.

This equation provides a surprisingly robust first-order model for system load. For an inference service, an increase in the arrival rate (λ) or the average processing time (W) directly translates to a linear increase in the number of concurrent requests (L) the system must handle. If L exceeds the system's capacity—the total number of requests that can be processed concurrently—queues will form, and wait times will grow non-linearly. However, this simple model assumes a stable, average service time, which is an oversimplification for LLMs.

A more accurate model is the M/G/1 queue, which assumes Markovian (random) arrivals, a General (arbitrary) service time distribution, and a single server. The 'G' is critical for LLMs because inference service time is not monolithic; it's distinctly bimodal, driven by two separate phases: prefill and decode.

1. Prefill Phase: This initial step processes the entire input prompt at once to generate the attention keys and values for the first output token. It's compute-bound and highly parallelizable across GPU cores. Its duration is proportional to the prompt length, $N_{prompt}$.
2. Decode Phase: This iterative phase generates subsequent tokens one by one. Each step is auto-regressive, meaning it depends on the previously generated token. This makes it memory-bandwidth-bound. The total time is proportional to the number of generated tokens, $N_{gen}$.

This bimodal nature breaks the assumptions of simpler models and is why schedulers often treat prefill and decode as separate task types. Fair-batching schedulers, for instance, try to prevent long, compute-heavy prefill jobs from starving the queue of quick decode steps, which are necessary to maintain a high token-generation rate (throughput) for already-running requests.

For models too large to fit on a single GPU, inference requires gang scheduling across a group of accelerators. Here, the performance of the entire group is dictated by its slowest member. This phenomenon introduces a 'straggler tax' on tail latency. The completion time for any synchronized step, such as an all-reduce operation in tensor parallelism, is not the average worker time, but the maximum time.

This has profound implications for P99 and P99.9 latency. Stragglers can arise from numerous sources: network jitter, OS noise, thermal throttling, or transient hardware faults. In a large cluster, the probability of at least one worker experiencing a delay in a given cycle increases with the size of the gang. As a result, tail latency doesn't scale linearly with the average worker latency; it scales with the tail of the maximum of N independent random variables. This makes tail latency reduction a problem of variance reduction and straggler mitigation, not just optimizing the average case.

Managing inference workloads requires balancing latency, throughput, and utilization by understanding the underlying mathematical principles that govern these complex, distributed systems.