When you add more processors to a problem, you expect it to get faster. But by how much? It turns out there are two fundamental ways to look at this question, each with its own law. The first, and perhaps more famous, is Amdahl's Law A. It operates on a simple, powerful assumption: every program has a portion that must run sequentially, one step at a time, and a portion that can be broken up and run in parallel.

The serial portion, no matter how small, becomes a bottleneck. Even if you have a thousand processors, they all have to wait for that one serial step to finish. Amdahl's Law shows that there's a hard limit to your speedup, dictated entirely by that serial fraction. If 10% of your code is serial ($1-p = 0.1$), you can never get more than a 10x speedup, even with infinite processors.

This seems pessimistic. But in the 1980s, researchers noticed something interesting. When they got access to bigger computers, they didn't just run the same old problems faster. They ran bigger problems. They increased the resolution of their weather simulations or trained larger neural networks. The problem size scaled with the number of processors.

This observation led to Gustafson's Law A. It reframes the question. Instead of asking "How fast can I solve this fixed problem?", it asks, "How much more work can I do in the same amount of time?" Gustafson's model assumes the parallel part of the workload scales with the number of processors, while the serial part stays constant.

Under this model, the speedup is nearly linear. If you have 1000 processors, you can do almost 1000 times the work. The tiny serial part becomes insignificant as the parallel workload grows to dominate the total runtime.

These two laws give us two different ways to measure performance, known as strong and weak scaling.

Strong scaling measures how much faster you can solve a fixed-size problem by adding more processors. It's a test of latency reduction. This is the world of Amdahl's Law. You'd care about strong scaling if you need an answer, any answer, as quickly as possible, like in real-time fraud detection.

Weak scaling measures how much you can increase the problem size while keeping the run time constant as you add more processors. It's a test of throughput. This is the world of Gustafson's Law. You'd care about weak scaling when you want to increase the fidelity or size of a task, like rendering a higher-resolution movie or running a more detailed climate simulation.

So which model matters more for today's massive, multi-thousand node clusters? For most large-scale scientific computing and AI training, weak scaling is the more relevant paradigm. The goal is rarely to train a 2018-era AI model faster; it's to train a massive 2025-era model in a reasonable amount of time.

When planning for a new cluster, engineers use these models predictively. First, they must identify the serial bottleneck A. This could be anything from data loading from a single disk, a master process that aggregates results, or a communication step that can't be parallelized. By profiling a smaller run, they can estimate the parallel fraction, $p$.

With an estimate for $p$, they can then apply Gustafson's Law to determine how many nodes are needed to tackle a target problem size within a specific time budget. For example, if a simulation for a 1-billion-particle system takes 24 hours on 1,000 nodes, they can predict the resources needed for a 10-billion-particle system.

However, Amdahl's Law still serves as a crucial check. Even in a weak scaling scenario, the serial fraction doesn't disappear. It sets a limit on how much work each individual processor must wait on. Understanding this helps architects optimize the serial parts of the code, as even small improvements there can unlock massive gains across the entire cluster.

Choosing the right scaling metric depends entirely on your goal. Are you trying to get an answer faster, or are you trying to ask a bigger question?