Training large language models demands immense computational power. A single GPU can only hold so much data and perform so many calculations. The obvious solution is to use more GPUs. The simplest way to do this is with data parallelism, where we split our dataset across multiple processors and train a copy of the model on each one. The goal is to get a single, well-trained model, but faster than a single GPU could ever manage.

But as with most things in computing, there is a simple way and there is a smart way. The distinction between them is crucial for scaling up effectively.

Imagine a team leader managing several workers. The leader takes a big stack of paperwork (the data batch), divides it up, and gives a small pile to each worker (GPU). Each worker processes their pile and calculates their results (gradients). Then, they all send their results back to the team leader. The leader single-handedly collects all the results, averages them, updates the master plan (the model weights), and then distributes the new plan back to every worker for the next round. This is how PyTorch's DataParallel (DP) module works.

In DP, one GPU acts as the master. It splits the data, sends model replicas to other GPUs, and, most importantly, gathers all the calculated gradients back to itself for the final update. This creates a significant bottleneck. The master GPU and its communication channel are overwhelmed, and all other GPUs sit idle waiting for the updated model. This is a form of single-process, multi-threaded parallelism, which introduces its own set of problems in Python due to the Global Interpreter Lock (GIL).

DistributedDataParallel (DDP) takes a more democratic approach. Instead of a single leader, each GPU runs its own independent process. There is no master GPU. Each process is a peer.

This solves the bottleneck problem. At the start, each process initializes its own identical copy of the model. To ensure each process gets a unique slice of the data without overlap, a DistributedSampler is used. It divides the dataset so that each process receives its own exclusive mini-batch for each training step.

After each process computes its gradients during the backward pass, they don't send them to a single master. Instead, they use a communication primitive called All-Reduce to average the gradients across all GPUs simultaneously. Each GPU ends up with the exact same averaged gradient, and each process updates its local model copy independently. Because all processes start with the same weights and receive the same averaged updates, the models remain perfectly synchronized throughout training.

The All-Reduce operation is efficient, but it still takes time for GPUs to communicate over the network. If we waited for every single gradient to be calculated before starting the All-Reduce, GPUs would still spend time idling. DDP employs a clever trick to hide this communication latency: gradient bucketing.

As gradients are computed during the backward pass, starting from the final layers of the model and moving to the initial layers, DDP collects them into buckets. Once a bucket is full, DDP immediately kicks off an asynchronous All-Reduce operation for the gradients in that bucket. While the network is busy transmitting that bucket's data, the CPU and GPU are already busy computing the gradients for the next layers. This overlaps computation with communication.

By the time the backward pass finishes calculating the gradients for the earliest layers, the All-Reduce for the initial buckets is often already complete. This pipeline of computation and communication dramatically reduces idle time and makes training much more efficient.

This process-based, bottleneck-free approach with overlapping communication is why DDP is the industry standard for multi-GPU training. It provides the foundation for scaling training across not just multiple GPUs in one machine, but across many machines in a cluster.