Data Parallelism (DP) is a straightforward way to speed up model training. You take a mini-batch of data, split it across multiple GPUs, and give each GPU a copy of the model. Each GPU processes its slice of data, computes gradients, and sends them back to a main GPU, typically gpu:0. This master node then averages the gradients, updates its model weights, and broadcasts the new weights back to all the other GPUs. It's a simple, effective strategy for smaller models.

However, this simplicity comes with a cost. As models grow, the master node in DP becomes a significant bottleneck. It has to handle all incoming gradients and all outgoing model updates, creating a traffic jam that slows down the entire training process. This synchronization overhead can negate the benefits of using multiple GPUs.

A major culprit is Python's Global Interpreter Lock (GIL). DP operates within a single Python process, using multiple threads to manage the GPUs. The GIL allows only one thread to execute Python bytecode at a time, even on a multi-core machine. This prevents true parallel execution and forces the gradient aggregation and model update steps into a largely sequential process on the master node.

To overcome these limitations, Distributed Data Parallel (DDP) was developed. Instead of using one process and multiple threads, DDP spawns a separate, independent process for each GPU. This simple architectural change has a profound impact: since each process has its own Python interpreter and memory space, the Global Interpreter Lock is no longer a factor. True parallelism is achieved.

With DDP, there is no master node. Each process works on its own data shard, computes gradients, and then communicates directly with the other processes to synchronize those gradients. This decentralized approach eliminates the single-point bottleneck of standard DP.

The key to DDP's efficiency is the Ring All-Reduce algorithm, a clever method for summing values across multiple nodes without a central server. This is typically handled by a highly optimized backend like NCCL (NVIDIA Collective Communications Library), which is deeply integrated into frameworks like PyTorch.

DDP's performance is further enhanced by overlapping communication and computation. During the backward pass, gradients are not computed all at once. They become available layer by layer, starting from the output layer and moving backward.

DDP takes advantage of this by using a technique called gradient bucketing. Instead of waiting for all gradients to be ready, it groups them into 'buckets' as they are computed. As soon as a bucket is full, DDP kicks off the Ring All-Reduce operation for that bucket's gradients while the next layers are still performing their backward pass.

Despite its advantages, DDP still has a fundamental limitation: each GPU must hold a full copy of the model parameters, gradients, and optimizer states. This means that if a single model is too large to fit into one GPU's memory, DDP alone is not enough. For those massive models, we need to turn to techniques that shard the model itself, such as model parallelism or more advanced data-parallel strategies.