To normalize the scores to have a zero mean and unit variance.

To scale the dot products by the square root of the key dimension to stabilize gradients.

To prevent tokens from attending to future tokens in the sequence, ensuring the autoregressive property.

To randomly drop out certain attention connections to prevent overfitting.

The output projection tensor after the attention mechanism.

The Query (Q), Key (K), and Value (V) tensors.

The Value (V) tensor only.

The Query (Q) and Key (K) tensors only.

LayerNorm -> Multi-Head Attention -> Residual Add -> LayerNorm -> FFN -> Residual Add

Residual Add -> LayerNorm -> Multi-Head Attention -> FFN -> Residual Add

Multi-Head Attention -> FFN -> LayerNorm -> Residual Add

Multi-Head Attention -> Residual Add -> LayerNorm -> FFN -> Residual Add -> LayerNorm

It accumulates gradients over multiple batches to simulate a larger batch size.

It scales the loss before the backward pass to prevent gradients computed in float16 from underflowing to zero.

It applies weight decay only to weights and not to bias or normalization parameters.

It clips gradients to a maximum norm to prevent exploding gradients.

O(1), because the previous state is fully captured in the cache.

O(N), because only the new token's Query needs to be compared against the N-1 cached Keys.

O(N^2), because the new token must attend to all previous N-1 tokens.

DDP shards the model weights across GPUs, while FSDP replicates the entire model on each GPU.

DDP replicates the full model, gradients, and optimizer states on each GPU, whereas FSDP shards the model weights, gradients, and optimizer states across GPUs.

DDP is designed for inference, while FSDP is designed for training.

DDP processes the entire batch on a single GPU, while FSDP splits the batch across GPUs.

torch.einsum('bhsd,bhsd->bhss', q, k)

torch.einsum('bhsd,bhd->bhs', q, k)

torch.einsum('bhs,bhd->bhsd', q, k)

torch.einsum('bhqd,bhkd->bhqk', q, k)

True

False

To aggregate information from different sequence positions.

To reduce the dimensionality of the token embeddings.

To normalize the outputs of the attention layer.

To apply a non-linear transformation to each token's representation independently.

It incorporates Nesterov momentum for faster convergence.

It computes second-order gradients for more precise updates.

It uses a cosine annealing schedule for the learning rate.

It decouples weight decay from the gradient-based update, applying it directly to the weights.

Top-K is used for training, while Top-P is used for inference.

Top-K sets a fixed vocabulary size (K) to sample from, while Top-P creates a dynamic vocabulary by selecting the smallest set of tokens whose cumulative probability exceeds a threshold P.

Top-P always selects the single most likely token, whereas Top-K introduces randomness.

Top-P selects the K most probable tokens, while Top-K selects a random number of tokens.

To enable cross-attention between different modalities like text and images.

To improve the numerical stability of the softmax calculation.

To automatically shard the attention computation across multiple GPUs.

To avoid the materialization of the large N x N attention matrix in GPU memory, reducing memory usage from O(N^2) to O(N).