At first glance, combining Dropout and Batch Normalization seems like a straightforward way to enhance regularization. Both techniques address internal covariate shift and overfitting. However, their interaction creates a subtle but significant mathematical conflict known as variance shift. This disharmony arises from their fundamentally different behaviors during training versus inference.

The core issue is that BatchNorm learns running statistics for mean and variance during training, assuming these statistics will accurately represent the data distribution at inference time. Dropout, however, alters the variance of activations stochastically during training, but becomes a deterministic identity function during evaluation. The running statistics learned by BatchNorm in the presence of stochastic dropout do not match the statistics of the deterministic, full-network activations used for inference. This mismatch leads to a degradation in model performance.

To understand the variance shift, we must first analyze Dropout's mathematical behavior. During training, Dropout applies a stochastic binary mask to its input. Let $x$ be the input to a Dropout layer for a single neuron, $p$ be the dropout probability (the probability of setting a neuron to zero), and $m$ be a random variable from a Bernoulli distribution, $m \sim \text{Bernoulli}(1-p)$. The output $z$ is scaled to preserve the expected value of the activation.

The scaling factor ensures that the expected value of the output remains unchanged, $E[z_{\text{train}}] = x$. However, the variance tells a different story. The variance of the output during training is a function of both the input and the dropout probability.

Since $m$ is a Bernoulli variable, $m^2 = m$. Therefore, $E[m^2 x^2] = x^2 E[m] = x^2(1-p)$. Substituting this back gives us the variance of the activation during training.

During inference (model.eval()), Dropout becomes an identity function, meaning $z_{\text{eval}} = x$. Consequently, the variance is simply $Var(z_{\text{eval}}) = Var(x)$. If we assume for simplicity that the input $x$ is zero-mean and has unit variance, then $Var(z_{\text{eval}}) = 1$, while $Var(z_{\text{train}})$ is amplified by the coefficient derived above.

The problem is now clear. A BatchNorm layer placed after a Dropout layer will calculate its running variance based on the inflated training-time variance. For an input $x$ with $E[x]=0$ and $Var(x)=\sigma^2$, the variance seen by BatchNorm during training is:

BatchNorm dutifully accumulates this inflated variance into its running_var statistic. However, during evaluation, the Dropout layer is inactive, and the variance of the activations passing into BatchNorm is simply $\sigma^2$. The BatchNorm layer then normalizes the inference-time data using the wrong statistic:

This mismatch means the activations are over-shrunk during inference. The numerical scale of the features is distorted, and this effect compounds as the signal propagates through deeper layers, leading to numerical instability and degraded accuracy. The network's performance suffers because the normalization applied at test time does not match the data's true statistical properties.

The conventional wisdom is to place Dropout after BatchNorm. In this ordering (Conv -> BN -> ReLU -> Dropout), BatchNorm normalizes the clean, pre-dropout activations. Its running statistics remain consistent between training and inference. While Dropout still deactivates neurons, it does so after they have been normalized, mitigating the variance shift problem. This simple change in layer ordering can significantly stabilize training and improve final model performance.

Understanding this subtle interaction is key to building robust and high-performing deep learning architectures.