Imagine trying to hit a moving target. Now imagine the target is not only moving, but also constantly changing its size and shape. Hitting it would be incredibly difficult. This is the challenge a neural network faces during training.

Each layer in a deep neural network learns from the output of the layer before it. As the network learns, the parameters in the early layers change. These changes alter the distribution of the outputs, or activations, that are fed into the subsequent layers. It's like the target is always shifting.

This phenomenon, where the distribution of layer inputs changes during training, is called internal covariate shift. It makes the learning process unstable and slow. Each layer is constantly trying to adapt to a new input distribution, which makes it hard for the network to converge on a good solution.

Batch normalization adds a step right before the activation function of a layer. For each small batch of training data, it performs two simple operations on the inputs to that layer.

First, it normalizes the inputs. The algorithm calculates the average (mean) and the spread (variance) of the inputs across the mini-batch. Then, it uses these values to center the data around zero and scale it to have a standard deviation of one.

Mathematically, for each input activation $x_i$ in a mini-batch, the normalized value $\hat{x}_i$ is calculated like this:

Here, $\mu_B$ is the mean of the batch, $\sigma^2_B$ is the variance of the batch, and $\epsilon$ is a tiny number added for numerical stability to avoid dividing by zero.

But there's a problem. Forcing the inputs to always have a mean of zero and a standard deviation of one might be too restrictive. For example, for a sigmoid activation function, inputs clustered around zero result in linear behavior, which could reduce the network's expressive power. The network might need inputs with a different distribution.

To solve this, batch normalization introduces two new parameters that are learned during training: a scale parameter, gamma ($\gamma$), and a shift parameter, beta ($\beta$). These parameters allow the network to scale and shift the normalized values to an optimal distribution for the next layer.

The final output, $y_i$, is then passed to the activation function. Because $\gamma$ and $\beta$ are learnable parameters, the network can figure out the best mean and variance for each layer's inputs. It could even learn to undo the normalization completely by setting $\gamma = \sqrt{\sigma^2_B + \epsilon}$ and $\beta = \mu_B$, if that's what's best for learning.

The main advantage of batch normalization is that it stabilizes and accelerates the training process. By keeping the input distributions more consistent, it allows for higher learning rates, which means the network can learn faster.

It also makes the network less sensitive to the initial weights. Without batch norm, a poor initialization can cause gradients to vanish or explode, stopping the learning process. Batch norm helps mitigate these issues, making training more robust.

Finally, batch normalization can have a slight regularization effect. The mean and variance are calculated for each mini-batch, so the normalization applied to any single training example is influenced by the other examples in its batch. This introduces a bit of noise, which can act like a form of regularization, similar to dropout. It can sometimes reduce the need for other regularization techniques.

By smoothing the learning process, batch normalization makes it easier to train deeper and more complex neural networks effectively.

Let's test your understanding of batch normalization.

By normalizing layer inputs, batch norm provides a powerful tool for building more stable and efficient deep learning models.