Imagine you're on a mountain in a dense fog. You know the lowest point in the valley is somewhere below, but you can't see it. How do you get there? You'd probably feel the ground with your feet and take a step in the steepest downward direction you can find. Repeat this process, and eventually, you'll reach the bottom.

This is exactly what gradient descent does. It's an optimization algorithm that navigates the "landscape" of a model's loss function to find the lowest point, which represents the minimum error. The key is knowing which way is "downhill."

In machine learning, "downhill" is the direction that makes our model's predictions less wrong. We find this direction using the gradient of the loss function. You already know that partial derivatives measure how a tiny change in one variable affects the function's output. The gradient simply bundles all of these partial derivatives—one for each model parameter (weight and bias)—into a single vector.

Since we want to minimize the loss, we need to go downhill. So, we move in the direction opposite to the gradient. This is the core mechanic of the update rule: we take our current parameters and nudge them in the negative direction of the gradient. This ensures each step we take is a step toward a lower error.

The process of updating the model's parameters is an iterative one. At each step, we adjust the weights and biases using a simple but powerful formula. For any given parameter $\theta$ (which could be a weight $w$ or a bias $b$), the update looks like this:

The minus sign is crucial. It flips the gradient (the direction of steepest ascent) into the direction of steepest descent. The learning rate $\alpha$ acts as a throttle. Too large, and we might overshoot the minimum; too small, and training will take forever. Finding the right learning rate is a key part of training a model effectively.

A typical neural network has thousands, or even millions, of parameters. Calculating each partial derivative and updating each parameter one-by-one would be incredibly inefficient. Instead, we use vectorized operations.

The weights of a model can be represented as a matrix, $\mathbf{W}$, and the biases as a vector, $\mathbf{b}$. The gradient of the loss function with respect to all weights, $\nabla_{\mathbf{W}} J$, is a matrix of the same dimensions. Likewise, the gradient with respect to all biases, $\nabla_{\mathbf{b}} J$, is a vector. This allows us to perform the updates for all parameters simultaneously.

This parallel update is extremely efficient, especially when run on hardware designed for matrix operations like GPUs (Graphics Processing Units). Every step in the training process applies this rule, moving the model's parameters closer to a configuration that minimizes the loss function and, hopefully, makes accurate predictions.

Let's test your understanding of how gradient descent tunes a model's parameters.

The update rule is the engine of learning in many machine learning models. By repeatedly calculating the gradient and taking a small step in the opposite direction, the model refines its parameters to make better and better predictions.