At its core, a neural network learns by adjusting its internal parameters to get better at a task. These parameters, known as weights and biases, are the knobs the network turns to fine-tune its output. The goal is to minimize error, and to do that, the network needs a way to measure how wrong it is. This is the job of a loss function.

A loss function calculates a single number, the 'loss,' that represents the difference between the network's prediction and the actual correct answer. A high loss means the prediction was far off; a low loss means it was close. The entire training process is a systematic effort to make this loss value as small as possible.

If the loss function defines a landscape of error, with mountains of high loss and valleys of low loss, then gradient descent is the algorithm that navigates this terrain. It's an optimization process that finds the bottom of a valley—the point of minimum loss.

The 'gradient' is a vector that points in the direction of the steepest ascent on the loss landscape. To decrease the loss, we simply take a step in the opposite direction of the gradient. The size of this step is controlled by a parameter called the learning rate. A well-chosen learning rate is crucial: too small, and the model learns excruciatingly slowly; too large, and it might overshoot the minimum entirely, causing the loss to bounce around or even increase.

Calculating the gradient requires using the entire dataset, which can be computationally expensive. To manage this, different variants of gradient descent are used.

Gradient descent tells us how to update the weights (by moving opposite the gradient), but it doesn't tell us how to calculate that gradient. This is where backpropagation comes in.

Backpropagation is the algorithm that computes the gradient of the loss function with respect to each weight and bias in the network. It does this by applying the chain rule from calculus, starting from the final layer and moving backward through the network. It calculates the error contribution of each neuron and propagates this information back, allowing each weight to be updated proportionally to how much it contributed to the overall error.

The chain rule allows us to calculate the derivative of a composite function. Since a neural network is essentially a giant, nested function, the chain rule is the perfect tool for the job. For a simple composition of functions $y = f(g(x))$, the derivative of $y$ with respect to $x$ is:

If a neural network were just a series of linear operations, it would be no more powerful than a single linear model, regardless of how many layers it had. The network needs a way to model complex, non-linear relationships in data—like the curve of a cat's ear or the subtle patterns in financial markets.

This is the role of activation functions. After each layer's linear calculation (multiplying inputs by weights and adding a bias), an activation function is applied. It introduces non-linearity, allowing the network to learn and approximate any arbitrarily complex function.

Common choices include:

• Sigmoid: Compresses numbers into a (0, 1) range, which is useful for binary classification outputs. It has largely been replaced in hidden layers due to issues with gradients.
• ReLU (Rectified Linear Unit): A simple and effective function. It outputs the input directly if it's positive, and zero otherwise. It's the most common activation function for hidden layers.
• Softmax: Used in the output layer for multi-class classification. It converts a vector of raw scores into a probability distribution, where each value is between 0 and 1 and all values sum to 1.

Thankfully, modern deep learning frameworks like PyTorch and TensorFlow automate these calculus-heavy processes. They build a computational graph of all the operations, allowing them to automatically calculate gradients and apply updates via backpropagation, freeing practitioners to focus on model architecture and design.

With these mechanics in place—a loss function to measure error, gradient descent to find the minimum, backpropagation to calculate the updates, and activation functions to model complexity—a neural network has everything it needs to learn from data.