A neural network learns through a cycle of prediction and correction. Think of it as a student attempting a problem, checking the answer, and then studying their mistake to do better next time. This process has three main steps: the forward pass, loss calculation, and the backward pass.

The forward pass is where the network makes its guess. Input data, like an image or text, travels through the network's layers. Each layer transforms the data using its current weights and an activation function. The final layer produces an output, which is the network's prediction.

Next, the loss calculation measures how wrong the prediction was. This is done by comparing the network's output to the correct answer (the ground truth) using a loss function. The result is a single number, the loss, which quantifies the error. A high loss means a big mistake; a low loss means the network is close to the correct answer.

Finally, the backward pass, powered by an algorithm called backpropagation, calculates how much each weight in the network contributed to the error. It uses calculus, specifically the chain rule, to work backward from the loss and compute the gradient for every weight. This gradient tells us the direction and magnitude of the change needed for each weight to reduce the error.

The choice of loss function depends on the task. For regression tasks, where the goal is to predict a continuous value like a price or temperature, Mean Squared Error (MSE) is common. It calculates the average of the squares of the differences between the predicted and actual values.

For classification tasks, where the goal is to assign a label (e.g., 'cat' or 'dog'), Cross-Entropy Loss is more effective. It measures the difference between two probability distributions: the predicted probabilities and the actual probabilities (where the correct class has a probability of 1 and all others have 0).

Once we have the gradients from backpropagation, we need to update the network's weights. This process is called optimization. The simplest optimizer is Gradient Descent, which adjusts the weights in the opposite direction of the gradient. The size of this adjustment is controlled by the learning rate.

Choosing the right learning rate is critical. Too high, and the training might overshoot the optimal weights and become unstable. Too low, and the training will be painfully slow. Modern optimizers like Adam and RMSProp improve upon basic gradient descent. They dynamically adjust the learning rate for each weight during training, often leading to faster convergence.

As networks get deeper with many layers, two problems can emerge during backpropagation: vanishing and exploding gradients. The chain rule involves multiplying many gradients together. If these gradients are consistently small (less than 1), their product can become astronomically tiny, or vanish. When this happens, the weights in the early layers of the network barely update, and learning stalls.

Clever architectural choices and activation functions help solve these issues. The Rectified Linear Unit (ReLU) activation function, which simply outputs the input if it's positive and zero otherwise, is a popular choice. Because its derivative is either 1 or 0, it's less likely to cause vanishing gradients compared to older functions like sigmoid or tanh, which have derivatives that are always less than 1. Techniques like gradient clipping (capping the gradient at a threshold) and careful weight initialization also help prevent gradients from exploding.

Understanding these mechanics transforms a neural network from a black box into a system of predictable, interconnected parts. By grasping how data flows forward, how errors are measured, and how corrections are propagated backward, you can better diagnose problems and build more effective models.