Supervised learning is fundamentally about finding a pattern. Given a set of inputs, called features ($x$), the model learns to predict an output, called a label ($y$). The core task is to approximate the true, unknown function, $f$, that perfectly maps features to labels. The model's attempt at this is called the hypothesis, $h(x)$.

The gap between the model's prediction, $\hat{y} = h(x)$, and the actual label, $y$, is the error. The entire training process is a systematic effort to make this error as small as possible. To do this, we first need a way to measure it.

A loss function is a mathematical formula that quantifies the cost of a wrong prediction. It takes the model's prediction and the true label and outputs a single number representing the error. A perfect prediction results in a loss of zero; a bad prediction results in a high loss. The choice of loss function depends on the task at hand, primarily whether it's a regression or classification problem.

For classification tasks, where we predict discrete categories like 'spam' or 'not spam', the goal is different. We're not just measuring distance, but the certainty of the prediction. For this, we often use Cross-Entropy Loss (also called Log Loss).

Now that we can measure error, how do we minimize it? Imagine the loss function creates a landscape, with the model's parameters (its internal weights and biases) determining our position. The height of the landscape at any point is the loss. Our goal is to find the lowest valley.

Gradient descent is the algorithm that gets us there. It's an iterative optimization process that works by taking small steps downhill. At any given point, it calculates the gradient, which is a vector pointing in the direction of the steepest ascent. To go down, we simply take a step in the opposite direction of the gradient.

The size of each step is controlled by a parameter called the learning rate. A small learning rate means slow but steady progress. A large learning rate can speed things up, but risks overshooting the minimum and bouncing around the valley. Finding a good learning rate is crucial for training a model effectively.

The mechanics of minimizing loss lead to different kinds of models.

In linear regression, the model learns a straight line (or a hyperplane in higher dimensions) that best fits the data. The goal is to minimize the average squared vertical distance from each data point to that line. The output is a continuous value anywhere along that line.

In logistic regression, despite its name, the goal is classification. The model still calculates a value based on a linear combination of features, but this value is then passed through a special function, the sigmoid or logistic function. This function squishes the output to be between 0 and 1, allowing it to be interpreted as a probability.

The model learns a decision boundary that best separates the classes. For a simple two-feature problem, this boundary is a line. Data points on one side are classified as class 1, and points on the other side are classified as class 0. The model's loss (cross-entropy) is minimized when this boundary does the best job of keeping the different classes on their correct sides.

Ultimately, both methods use the same core principle: define a loss function that measures error, then use an optimization algorithm like gradient descent to tweak the model's internal parameters until that loss is as low as it can be.