Every supervised learning model tries to find a pattern in data. But how does it know when it's found a good one? It needs a way to measure its own error. This measure is called a cost function, or a loss function. The entire goal of training is to adjust the model's internal parameters to make the value of this function as low as possible.

For linear regression, where we predict a continuous value, a common choice is the Mean Squared Error (MSE). The idea is simple: for each data point, we measure the difference between the model's prediction and the actual value, square it, and then average these squared differences across all data points.

Classification is different. We're not predicting a number on a line; we're predicting a class, often as a probability. Using MSE here doesn't work well because the math creates a complex error landscape with many local minima, making it hard to find the best solution.

Instead, for logistic regression, we use a cost function called Binary Cross-Entropy, or Log Loss. It measures how far off our predicted probabilities are from the actual class labels (0 or 1). It heavily penalizes a model that is confident about a wrong prediction.

Once we have a cost function, we need a strategy to minimize it. The most common method is Gradient Descent. Imagine you're standing on a foggy hillside and want to get to the bottom of the valley. You can't see the whole landscape, but you can feel the slope right where you're standing. Your best bet is to take a step in the steepest downward direction.

Gradient descent does exactly this. It calculates the gradient of the cost function, which is a vector pointing in the direction of the steepest ascent. To go down, we simply take a step in the opposite direction. We repeat this process, taking small steps downhill until we reach the bottom, where the slope is flat.

The size of each step is controlled by a parameter called the learning rate, often denoted by alpha ($\\( \\alpha \\)$). A small learning rate means slow but steady progress. A large learning rate might get you there faster, but it also risks overshooting the minimum and bouncing around the valley forever. The update rule for a single model weight ($\\( \\theta_j \\)$) looks like this:

Linear regression produces an output that can be any real number. For classification, we need a probability, a value between 0 and 1. How do we map the unbounded output of a linear model to this specific range?

This is where the sigmoid function comes in. It takes any real number as input and squishes it into the range (0, 1), making it perfect for representing probability.

This function acts as a "link" between the linear model and the probability output. The inverse of the sigmoid is called the logit function, which gives logistic regression its name. It takes a probability and maps it back to the full range of real numbers. In essence, logistic regression is just a linear model where the output is passed through the sigmoid function.

Calculating gradients and updating weights one by one for millions of data points is incredibly slow. Modern machine learning relies on linear algebra to perform these operations on all data points at once.

We can represent our entire dataset $X$ as a matrix, where each row is a data point and each column is a feature. Our model parameters can be a vector $\theta$, and the true values a vector $y$. The linear model's predictions for all data points can then be calculated in one go:

This vectorization makes the math cleaner and the computations vastly more efficient, allowing libraries like NumPy or TensorFlow to leverage highly optimized hardware. The gradient descent update rule can also be expressed concisely in matrix form, updating all parameters simultaneously.

Understanding this mathematical machinery isn't just academic. It helps you diagnose when a model is failing, choose the right optimization algorithm, and tune parameters like the learning rate effectively. It's the difference between using a tool and truly understanding it. Now let's review the key ideas.

Time to check your understanding.

By grasping how cost functions, optimization, and link functions work, you've moved past treating these fundamental models as black boxes and can now reason about how they learn.