Machine learning models aren't magic. They are mathematical machines designed to find patterns. To understand how they learn, we first need to speak their language: linear algebra. Let's imagine we're trying to predict a house's price. We might collect data on two key features: its size in square feet and the number of bedrooms.

We can organize this information neatly. Each row represents a different house, and each column represents a feature. This creates a matrix, which we'll call $X$. If we have data for four houses, our feature matrix might look like this:

The corresponding sale prices for these houses are stored in a separate vector, usually called $y$.

A simple linear regression model makes a prediction by giving each feature a weight (a coefficient) and adding a bias term. For our two features, the prediction, $\hat{y}$, for a single house is calculated as:

$\hat{y} = w_1 x_1 + w_2 x_2 + b$

Here, $x_1$ is the square footage and $x_2$ is the number of bedrooms. The model's job is to find the best possible values for the weights ($w_1$, $w_2$) and the bias ($b$) to make its predictions as close to the actual prices in $y$ as possible. But how does it know what

A model starts by guessing some initial weights and a bias. Unsurprisingly, its first predictions will be far from the actual prices. We need a way to measure the total error across all our data points. This is where the cost function, or loss function, comes in.

One of the most common cost functions is the Mean Squared Error (MSE). It calculates the difference between the actual price ($y_i$) and the predicted price ($\hat{y}_i$) for each house, squares it to make sure the error is positive, and then averages these squared errors across all $m$ houses in our dataset.

The goal of training a model is to find the weights and bias that make this cost function as small as possible. Think of the cost function as a landscape with hills and valleys. Our goal is to find the lowest point in the deepest valley. To do that, we need a guide, and that guide is calculus.

The algorithm that finds this lowest point is called Gradient Descents. It works by taking small, iterative steps in the direction of the steepest descent—the direction that reduces the cost the most.

How do we find that direction? By using partial derivativess. A partial derivative tells us the slope of the cost function with respect to just one of the parameters (like $w_1$), while holding all the others constant. The collection of all these partial derivatives forms a vector called the gradient. The gradient always points directly uphill, in the direction of the steepest increase in cost.

Since we want to go downhill, we simply take a step in the opposite direction of the gradient. We update each weight by subtracting a small fraction of its corresponding partial derivative. This small fraction is a parameter called the learning rate, denoted by $\alpha$. It controls how big of a step we take.

We repeat this process for each weight and the bias, over and over, for many iterations. With each step, the model's predictions get slightly better, and the value of the cost function gets smaller. Eventually, the steps become tiny, and we converge on a set of weights and a bias that minimizes our cost function. At this point, the model has "learned" the best possible linear relationship between the features and the target.