A linear regression model tries to find the best straight line that cuts through a scatter plot of data. But what does 'best' actually mean? It means finding a line that minimises the overall error between the predicted values (the points on the line) and the actual values (the real data points).

This error for a single data point is called a residual. It's the vertical distance between the data point and the regression line. To find the 'best' line, we can't just add up all the residuals. Positive errors would cancel out negative ones, giving us a misleading picture. Instead, we need a way to measure the total magnitude of the error, regardless of direction.

This is where a cost function comes in. It's a formula that calculates a single number representing the total error for the model. For linear regression, the most common cost function is the Mean Squared Error (MSE).

The idea is simple: for each data point, you calculate the residual, square it, and then find the average of all these squared residuals. Squaring the residuals accomplishes two things: it makes all the error values positive, and it penalises larger errors more heavily than smaller ones. A mistake of 2 units becomes an error of 4, while a mistake of 10 becomes an error of 100.

The goal of training the model is to find the specific slope and intercept for our line that makes this MSE value as small as possible. There are two primary ways to do this: an iterative search and a direct calculation.

Imagine the cost function as a three-dimensional landscape, like a valley. The two horizontal directions represent the possible values for the slope and intercept of our line, and the vertical direction represents the MSE. Our goal is to find the very bottom of this valley, the point with the lowest possible error. Gradient descent is like walking down into that valley.

The algorithm starts with random values for the slope and intercept. It then calculates the gradient, which is the steepness and direction of the slope at its current position in the valley. By taking a small step in the opposite direction of the gradient (downhill), it moves closer to the minimum. This process is repeated until the steps become tiny, meaning we've arrived at or very near the bottom. The size of these steps is controlled by a parameter called the learning rate a crucial factor in how efficiently the model trains.

The second method is Ordinary Least Squares (OLS). Instead of searching for the best parameters, OLS solves for them directly using linear algebra. It's like having a map that tells you the exact coordinates of the valley's bottom from the start. For datasets that aren't excessively large, OLS is computationally faster and provides a precise answer in one go. However, as the number of features in your data grows, the calculations can become very demanding, making gradient descent a more practical choice.

Linear regression works best when a few key assumptions about the data are met. One is homoscedasticity, which is a fancy way of saying the variance of the residuals should be constant across all levels of the independent variables. In other words, the model's predictions should be equally bad (or good) across the entire range of data.

Another assumption is that the residuals are normally distributed. This means that if you made a histogram of all the errors, it should look like a bell curve centred at zero. Violating these assumptions doesn't mean the model is useless, but it can make the coefficient estimates less reliable.

These core mechanics form the basis for how a simple, yet powerful, linear regression model learns from data to make predictions.