Every model makes predictions, but how do we know if those predictions are any good? We need a way to measure the model's error. In linear regression, the most common way to do this is with a cost function, specifically the Mean Squared Error (MSE). The goal is to find the line that makes this error as small as possible.

Imagine you have a set of data points. A linear model draws a straight line through them. The error for a single point is the vertical distance between the actual data point and the point predicted by the line. This distance is called a residual. To calculate the total error for the whole dataset, we can't just add up the residuals, because some will be positive and some negative, and they might cancel each other out. To solve this, we square each residual first. This makes all the error values positive. Finally, we take the average of all these squared residuals. That's the Mean Squared Error.

The entire process of training a linear regression model is about finding the specific values for the model's parameters, its slope and intercept, that result in the lowest possible MSE.

Minimizing the MSE is like standing on a hillside in a thick fog and trying to find the lowest point in the valley. You can't see the bottom, but you can feel which way is downhill from where you are. You take a step in that direction, check the slope again, and take another step. You repeat this until you can't go any further down. This is the core idea behind Gradient Descent an iterative optimization algorithm used to find the minimum of a function.

In the context of linear regression, the 'hillside' is the cost function, and the 'location' is defined by the current values of our model's parameters (let's call them $\theta_0$ for the intercept and $\theta_1$ for the slope). The 'downhill direction' is given by the partial derivative of the cost function with respect to each parameter. We update the parameters by taking a small step in the negative direction of this gradient. The size of that step is controlled by a value called the learning rate, symbolized by $\alpha$ (alpha).

A small learning rate means tiny, slow steps, but it's unlikely to overshoot the minimum. A large learning rate gets you there faster, but you risk overshooting the bottom and bouncing around, or even diverging completely. The process is repeated until the change in the cost function becomes negligible, meaning we've reached the bottom.

Gradient descent is iterative, but what if there was a way to solve for the optimal parameters directly, in one mathematical step? There is, and it's called the Normal Equation It's an analytical approach that comes from the calculus of finding a minimum: you take the derivative of the cost function, set it to zero, and solve. When applied to the MSE cost function in matrix form, this derivation gives us a beautiful, compact formula.

So why use gradient descent if the Normal Equation can solve the problem in one shot? The trade-off is computational complexity. Calculating the inverse of a matrix, $(X^T X)^{-1}$, is a very expensive operation. For a dataset with $m$ features, this is roughly an $O(m^3)$ operation. If you have thousands of features, this becomes computationally infeasible.

Gradient descent, on the other hand, works well even with millions of features. The Normal Equation is great for smaller datasets where the number of features is in the hundreds or low thousands, but for the massive datasets common in modern machine learning, gradient descent is the go-to method.

Linear regression is powerful, but its results are only reliable if certain assumptions about the data hold true. Violating these assumptions can lead to misleading or completely incorrect conclusions.

Checking these assumptions, often through visual inspection of residual plots, is a crucial step in the modeling process. It ensures that the mathematical machinery of linear regression is being applied in a situation where it can actually succeed.