In linear regression, our goal is to model the relationship between variables by fitting a straight line to a set of data points. But among all the possible lines we could draw, how do we find the one that fits best? The answer lies in minimizing the errors our line makes.

Each error, or residual, is the vertical distance between an actual data point and the point predicted by our line. We want to make the total of these errors as small as possible. Simply summing them up won't work, because positive and negative errors would cancel each other out. Instead, we square each residual before adding them together. This ensures all errors are positive and penalizes larger errors more heavily.

The Ordinary Least Squares (OLS) method finds the specific values for the intercept (eta_0) and slope (eta_1) that make the RSS as small as possible.

To find the values of $\beta_0$ and $\beta_1$ that minimize the RSS, we use calculus. We take the partial derivative of the RSS function with respect to each coefficient and set the results to zero. This process finds the point where the function's slope is flat, which corresponds to its minimum value.

Solving this system gives us what are known as the Normal Equations. These equations provide the formulas for the optimal intercept and slope.

Once we have the slope, finding the intercept is straightforward. The regression line is guaranteed to pass through the point representing the mean of both variables, $(\bar{x}, \bar{y})$.

We can also understand OLS from a geometric perspective using linear algebra. Imagine our response variable, $y$, as a vector in a multi-dimensional space. Our predictor variables (in simple regression, just $x$ and a vector of ones for the intercept) form a subspace, which you can visualize as a plane within that larger space.

The OLS regression doesn't try to make the $\hat{y}$ vector equal to the $y$ vector, which is usually impossible. Instead, it finds the projection of the $y$ vector onto the subspace spanned by the predictors. This projection, $\hat{y}$, is the vector of fitted values. It's the point on the predictor plane that is closest to our actual data vector, $y$.

The residual vector, $e = y - \hat{y}$, is the difference between the actual outcomes and our predictions. Geometrically, this vector is orthogonal (perpendicular) to the predictor subspace. This orthogonality is the geometric equivalent of solving the Normal Equations and minimizing the RSS.

Understanding OLS through both its algebraic minimization and its geometric interpretation provides a robust foundation for building and interpreting regression models.