No history yet

Regression Math Intuition

Finding the Best Fit

In supervised learning, our goal is to teach a model to find patterns in data. For linear regression, this means finding the straight line that best represents the relationship between our input features and the output. But what does 'best' actually mean?

The most common method for finding this line is called Ordinary Least Squares (OLS). The idea is simple but powerful: for every data point, we measure the vertical distance between the point and our line. This distance is called a residual. Since some residuals will be positive (above the line) and some negative (below it), we square them to make all the values positive. This also penalises larger errors more heavily.

OLS then adds up all these squared distances. The 'best fit' line is the one that makes this total sum as small as possible. It's the line that minimises the sum of squared residuals.

minimize∑i=1n(yi−y^i)2\text{minimize} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

The model adjusts its parameters, the slope and the intercept, to shrink this sum. This process is often handled by an optimisation algorithm like Gradient Descent, which iteratively tweaks the parameters to find the values that produce the lowest possible error. The final slope and intercept define our best-fit line.

From Lines to Probabilities

Linear regression is great for predicting continuous values like a house price. But what if we want to predict a binary outcome, like whether a patient has a disease or not? A straight line doesn't work well here, as it can predict values far above 1 or below 0, which makes no sense for a probability.

This is where logistic regression comes in. It takes the output of a linear equation and 'squashes' it into a range between 0 and 1 using a special function called the Sigmoid function. The result can be interpreted as a probability.

σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}

The S-shaped curve of the Sigmoid function provides a natural way to model the probability of a binary event. We can set a threshold, typically 0.5, to make a final decision. If the output probability is greater than 0.5, we predict one class (e.g., 'Yes'), and if it's less, we predict the other ('No'). This threshold forms the decision boundary that separates the two classes.

Measuring the Right Error

Since we're now dealing with probabilities instead of continuous values, the (MSE) we used in linear regression is no longer the best tool. Using MSE for a classification problem creates a cost function with many local minima, making it hard for Gradient Descent to find the true minimum error.

Instead, logistic regression uses a cost function called Log Loss, also known as Binary Cross-Entropy. Log Loss heavily penalises the model when it is confident about a wrong prediction. For example, if the correct answer is 1, and the model predicts 0.01, the penalty is huge. If it predicts 0.99, the penalty is very small.

In short: linear regression minimises the distance between points and a line, while logistic regression minimises the uncertainty in its probabilistic predictions.

J(θ)=−1m∑i=1m[y(i)log⁡(hθ(x(i)))+(1−y(i))log⁡(1−hθ(x(i)))]J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} [y^{(i)} \log(h_\theta(x^{(i)})) + (1-y^{(i)}) \log(1-h_\theta(x^{(i)}))]

Both OLS and Log Loss provide a way to measure our model's error. With that measure in hand, we can use an optimiser to systematically improve the model's parameters until it performs as well as possible on the training data.

Time to test your understanding of these core concepts.

Quiz Questions 1/6

What is the primary goal of the Ordinary Least Squares (OLS) method in linear regression?

Quiz Questions 2/6

A data scientist wants to build a model to predict whether an email is spam or not spam based on its content. Which type of model is more appropriate for this task?

Understanding how these models measure and minimise error is the key to knowing how they learn from data.