Linear Gradient Optimization

The Quest for the Best Fit

In linear regression, our goal is to draw a line that best fits our data. But what does "best" actually mean? We need a way to measure how wrong our model's predictions are compared to the actual data. This measure is called a cost function or loss function.

For regression problems, a common and effective cost function is the Mean Squared Error (MSE). It works by taking the difference between the predicted value and the actual value for each data point, squaring that difference, and then calculating the average of all these squared differences. Squaring the error ensures that we treat underestimates and overestimates equally, and it penalizes larger errors more heavily.

J(m, b) = \frac{1}{N} \sum_{i=1}^{N} (y_i - (mx_i + b))^2

Our goal is to find the specific values of the slope ( $m$ ) and intercept ( $b$ ) that make the MSE as small as possible. This turns our modeling problem into an optimization problem: we are searching for the minimum point of the cost function.

Following the Slope Downhill

Imagine you're standing on a foggy hillside and want to get to the lowest point in the valley. You can't see the whole landscape, but you can feel the steepness of the ground right where you are. The most straightforward strategy is to take a step in the steepest downhill direction. You repeat this process, and eventually, you'll arrive at the bottom. This is the core intuition behind gradient descent an iterative optimization algorithm that finds the minimum of a function.

In the context of our cost function, the "steepness" is given by the gradient. The gradient is a vector of partial derivatives; it points in the direction of the steepest ascent. To go downhill, we simply move in the opposite direction of the gradient. We update our parameters, $m$ and $b$ , by taking our current values and subtracting a small fraction of the gradient.

\begin{aligned} m &:= m - \alpha \frac{\partial J}{\partial m} \\ b &:= b - \alpha \frac{\partial J}{\partial b} \end{aligned}

Setting the Right Pace

The learning rate $\alpha$ is a critical hyperparameter. It determines how big of a step we take with each iteration. Choosing the right learning rate is crucial for successful training.

A learning rate that is too small will cause the algorithm to converge very slowly. A learning rate that is too large might cause it to overshoot the minimum and fail to converge at all, or even diverge.

Finding a good learning rate often involves some experimentation. A common practice is to start with a small value (like 0.01) and gradually increase it, monitoring the cost at each iteration. If the cost consistently decreases, you're on the right track. If it bounces around or increases, your learning rate is likely too high.

From Theory to Practice

Let's apply these concepts to our house price prediction example. We'll use batch gradient descent, which means we calculate the gradients using our entire dataset in each iteration. For our MSE cost function, the partial derivatives are:

\begin{aligned} \frac{\partial J}{\partial m} &= -\frac{2}{N} \sum_{i=1}^{N} x_i(y_i - (mx_i + b)) \\ \frac{\partial J}{\partial b} &= -\frac{2}{N} \sum_{i=1}^{N} (y_i - (mx_i + b)) \end{aligned}

The training process involves initializing $m$ and $b$ (often to 0), and then repeating the update step for a set number of iterations or until the cost stops decreasing significantly. This stopping point is our convergence criteria.

# Initialize parameters
m = 0.0
b = 0.0

# Hyperparameters
learning_rate = 0.01
iterations = 1000
N = float(len(data))

# Gradient Descent Loop
for i in range(iterations):
    # Initialize gradients
    m_gradient = 0
    b_gradient = 0

    # Calculate gradients over the whole dataset (batch)
    for point in data:
        x = point['size']
        y = point['price']
        # Partial derivative for m
        m_gradient += -(2/N) * x * (y - (m * x + b))
        # Partial derivative for b
        b_gradient += -(2/N) * (y - (m * x + b))

    # Update parameters
    m = m - learning_rate * m_gradient
    b = b - learning_rate * b_gradient

    # Optionally, print the cost to watch it decrease
    if i % 100 == 0:
        print(f"Iteration {i}: Cost = {calculate_mse(m, b, data)}")

After the loop finishes, the final values of $m$ and $b$ define the line of best fit that our algorithm has learned. Because the MSE cost function for linear regression is convex—meaning it's shaped like a bowl with only one global minimum—gradient descent is guaranteed to find the optimal parameters, provided the learning rate is set correctly.

Time to test your knowledge on finding the best fit.

Quiz Questions 1/5

What is the primary purpose of a cost function in linear regression?

Quiz Questions 2/5

In the analogy of gradient descent being like walking down a foggy hill, what does the 'steepness' of the ground at your current position represent?

By understanding how to minimize a cost function, you've unlocked the core mechanism that powers the training of countless machine learning models.

Applied Machine Learning Foundations