Machine learning models aren't magic boxes. At their core, they are optimization problems. The goal is to find the best possible set of model parameters, or weights, that minimize a specific type of error. This error is quantified by a cost function, which measures how far off our model's predictions are from the actual data.

The primary tool for this minimization task is an algorithm called Gradient Descent. Think of it as a methodical way of finding the bottom of a valley. By calculating the slope (the gradient) of the cost function at our current position, we can determine which direction is 'downhill' and take a small step that way. Repeating this process iteratively leads us to a minimum point, where the model's error is as low as possible.

The general update rule for any weight (often denoted as $\beta$) in our model is governed by this simple equation:

This process is applied simultaneously to all weights in the model. The choice of the cost function $J$ is critical and depends entirely on the type of problem we are solving.

For linear regression, where we predict a continuous value, the most common cost function is the Mean Squared Error (MSE). Its logic is straightforward: for each data point, we measure the vertical distance between our model's prediction and the actual value, square that difference, and then find the average of all these squared differences. Squaring the error ensures that negative and positive errors don't cancel each other out and it penalizes larger errors more heavily.

When we use MSE, the cost function is convex, meaning it has a single global minimum. This is convenient because it guarantees that Gradient Descent, given a proper learning rate, will eventually find the best possible set of weights.

Classification is different. In logistic regression, the model outputs a probability between 0 and 1. If we tried to use MSE here, the interaction with the sigmoid function (which squashes the output) would create a non-convex cost function with many local minima. Gradient Descent could easily get stuck in a sub-optimal solution.

Instead, we use a function called Log-Loss, also known as Binary Cross-Entropy. This function is designed specifically for probabilities. It heavily penalizes the model when it makes a confident but incorrect prediction. For instance, if the correct answer is 1 but the model predicts 0.01, the penalty is huge. If the model predicts 0.4, the penalty is much smaller.

This cost function is convex for logistic regression, allowing Gradient Descent to find the optimal solution just as it does for linear regression with MSE.

Unsupervised learning finds patterns without labels. The algorithms still rely on mathematical optimization, just with a different objective.

For K-Means clustering, the goal is to partition data into K distinct groups. The process is a simple and elegant iterative algorithm known as Expectation-Maximization.

1. Initialization: Randomly place K centroids in the data space.
2. Assignment Step (Expectation): For each data point, calculate its distance to every centroid. Assign the point to the cluster of the nearest centroid.
3. Update Step (Maximization): Recalculate the position of each centroid by taking the mean of all data points assigned to its cluster.

These two steps are repeated until the cluster assignments no longer change, meaning the centroids have settled into their optimal positions. The underlying goal is to minimize the within-cluster sum of squares.

Another powerful unsupervised technique is Principal Component Analysis (PCA). It's used for dimensionality reduction. PCA transforms a high-dimensional dataset into a new, lower-dimensional dataset while retaining as much of the original variance as possible.

It achieves this by finding a new set of orthogonal axes, called principal components. The first principal component (PC1) is the direction in the data that accounts for the largest possible variance. The second (PC2) is the direction, orthogonal to PC1, that accounts for the next largest variance, and so on. Mathematically, these components are the eigenvectors of the data's covariance matrix. The corresponding eigenvalues indicate the amount of variance captured by each eigenvector. By projecting the original data onto the first few principal components, we reduce its dimensionality while losing minimal information.

Time to check your understanding of these core concepts.