You already know that Singular Value Decomposition (SVD) can uncover hidden patterns in user-item data. It's powerful, but it's just the start. Modern recommendation systems rarely use textbook SVD. Instead, they rely on more flexible Latent Factor Models that directly model the interactions we care about.

The core idea is to predict a user $u$'s rating for an item $i$, denoted $\hat{r}_{ui}$, by taking the dot product of a user-vector $p_u$ and an item-vector $q_i$. Both vectors live in a low-dimensional latent space of size $f$.

This is a good start, but we can do better. Some users are consistently harsh critics, while others are easy graders. Some movies are universally loved, while others are panned. These are biases in the data. We can account for them by adding bias terms to our model.

This model is often called SVD++ because it adds these intelligent baselines to the core factorization logic. Now, the challenge is figuring out the best values for all those $b$'s and latent factor vectors.

To find the optimal factor vectors and biases, we need to minimize the error between our predictions and the actual ratings in our training data. We define a loss function, typically the sum of squared errors, and add a regularization term to prevent overfitting.

Two popular algorithms for this optimization are Stochastic Gradient Descent (SGD) and Alternating Least Squares (ALS).

Alternating Least Squares (ALS) takes a different approach. If we hold the user factors constant, the loss function becomes a simple quadratic equation in terms of the item factors. We can solve this optimally with linear regression. The same is true if we hold the item factors constant and solve for the user factors.

ALS leverages this by alternating: first, it fixes the user factors and solves for the best item factors. Then, it fixes the new item factors and solves for the best user factors. It repeats this process until the model converges. Because the calculations for each user (or item) are independent in each step, the algorithm is highly parallelizable, making it a favorite for distributed computing frameworks like Apache Spark's MLlib.

Star ratings are great, but most user feedback is implicit. People click on articles, watch videos, listen to songs, and buy products. These actions signal preference, but they're different from explicit ratings. A user not watching a movie doesn't mean they dislike it; they might not even know it exists. Implicit feedback is a record of positive events only.

To handle this, we introduce two new concepts: preferences ($p_{ui}$) and confidence levels ($c_{ui}$).

Our confidence in a preference of 0 (no interaction) is low. For a preference of 1, confidence increases with the number of interactions. For example, watching a video once gives us some confidence you liked it; watching it ten times gives us much more. We can model confidence as:

This leads to a new loss function for our model, often called Weighted Alternating Least Squares (WALS). We want to find factors that explain the preferences, weighted by our confidence.

Because this formula includes every possible user-item pair (including all the zeros), it's computationally intensive. However, the structure of the problem allows for an efficient solution using a modified ALS algorithm, which is a key reason for its popularity in production systems that handle massive amounts of implicit feedback.

Let's test your understanding of these advanced factorization techniques.

By moving beyond simple matrix decomposition and incorporating biases, implicit signals, and robust optimization methods, you can build far more accurate and scalable recommendation engines.