Transcript
Beau
Alright Jo, so we've done all this work. We've gone from BM25 to dense vectors, we've fine-tuned our bi-encoders, and we've got this beautiful, complex LambdaMART model for re-ranking. The offline tests look great. But now what? How do we prove it's actually better in the wild without, you know, breaking everything?
Jo
That is the million-dollar question. Because a good score in a lab doesn't pay the bills. The first step is to make sure our lab, our offline evaluation, is as robust as possible. We're not using simple accuracy or RMSE here. For search, we need metrics that understand that the order of results is everything.
Beau
Right, because position one is beachfront property and position ten is... in a different zip code. I assume you're talking about things like NDCG?
Jo
Exactly. Normalized Discounted Cumulative Gain. It's a mouthful, but it's brilliant. It does two things. First, it assigns a relevance score to each result—not just 'correct' or 'incorrect', but maybe a scale of 1 to 5. Then it 'discounts' that gain based on position. A highly relevant item at position one gets full credit, but that same item at position ten gets its value heavily penalized.
Beau
So it's basically rewarding the model for thinking like a user, who's most likely to look at the first few results. What about... hmm, what about something like MRR? I've seen that one too. Mean Reciprocal Rank. How's that different?
Jo
MRR is simpler and has a different focus. It only cares about the rank of the *first* correct answer. If the first relevant result is at position one, the score is 1. If it's at position two, the score is 1/2. Position three, 1/3, and so on. It's great for 'known-item' searches.
Beau
Ah, so like when a user is searching for a specific product they already know exists, or a login page. They just want that one link, and they want it at the top. So you'd use NDCG for broad, informational queries and MRR for more navigational or 'find-the-thing' queries?
Jo
You got it. And then there's MAP, Mean Average Precision, which kind of splits the difference. It rewards you for the average precision across all the relevant documents in the list. It cares about recall more than MRR does. But honestly, NDCG is the industry standard for most nuanced ranking tasks.
Beau
Okay, so our offline NDCG is up by 5%. We're feeling good. We're ready to show it to real people. This is where A/B testing comes in, right? The classic showdown.
Jo
Yep. Control group gets the old ranking model, treatment group gets our shiny new LambdaMART model. We watch the clicks, the conversions, session length... all the business metrics. But search has a tricky problem: selection bias. The data we collect is biased by the very results we decided to show the user.
Beau
Because users can't click on what they don't see. If our old model buried a perfect result on page two, we'd never get a positive click signal for it, so our new LTR model might not learn to rank it highly either. It's a vicious cycle.
Jo
Precisely. And that makes it hard to evaluate a new model using old click logs. That's where counterfactual evaluation comes in. Techniques like Inverse Propensity Scoring, or IPS. It's a way to re-weigh the click data you have to estimate what *would have happened* if you'd shown a different set of results.
Beau
So it's like a statistical 'what if' machine. It looks at a click on a result at position 5 and says, 'Okay, the user clicked this, but the old model gave it a very low probability of being shown this high, so this click is actually a super valuable signal.' It up-weights the surprising clicks?
Jo
That's a fantastic way to put it. It helps you get a more honest signal from biased historical data, letting you test many potential models offline before committing to a costly A/B test. Now, let's say the A/B test is a success. We have to put this thing in production. And our new model, maybe it uses a big cross-encoder, is... slow.
Beau
Right, we can't have the user waiting two seconds for search results. That latency-precision trade-off we talked about in the very beginning. We need sub-second responses. How do you deploy a heavy-duty transformer model without grinding everything to a halt?
Jo
With a 'Neural Cascade'. It's just like the multi-stage funnels we've been discussing, but applied at the final ranking step. Stage one is super fast. Maybe it's just BM25 and a few simple features. It takes 1000 candidates and cuts them down to 100. Then stage two is a medium-weight model, maybe our LambdaMART, that re-ranks those 100 down to 20.
Beau
And only then, for that tiny final set of 20, do you bring out the big guns—the slow, expensive, but incredibly accurate cross-encoder model to get the final order for the top 10. So you're only paying the high computational cost on a very small number of documents.
Jo
Exactly. It's an engineering solution to a machine learning problem. You get the best of both worlds: speed and accuracy. But once it's deployed, you're not done. You have to monitor it. You have to watch for 'rank drift'—where the model's performance slowly degrades over time as user behavior or the document collection changes.
Beau
And new items are being added all the time. The cold start problem. A new document doesn't have any click history, so our LTR model doesn't know what to do with it. It has zero for all the user-behavior features.
Jo
Right. So for those, your model has to rely purely on content features—the BM25 score, the semantic similarity from the bi-encoder. You might even temporarily boost new documents to help them gather some initial click data. It's a constant cycle of offline evaluation, online testing, careful deployment, and active monitoring.
Beau
It really drives home that building the model is just one small part of the job. The engineering and evaluation infrastructure around it is what makes it actually work and... continue to work. It's not a 'fire and forget' missile, it's more like tending a garden.
Jo
I like that. A very complex, computationally expensive garden that users expect to be perfect every single time they visit.