A trained machine learning model is powerful, but it's not a complete search system. When a user searches for a new job on LinkedIn or a movie on Netflix, the system doesn't run a complex ranking model on every single item in the database. With millions or even billions of documents, that would be incredibly slow. Users expect results in milliseconds, not minutes.

To solve this, production search systems use a multi-stage retrieval pipeline, often called the 'Search Funnel.' The core idea is simple: use fast, less-complex methods to quickly filter a huge collection of documents down to a manageable size, and then apply progressively slower, more accurate models on that smaller set.

The first stage of the funnel is candidate generation, where the primary goal is recall. This means we want to cast a wide net and retrieve all potentially relevant documents, even if many of them are not perfect matches. Speed is critical here. This stage often uses efficient retrieval methods like inverted indexes for keyword matching or approximate nearest neighbor (ANN) search for finding semantically similar items based on their vector embeddings. We aren't trying to find the best 10, just a good few thousand to pass to the next stage.

Once we have a smaller set of candidates, the focus shifts to precision. In the subsequent ranking and re-ranking stages, we use more computationally expensive models. These models can look at hundreds of features—about the user, the query, and the items themselves—to score each candidate and produce a final, ordered list. A first-pass (L1) ranker might be a relatively simple model, like logistic regression, that filters the candidates from thousands down to hundreds. A final (L2) re-ranker might be a large deep learning model that meticulously orders the top results for the user to see.

This entire funnel architecture is driven by a critical constraint: latency—the time it takes to return results to the user. The parts of the system that respond directly to a user's query are called online components. This includes the entire multi-stage funnel. These components must be extremely fast.

However, a lot of the heavy lifting happens in offline components. These are the processes that run in the background, not in response to a live query. This includes training the ranking models, generating vector embeddings for all documents, and building the indexes used for candidate generation. This separation allows us to perform computationally intensive tasks ahead of time, ensuring the online system has everything it needs to be fast and responsive.

Think of it like a restaurant. The offline process is the 'mise en place'—chopping vegetables, preparing sauces, and pre-cooking certain ingredients. This is like training our models and indexing documents. The online process is what happens when a customer places an order: the chef quickly assembles the prepped ingredients, cooks the dish, and sends it out. Without the offline prep, every order would take hours.

Understanding this architecture is key for anyone working in search. It explains why a data scientist's job isn't just about building the most accurate model, but about building a model that fits effectively into one of these stages, balancing the constant trade-off between accuracy and speed.