At scale, single-stage retrieval using only a sparse retriever like BM25 is insufficient. It fails to capture semantic nuances and often struggles with queries that don't share keywords with the document corpus. The solution is a multi-stage pipeline that balances computational efficiency with semantic relevance. The foundational trade-off in this architecture is between bi-encoder and cross-encoder models. Bi-encoders generate document and query embeddings independently, making them extremely fast for first-pass candidate retrieval from billions of documents. Cross-encoders, which process the query and document simultaneously, offer superior relevance but are too slow for first-stage retrieval, relegating them to a later reranking stage on a small candidate set.

The first stage generates a candidate set, prioritizing recall over precision. This is typically a hybrid approach combining a sparse keyword-based search (BM25) with a dense vector search over embeddings. The results from these parallel retrievers are then fused. While a simple linear combination of scores is an option, Reciprocal Rank Fusion (RRF) is often more robust as it doesn't require score normalization. RRF combines ranked lists by giving more weight to items that appear high up in multiple lists, irrespective of their absolute scores.

The second stage, or reranking, operates on this much smaller candidate set (e.g., the top 100-200 documents from stage one). Here, we can afford more computationally intensive models. Instead of a full cross-encoder, a late interaction model like ColBERT provides a compelling middle ground. It computes query embeddings and pre-computes document embeddings in a more granular, token-level way. At query time, it performs cheap, token-level similarity calculations, capturing fine-grained interactions without the full cost of a cross-encoder.

For the dense retrieval part of the first stage, performance hinges on the Approximate Nearest Neighbor (ANN) search algorithm. Most large-scale systems use Hierarchical Navigable Small Worlds (HNSW). The key to HNSW is tuning its parameters to balance the recall-latency tradeoff. The m parameter (number of neighbors per node per layer) affects the graph's quality and memory footprint, while ef_construction controls the quality during indexing. At query time, ef_search dictates the size of the dynamic list used to find the nearest neighbors; a higher value increases recall but also latency. Continuous A/B testing is essential to find the optimal values for your specific data distribution and latency requirements.

On a global platform with diverse content, a single massive index is inefficient. Sharding strategies are critical. By partitioning the index based on high-cardinality metadata, such as course language or subject category, we can drastically reduce the search space for any given query. This not only improves latency but also enhances cache hit ratios, as queries for popular categories or languages are localized to specific nodes. The choice of sharding key depends on the query patterns of your users.

Let's review these advanced retrieval concepts.

By implementing these multi-stage strategies, retrieval systems can move beyond simple keyword matching to provide semantically rich, fast, and scalable search experiences.