Keyword search and semantic search each have a critical weakness. A keyword search for "AI safety concerns" might completely miss a vital document that discusses the "risks of artificial intelligence." It's too literal. On the other hand, a purely semantic search might fail with specific, rare terms. If you search for a product code like "XJ-77B" or a unique project name, a vector model might blur that specific identifier into a generic concept like "technical component," causing you to miss the exact match.

Neither system is sufficient on its own. Keyword systems excel at precision for exact terms, while semantic systems grasp context and meaning. Hybrid search combines these two approaches to get the lexical precision of keyword search and the contextual understanding of semantic search.

The architecture of a hybrid search system is elegantly simple: it runs two searches in parallel. When a query comes in, the system doesn't choose one method over the other. Instead, it sends the query down two separate but simultaneous paths.

1. The Sparse Path: The query is tokenized, meaning it's broken down into individual words or sub-words. These tokens are then used by a sparse retrieval algorithm like BM25 to find documents containing those exact keywords. This path generates a list of candidate documents, each with a BM25 score indicating its relevance based on term frequency and inverse document frequency.

2. The Dense Path: The same original query is fed into an embedding model. This model converts the query's semantic meaning into a dense vector, a list of numbers. This vector is then used to find documents with similar vectors in the database, typically using cosine similarity as the metric. This path also generates a list of candidate documents, each with a similarity score.

At the end of this parallel process, we have two independent lists of results. The challenge isn't getting the results; it's figuring out how to merge them when their scoring systems are fundamentally different.

The scores from each pipeline live in different universes. A BM25 score is an unbounded positive number. A score of 20.5 is better than 10.2, but there's no theoretical maximum. In contrast, cosine similarity scores are neatly contained, usually between -1 and 1. A score of 0.98 is extremely similar, while a score of 0.1 is not.

Comparing a BM25 score of 15.0 to a cosine similarity of 0.85 is meaningless. It's like asking whether 15 feet is more than 0.85 kilograms. Before the results can be merged, the scores must be converted to a common scale. This process is called normalization.

Normalization rescales numbers from different distributions into a shared range, typically 0 to 1. This allows for a fair comparison. Two common techniques are Min-Max Normalization and Z-score Normalization.

Min-Max Normalization is straightforward. For each list of scores (e.g., all the BM25 scores for a given query), you find the minimum and maximum values. Then, you apply a formula to each score.

Another method, Z-score Normalization, approaches the problem differently. Instead of focusing on min and max values, it rescales scores based on the mean (average) and standard deviation of the dataset. It tells you how many standard deviations away from the mean a particular score is.

While Z-scores aren't naturally bounded between 0 and 1, they standardize the distributions, which can then be mapped to a common range. This method is less sensitive to extreme outliers than Min-Max scaling.

Once both sets of scores—from the sparse and dense paths—are normalized, they can be combined. The system now has a unified basis for ranking, allowing it to leverage the strengths of both retrieval methods to produce a single, more intelligent list of results.