You already know that embeddings turn complex data like text and images into numerical vectors. But how does a system search through millions or even billions of these vectors to find the most similar ones? A simple one-by-one comparison, known as k-Nearest Neighbor (k-NN), is computationally brutal. It guarantees perfect accuracy but at a cost that makes it unusable for real-time applications.

To solve this, modern systems use Approximate Nearest Neighbor (ANN) search. Instead of finding the exact closest vectors, ANN finds almost the closest vectors. This trade-off is crucial: by sacrificing a tiny amount of precision, we gain an enormous boost in speed. It’s the difference between a search that takes minutes and one that takes milliseconds. ANN algorithms build clever data structures, like graphs or trees, that act as roadmaps through the high-dimensional vector space, allowing them to quickly navigate to the most relevant neighborhood of vectors without checking every single one.

To find the "nearest" neighbors, a vector database needs a way to measure the distance or similarity between two vectors. The choice of metric is not arbitrary; it depends on the nature of the embeddings and the problem you're solving. Three common metrics dominate the field.

Choosing the right metric is vital. Using Euclidean distance on embeddings normalized for cosine similarity can lead to poor results, as it would incorrectly interpret vector magnitude as a meaningful signal.

As the number of dimensions in an embedding vector increases (often to 1536 or more), a strange phenomenon occurs known as the curse of dimensionality . In high-dimensional spaces, the concept of "nearby" breaks down. The distance between the closest and farthest points can become almost indistinguishable, making it difficult for algorithms to find meaningful clusters. All points start to seem equally distant from each other.

This is why traditional database indexes, like B-trees, fail spectacularly for vector search. They are designed for low-dimensional data and cannot cope with the geometric weirdness of high-dimensional spaces. Vector databases use specialized indexing algorithms (the engines behind ANN) that are explicitly built to navigate this counter-intuitive environment.

Beyond the vector itself, the associated data, or metadata, is often just as important. Imagine searching for a product image but only among items that are in stock and under $50. This is where metadata filtering comes in.

A system can perform pre-filtering, where it first narrows down the dataset based on metadata (e.g., is_in_stock = true) and then performs the vector search on the smaller subset. Alternatively, it can use post-filtering, running the vector search first to get the top K most similar items and then filtering that small result set by the metadata. Pre-filtering is more precise but can be slower and miss relevant vectors outside the filtered set, while post-filtering is faster but might return fewer than K results if the top matches don't meet the metadata criteria.

Finally, a key architectural decision is whether the vector index is stateful or stateless. A stateless index might be built on the fly from embeddings stored in a separate blob storage, which is flexible but slower for queries. A stateful vector database maintains a persistent, ready-to-query index of the embeddings alongside the data. This provides low-latency retrieval because the index is always hot. It's the difference between building a map every time you need to find a location versus having a GPS that's always on and ready to give directions. For most production RAG systems, stateful persistence is non-negotiable.

Time to check what you've learned.