We're used to thinking of vectors as arrows pointing from an origin to a location in 2D or 3D space. This is a helpful starting point, but for data, it's more useful to think of a vector as a point in a space with hundreds or even thousands of dimensions. Each dimension represents a specific feature or attribute. A vector isn't just a direction; it's a rich, dense representation of an object, like a word, an image, or a user profile.

This high-dimensional space is often called a latent space because the features it represents are not always explicit or human-interpretable. Instead, they are 'latent' attributes learned by a machine learning model. For example, in a latent space for words, the dimensions might not correspond to 'is a verb' or 'is a color'. Instead, they capture complex semantic relationships. The words 'king', 'queen', 'prince', and 'princess' would cluster together because their vector representations are close to each other in this space.

Our intuition, built on a three-dimensional world, fails us in high-dimensional spaces. Geometric properties change in bizarre ways. This collection of strange behaviors is often called the curse of dimensionality a term coined by computer scientist Richard Bellman while working on dynamic programming.

Consider a hypercube and a hypersphere that fits perfectly inside it. In 2D, a circle takes up about 78.5% of the area of a square. In 3D, a sphere occupies about 52.3% of the volume of a cube. But as we increase the dimensions, the volume of the hypersphere shrinks to almost zero relative to the hypercube. Almost all the volume of the hypercube gets pushed out into its 'corners', far from the center.

This strange geometry has a profound consequence: as dimensions grow, the concept of 'close' and 'far' begins to break down.

In a high-dimensional space, the distance between any two randomly chosen points becomes almost the same. This is known as distance concentration The difference between the distance to your nearest neighbor and your farthest neighbor shrinks to almost nothing. If every point is roughly equidistant from every other point, how can we find 'similar' items? Your closest neighbor isn't meaningfully closer than a random point.

This completely upends traditional search. Keyword matching is exact; a document either contains 'database' or it doesn't. But vector search relies on finding nearest neighbors to identify semantically similar items. If all distances are the same, how can a vector database find the true nearest neighbors that represent the best semantic matches?

The solution isn't to abandon distance but to be smarter about how we search. While the absolute distances might converge, the relative order of neighbors still holds meaning. The challenge for vector databases is to efficiently find the top-k nearest neighbors in a space where almost everything is 'far away'. This requires specialized indexing algorithms that can navigate the vast, sparse landscape of high-dimensional space without having to check every single point.