In a Retrieval-Augmented Generation (RAG) system, the quality of the final answer depends heavily on the quality of the documents retrieved. If you feed the language model irrelevant information, you'll get an irrelevant or incorrect answer. So, how do you measure how well your retriever is doing its job?

The first step is to create a 'golden dataset'. This is a curated collection of test cases, each containing a sample query and the ID of the document chunk that holds the correct answer. This dataset serves as our ground truth, the benchmark against which we measure performance.

Once you have this dataset, you can run your queries through the retrieval system and compare the results to your ground truth. This comparison is where traditional retrieval metrics come into play. They provide a quantitative score for your retriever's performance based on a simple idea: binary relevance. For a given query, a document is either the correct one or it isn't. There's no middle ground. Let's look at the most common metrics.

For many applications, the position of the correct document in the search results is crucial. A user isn't going to sift through ten irrelevant documents to find the one good one. They expect the best result at the top. This is where Mean Reciprocal Rank (MRR) is useful.

The reciprocal rank for a single query is simply $1/rank$. If the first relevant document is at rank 1, the score is 1. If it's at rank 2, the score is 1/2. If it's at rank 3, it's 1/3, and so on. If no relevant document is found in the results, the score is 0. MRR is the average of these scores over all your test queries.

Imagine a technical support chatbot. A user asks a question, and getting the right answer quickly is essential. Let's say we test our retriever with three queries from our golden dataset.

To calculate the MRR for this set, we average the reciprocal ranks:

An MRR of 0.61 tells us that, on average, the system is good at placing the correct document near the top of the results, but there's room for improvement.

Sometimes, you don't care as much about the exact rank, as long as the correct document appears somewhere in the top few results. For instance, if you're retrieving five chunks to give an LLM context, any of those five being the correct one might be good enough. This is measured by Hit Rate, also known as Recall@k.

Let's use our previous example and calculate the Hit Rate@3. This means we consider it a 'hit' if the correct document appears in the top 3 results.

• Q1: Correct doc was at rank 1. This is $\leq 3$, so it's a hit.
• Q2: Correct doc was at rank 3. This is $\leq 3$, so it's a hit.
• Q3: Correct doc was at rank 2. This is $\leq 3$, so it's a hit.

All three queries were hits. The Hit Rate@3 is the number of hits divided by the number of queries, which is $3/3 = 1.0$, or 100%.

What about Hit Rate@1? Only Q1 had the correct document at rank 1. So the Hit Rate@1 would be $1/3 \approx 0.33$. The value of 'k' you choose depends entirely on how many documents your RAG pipeline passes to the LLM.

These traditional metrics are fast, cheap, and provide a solid baseline for your retriever's performance. They tell a clear story based on your golden dataset. However, their reliance on binary relevance is also their biggest limitation. They can't capture nuance, assess partially correct documents, or evaluate the quality of the generated text itself. For that, we need to bring the LLM into the evaluation loop.