Evaluating RAG Retrieval in Python: Recall@k, MRR, and nDCG

Evaluate a RAG retriever and reranker with Recall@k, MRR, and nDCG. Build relevance judgments and separate retrieval failures from context assembly and generation errors.

A reranker can find room for a direct answer at rank one even when the number of relevant results stays the same. Recall@k tracks missing evidence, MRR tracks the first relevant rank, and nDCG rewards placing the most relevant evidence near the top. We’ll use three queries and a small Python example to see where these metrics agree and where they diverge.

For the architecture, see RAG fundamentals . Here, humans judge whether each chunk supports answering a question. If you are benchmarking approximate nearest-neighbor search against an exact vector search, measure agreement with those exact results separately.

Choose the metrics

MetricPer-query definitionQuestion it answers
Recall@kRelevant items in the first k results divided by the total known relevant itemsDid we retrieve the evidence we need?
Hit@kOne if any relevant item is in the first k results, zero otherwiseDid we retrieve at least one relevant item?
RR@kInverse rank of the first relevant item, or zero if none appears in the first kHow early do we see a useful result?
nDCG@kDiscounted cumulative gain through rank k divided by the gain of an ideal rankingAre highly relevant chunks near the top?

MRR@k averages reciprocal rank across queries. For nDCG, this example uses a contribution of (2**grade - 1) / log2(rank + 1) at each rank, starting at one. Build the ideal ranking from all judged chunks for the query. Sorting only the chunks the retriever returned would omit relevant chunks it missed from the denominator. Different implementations use different gain functions. This example follows the TREC Web Track definition .

Build relevance judgments

For each question, record the corpus version, stable chunk IDs, grades, and reasons for the judgments. Here, grade 2 means direct evidence, grade 1 means useful supporting evidence, and grade 0 means no useful evidence. Both positive grades count toward binary relevance. Run a separate evaluation with a stricter judgment threshold if you want to measure retrieval of direct evidence alone.

When comparing retrievers, pool candidates from both and have someone assess the union so that either system can receive credit for finding evidence. This example gives unjudged chunks zero gain. In a real evaluation, check how many results are unjudged. When comparing chunking strategies, store the underlying evidence spans and map them to each strategy’s chunk IDs. This helps avoid rewarding systems for retrieving multiple overlapping copies of the same passage.

Compare two rankings in Python

The example runs with the standard library. It evaluates the first two results for each query. A query with known relevant evidence still contributes to the mean when the retriever returns an empty list.

from math import log2
from statistics import mean

# 2: direct evidence, 1: relevant supporting evidence.
qrels = {
    "q1": {"policy": 2, "exceptions": 1},
    "q2": {"steps": 2},
    "q3": {"limits": 2},
}
runs = {
    "baseline": {
        "q1": ["noise", "exceptions", "policy"],
        "q2": ["steps", "noise"],
        "q3": [],
    },
    "reranked": {
        "q1": ["policy", "noise", "exceptions"],
        "q2": ["noise", "steps"],
        "q3": [],
    },
}


def dcg(grades):
    return sum(
        (2**grade - 1) / log2(rank + 1)
        for rank, grade in enumerate(grades, start=1)
    )


def score(ranking, grades, k):
    if k <= 0 or len(set(ranking)) != len(ranking):
        raise ValueError("Use a positive k and unique ranked IDs")
    relevant = {doc for doc, grade in grades.items() if grade > 0}
    if not relevant:
        raise ValueError("Evaluate unanswerable queries separately")
    top = ranking[:k]
    hits = len(relevant.intersection(top))
    rr = next(
        (1 / rank for rank, doc in enumerate(top, 1) if doc in relevant),
        0.0,
    )
    ideal = dcg(sorted(grades.values(), reverse=True)[:k])
    actual = dcg(grades.get(doc, 0) for doc in top)
    return hits / len(relevant), rr, actual / ideal


for name, run in runs.items():
    scores = []
    for query, grades in qrels.items():
        result = score(run[query], grades, k=2)
        scores.append(result)
        print(name, query, *(f"{value:.3f}" for value in result))
    print(name, "macro", *(f"{mean(col):.3f}" for col in zip(*scores)))
RunQueryRecall@2RR@2nDCG@2
Baselineq10.5000.5000.174
Rerankedq10.5001.0000.826
Baselineq21.0001.0001.000
Rerankedq21.0000.5000.631
Bothq30.0000.0000.000

Mean Recall@2 and MRR@2 both stay at 0.500. Mean nDCG@2 rises from 0.391 to 0.486. The reranker moves the direct evidence for q1 to first place, but pushes the evidence for q2 down to second. It has nothing to reorder for q3. Save per-query differences alongside mean scores to catch regressions like q2. These three queries illustrate metric behavior. Use actual questions to make a deployment decision.

Find the failing stage

  1. Candidate retrieval. Measure recall over a larger set, such as the first 50 candidates. If evidence is missing, check corpus coverage, search filters, chunking, and the candidates from keyword and vector search.
  2. Reranking. Hold the upstream candidate list fixed and compare recall and nDCG for a smaller cutoff, such as five. For multi-part questions, also measure the fraction of queries for which every required piece of evidence is present.
  3. Context assembly. Record which chunks survive the token limit. If a chunk is truncated, check whether the evidence span itself survived.
  4. Generation. Evaluate correctness and citations on answerable questions, and appropriate abstention on questions that lack evidence.

Keep the query set, corpus version, relevance policy, cutoff, and context budget fixed across runs. Record retrieval latency and the number of prompt tokens to see what the improvement costs.

When the chunks themselves change, keep the evidence locations and context budget fixed. Evaluate retrieval across RAG chunking settings

FAQ

How does Recall@k differ from Hit@k?

Recall@k measures the fraction of known relevant items retrieved in the first k results. Hit@k checks whether at least one relevant item appears there.

Can I evaluate a reranker when Recall@k is unchanged?

Yes. Use MRR to measure the rank of the first relevant result and nDCG to capture graded relevance across ranks.

Do good retrieval metrics guarantee a correct RAG answer?

Evaluate answer correctness separately. Context assembly can drop evidence, and the model can misinterpret the evidence it receives.