How to Evaluate RAG Chunking: Compare Chunk Size and Overlap in Python

Compare RAG chunk sizes and overlap using fixed evidence spans and a shared input token budget. Measure evidence coverage, complete retrieval, and duplicated tokens in Python.

Give two RAG systems their top five chunks and the system with larger chunks can send more text to the LLM. To compare chunk size and overlap, label the evidence in the original documents and inspect what reaches the model under the same input token budget. A small Python example shows how to measure both missing evidence and duplicated text.

Label evidence in the source documents

For each question, mark the source text needed to answer it. Store a document ID, a document revision, and the start and end positions for each evidence span. For example, label character offsets in a fixed source revision using half-open ranges [start, end), then map them to source token positions with a shared tokenizer at evaluation time. Preserve a mapping from normalized chunks back to the source so you can compare different preprocessing or tokenization strategies.

A question asking for both the retention period and the encryption requirements can have multiple spans. If several documents offer equivalent evidence, record alternative sets of spans and accept any one complete set. Requiring every copy would penalize systems for skipping redundant passages. The example below uses one required set per question.

See Evaluating RAG Retrieval in Python for comparing retrievers or rerankers when chunk IDs are stable. Chroma’s technical report also uses token-level evidence to compare chunking strategies. Evaluating Chunking Strategies for Retrieval

Measure what reaches the LLM under a shared budget

Keep the corpus revision, questions, embedding model, retrieval and reranking settings, and context assembly policy fixed. Build a new index for each chunking configuration and record the cap on candidate chunks. When many questions hit that cap, a run with a higher cap can help distinguish candidate starvation from other effects of chunking.

For a production test, reserve room for the template, question, and expected output to determine the evidence budget. Use the generation model’s tokenizer to count the actual input, including any titles, separators, and citation metadata. If you truncate a chunk, merge neighboring chunks, or remove overlap, measure evidence positions in the text that remains just before sending the request.

  • Evidence coverage is the fraction of required source token positions present in the input. Each position counts once.
  • Complete evidence rate is the fraction of questions for which every required position made it into the input. A question with two required spans fails this check if only one arrives.
  • Tokens used and duplicated show how the packed context spends its budget. Count each extra copy of a repeated source position as a duplicate token.

The first two metrics use sets of source positions. They discard ordering and treat adjacent positions as covered even if they appear in different chunks. Check generated answers and citations under the same conditions to see whether changes in order or boundaries affect answers.

Compare chunk size and overlap in Python

This example treats whitespace-delimited words as tokens, with source positions starting at zero. It ranks chunks by the number of distinct terms they share with the question, breaking ties by source order. Each question gets the same budget of 12 tokens. A chunk is skipped if it cannot fit in the remaining budget. There are no headers or separators in the example, so only the document words use up the budget.

from statistics import mean

source = (
    "intro read the guide before backups retain seven daily snapshots "
    "encrypt archives after each export verify checksums weekly"
).split()
questions = [
    ("backups retain seven daily snapshots", [(5, 10)]),
    ("encrypt archives verify checksums", [(10, 12), (15, 17)]),
]
budget = 12


def split_chunks(size, overlap):
    for start in range(0, len(source), size - overlap):
        end = min(start + size, len(source))
        yield start, end
        if end == len(source):
            break


def evaluate(chunks, question, spans):
    terms = set(question.split())
    ranked = []
    for start, end in chunks:
        score = len(terms.intersection(source[start:end]))
        if score:
            ranked.append((-score, start, end))

    covered = set()
    used = 0
    for _, start, end in sorted(ranked):
        length = end - start
        if length > budget - used:
            continue
        covered.update(range(start, end))
        used += length

    gold = {index for start, end in spans for index in range(start, end)}
    return len(gold & covered) / len(gold), gold <= covered, used, used - len(covered)


print("size overlap coverage complete used duplicate")
for size, overlap in [(6, 0), (6, 2), (9, 0), (9, 3)]:
    chunks = list(split_chunks(size, overlap))
    results = [evaluate(chunks, question, spans) for question, spans in questions]
    coverage, complete, used, duplicate = map(mean, zip(*results))
    print(
        f"{size:4} {overlap:7} {coverage:8.3f} {complete:8.3f}"
        f" {used:4.1f} {duplicate:9.1f}"
    )
size overlap coverage complete used duplicate
   6       0    1.000    1.000 12.0       0.0
   6       2    1.000    1.000 12.0       2.0
   9       0    0.900    0.500  9.0       0.0
   9       3    0.650    0.000  9.0       0.0

coverage is the mean of per-question evidence coverage, and complete is the fraction of questions with all their evidence. used and duplicate are also averaged across questions. An answerable question with an empty result scores zero on both evidence measures.

At chunk size 6, an overlap of 2 leaves coverage unchanged and adds two duplicated tokens to the input. With size 9 and overlap 3, the second question produces a tie between a nine-token chunk about encryption and a six-token chunk about checksums. Source order puts the nine-token chunk first, leaving only three tokens for the checksum passage. The result depends on the interaction between chunk size, tie-breaking, and how the context is packed.

On real data, include questions that require a single passage and questions that require multiple passages. Inspect questions that gained or lost complete evidence. If evidence near boundaries is missing, compare overlap with splitting on document headings or paragraphs. When two settings deliver similar coverage, compare their input token usage, number of indexed chunks, and retrieval latency.

Frequently asked questions

Can I use Recall@k to compare chunk sizes?

Recall@k works for comparing retrievers with a shared chunking scheme. Across different chunking schemes, use the same source evidence positions and LLM input token budget for a direct comparison.

How should I count evidence repeated in overlapping chunks?

Count each source position once for evidence coverage. Count every extra copy sent to the LLM toward the input budget and duplicated tokens.

Does complete evidence coverage guarantee a correct answer?

Evaluate evidence retrieval and answer correctness separately. Check whether the LLM connects evidence from different chunks and uses it correctly.