RAG Chunking: Choose Chunk Size With Evals

Short answer: there is no best chunk size. There is only the size your retrieval evaluation says works for your documents and your questions. Fixed 512-token chunks are a reasonable default, not a rule. Teams that guess spend weeks tuning prompts to compensate for a retrieval unit they never measured. The fix is unglamorous: build thirty real questions, index the same corpus at three sizes, and compare recall@k.

Chunk size is a retrieval decision, not a formatting one. Measure it the way you measure any other retrieval parameter: hold the corpus and embedding model fixed, vary only the chunker, and score whether the passage containing the answer appears in the top-k results for real user questions.

What chunking actually controls

A vector search returns units. Whatever you put in a unit is what the model gets, and what the model gets is what it can answer from.

That creates a trade-off with no universal answer:

• Too small: the chunk loses the qualifier that made the sentence true. “Interest accrues at 4%” without “for balances under $5,000” is worse than no answer.
• Too large: the embedding averages several topics, so similarity to any single question drops. You also spend context on text nobody asked about.

Both failures look identical at the application layer: the model says it cannot find the information.

The four strategies worth knowing

Strategy	How it splits	Strong when	Weak when
Fixed-size	Every N tokens, with overlap	Uniform prose, quick baseline	Cutting mid-sentence and mid-table
Recursive	Paragraph, then sentence, then word, until it fits	Mixed-length prose, most docs	Documents with deep structure
Semantic	Split where consecutive sentences diverge in embedding space	Long unstructured narration	Cost and reproducibility; thresholds are fiddly
Structure-aware	Respect headings, code blocks, table boundaries	Markdown docs, API references, legal text	Poorly formatted source material

For technical documentation, structure-aware usually wins outright. It is also the cheapest to explain to a reviewer: a chunk is a section, and a section has a heading you can show as a citation.

Build the measurement first

Before touching the chunker, write down what a correct retrieval looks like. You need a query set where you know the source passage.

# eval_set.jsonl  -  one object per line
{"q": "What is the default request timeout?",
 "must_include": "docs/api/timeouts.md#section-2"}
{"q": "How do I rotate an API key without downtime?",
 "must_include": "docs/security/keys.md#rotation"}

Thirty to fifty questions is enough to see a difference between chunkers. Fewer than twenty and the noise swamps the signal. Pull them from support tickets, sales calls, and your own search logs — not from the documentation’s table of contents, which is already organised the way the docs are organised, not the way people ask.

The chunker

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def fixed_chunks(text: str, size: int = 512, overlap: int = 64):
    toks = enc.encode(text)
    step = size - overlap
    for start in range(0, len(toks), step):
        piece = toks[start:start + size]
        if not piece:
            break
        yield enc.decode(piece)

Two details that matter more than the size itself. Overlap should be roughly ten to fifteen percent of the chunk, so a sentence split across a boundary appears whole in at least one unit. And chunk on token counts, not characters — character counts vary by a factor of three across languages and code.

The eval loop

def recall_at_k(queries, build_index, k=5):
    index = build_index()
    hits = 0
    misses = []
    for item in queries:
        results = index.search(item["q"], k=k)
        sources = [r.source_id for r in results]
        if any(item["must_include"].startswith(s) for s in sources):
            hits += 1
        else:
            misses.append(item["q"])
    return hits / len(queries), misses

Run it once per chunker configuration, holding the embedding model, the distance metric, and k constant. Change one variable at a time or the result means nothing.

What the output looks like

The table below is an illustration of the shape of the result, not a benchmark you should quote. Your numbers will differ, and they should — that is the entire point.

Configuration	Recall@5	Mean chunk tokens	Index size
Fixed 256	0.71	256	1.0×
Fixed 512	0.84	512	0.5×
Fixed 1024	0.86	1024	0.25×
Recursive 512	0.88	470	0.55×
Structure-aware	0.93	610	0.42×

Read it like this. Going from 256 to 512 buys fifteen points of recall for half the index size — an easy win. Going from 512 to 1024 buys almost nothing while doubling the text the model must read. Structure-aware wins because the documents have headings, not because the number is special.

That plateau is the signal you are looking for. Keep increasing size until recall stops moving, then step back one configuration and stop.

Failure modes that no recall score catches

Failure	Symptom	Fix
Table split across chunks	Model reads a row without its header	Never split inside a table; prepend the header row
Orphaned heading	Answer arrives with no document context	Prefix every chunk with its heading path
Mid-sentence cut	Truncated claim, confident wrong answer	Overlap, or split on sentence boundaries
One giant chunk	Crowds out the other four results	Hard cap at the token limit, always
Context lost to pronouns	“It requires approval” with no antecedent	Prepend document title to each chunk

Heading prefixes are the cheapest quality fix in the whole pipeline. A chunk that begins with Billing > Refunds > Partial refunds is findable even when the body text is generic.

Production checklist

1. Store the heading path with the chunk, not just the text.
2. Keep a stable source_id per chunk so eval results map back to documents.
3. Version your chunker. A silent change to chunking invalidates your index and your eval baseline.
4. Rebuild the full index when the chunker changes — partial reindexing produces two incompatible retrieval behaviours.
5. Log which chunks were retrieved for every answer, so you can debug a bad answer later.
6. Re-run the eval when you change embedding model, not only when you change chunk size.

Chunking changes are exactly the kind of thing that should fail CI, not production. The same harness pattern applies: fixed query set, score, threshold, block the merge. We wrote the CI side up separately in building an LLM eval harness that runs in CI.

Frequently asked questions

Is 512 tokens a good default?

It is a reasonable starting point for prose. It is a bad default for code, tables, and structured records, where boundaries matter more than length.

Should chunk size and overlap match?

No. Overlap exists to prevent boundary loss. Ten to fifteen percent of the chunk is typical; higher overlap means more index cost for diminishing returns.

Do bigger chunks always mean worse retrieval?

No. Bigger chunks improve recall up to a point because they contain the answer more often. Past that point you pay context cost for no gain. The plateau is what you measure.

How many questions do I need in the eval set?

Thirty minimum, fifty better. Below twenty, a two-point recall difference is indistinguishable from noise.

Can I skip evaluation and just use semantic chunking?

You can, but semantic chunking adds cost and a threshold to tune, and it is not reliably better than structure-aware chunking on well-formatted docs. Measure before you pay for it.

What about late chunking or contextual retrieval?

Both are worth testing as a fourth variable once your baseline is stable. Do not change technique and chunk size in the same experiment.

Sources

Retrieval evaluation methodology follows standard information-retrieval practice for recall@k. Chunking strategies are described from implementation experience; the numbers in the table are illustrative and must not be cited as benchmarks. Token counts use the cl100k_base encoding. Checked 22 October 2026.

Last updated 22 October 2026.