Choosing Chunk Size and Overlap for RAG Document Ingestion

Quick answer: there is no universal best chunk size. Small chunks retrieve precise passages but lose surrounding context, while large chunks keep context but dilute the embedding and waste prompt tokens. Start with chunks of a few hundred tokens, split along document structure instead of fixed character counts, add modest overlap, and tune using retrieval tests on your own questions.

What Trade-offs Does Chunk Size Create?

An embedding represents the overall meaning of a chunk. When a chunk is large and covers several topics, its embedding becomes a blur of all of them, so a specific question may match it weakly. When a chunk is tiny, the embedding is sharp, but the passage may lack the context needed to answer, such as the subject of a pronoun or the conditions attached to a rule.

Chunk size also affects cost and answer quality downstream. Retrieved chunks are pasted into the model prompt, so large chunks consume the context budget quickly and add irrelevant text that can distract the model. Small chunks let you include more distinct sources, but you may need to retrieve many of them to cover a single answer.

Different content calls for different sizes. A policy document with self-contained clauses suits smaller chunks, while a narrative or a technical explanation that builds across paragraphs needs larger ones. Treat any single global setting with suspicion, and check whether your corpus has distinct document types that deserve separate settings.

How Should You Split Documents?

Fixed-size splitting by characters is simple, but it cuts through sentences, lists, and code blocks. Structure-aware splitting follows boundaries the author already created: headings, paragraphs, list items, and code fences. Chunks that respect these boundaries are more coherent, embed better, and read more naturally when shown to the model.

• Split on headings first, then paragraphs, then sentences, so a chunk ends at the largest natural boundary under your size limit.
• Keep code blocks, tables, and numbered procedures intact where possible.
• Store the document title and section path as metadata, and consider prepending them to the chunk text so each chunk carries its context.
• Strip repeated boilerplate such as headers, footers, and navigation, which pollutes embeddings.

Overlap repeats a small portion of text between neighboring chunks, so a sentence split at a boundary still appears whole in at least one chunk. A modest overlap, often ten to twenty percent of the chunk size, is a common starting point. Too much overlap inflates your index and returns near-duplicate results.

How Do You Tune It Without Guessing?

Build a small evaluation set of real questions with known source passages, then measure whether retrieval returns the right chunk in the top results. Compare a few configurations, such as different sizes and overlaps, and look at both hit rate and the amount of irrelevant text returned. The best choice is the one that wins on your data, not the one recommended in an article.

A useful pattern is parent-child retrieval. Embed small chunks for precise matching, but return the larger surrounding section to the model when a small chunk matches. This gives sharp retrieval and rich context, at the price of a more complex index. It works well for long documents with clear sections.

Re-evaluate when your content, embedding model, or question types change. Changing the embedding model usually means re-embedding everything, so test chunking and model choices together, and keep the ingestion pipeline reproducible so you can rebuild the index with new settings.

Frequently Asked Questions

What chunk size should I start with?

A few hundred tokens is a common starting point. Test smaller and larger values against your own questions before committing.

Is overlap always necessary?

Not always. It helps when boundaries cut through ideas. If you split on natural structure, you may need little or none.

Should chunk size be measured in characters or tokens?

Tokens are better, because embedding models and prompts are limited by tokens. Use the tokenizer that matches your embedding model where possible.

Do I need different chunking for tables and code?

Often yes. Keep tables and code blocks intact, or convert tables to a text form that preserves headers with each row.

Summary

Choose chunk size by balancing precision against context, split along document structure, add modest overlap, and carry titles as metadata. Tune with retrieval tests on real questions and revisit when content or models change. Consult your embedding model documentation for token limits.