How to Chunk Documents for RAG: A Founder's Guide

If you are wondering how to chunk documents for RAG, start here: cut each document where its meaning changes, usually at its headings, keep each piece small enough to be about one thing, repeat a few lines across each cut, and store the page title with every piece. Then test the result with real questions. Chunking is the step where most retrieval answers quietly go wrong, because the model can only answer from the pieces it is handed.

Retrieval is the second lesson of The AI Product Stack course on Age of AI Entrepreneur, an Intermediate course on model APIs, giving models your data and choosing a model; sign in with your email and an emailed code to open it.

What chunking is and why RAG needs it

RAG stands for retrieval augmented generation. The idea was set out in a 2020 research paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", which describes models that combine what a model learned in training with an outside store of documents it can look things up in. In a product, that means your help pages, contracts or notes sit in a search index, and each question pulls back the most relevant pieces to put in front of the model.

Those pieces are the chunks. You cannot hand the model a whole library on every request: each request has a limit on how much it can read, set by the model's context window, and every extra token adds to the cost. So you cut your documents into chunks once, store them, and on each question send only the few that match.

Each chunk is turned into an embedding, a list of numbers that captures its meaning, so the search can match a question to chunks that mean the same thing even when the words differ. Our guide to what embeddings are covers that step.

Where chunking sits in the pipeline

The diagram shows the four steps. Chunking happens once, at the start, and every answer after that depends on it.

If the cut falls in the wrong place, the search may still find the right chunk, but the chunk will be missing the line that held the answer. The model then fills the gap with a guess. That is why a RAG product can retrieve "the right page" and still answer wrongly.

How to chunk documents for RAG, step by step

1. Clean the text first. Strip menus, footers, cookie notices and repeated page furniture. They match many questions and crowd out the real content.
2. Cut at structure. Split at headings, then at paragraphs. A section called "Refunds for annual plans" is already one idea, so let it be one chunk.
3. Cap the size. If a section runs long, split it again at paragraph breaks. A chunk should answer one kind of question. A whole chapter in one chunk matches too many questions; a single sentence lacks the context around it.
4. Add a little overlap. Repeat the last sentence or two of one chunk at the start of the next, so a rule that runs across a cut is not lost.
5. Keep the label. Store the document title and the section heading with each chunk, and put them in the chunk's text too. "Within 30 days" means nothing until the chunk also says it is about refunds.
6. Keep the source. Store the page address with each chunk, so your product can show the reader where an answer came from.

There is no chunk size that suits every product. Short help-center answers work well as small chunks, while contracts or research notes often need larger ones to keep a clause and its exceptions together. Treat any size as a starting guess and let the test below decide.

A worked example: a help center with 40 pages

Say you run a small online shop and want a support assistant that answers from your 40 help pages.

1. Export the pages as text and strip the site menu and footer from each.
2. Split each page at its subheadings. A page on refunds becomes "Refunds", "Refunds for annual plans" and "How long a refund takes".
3. One page, on shipping zones, runs long. Split it at its paragraphs so each zone is its own chunk.
4. Start each chunk with its page title and heading, for example "Refund policy > Refunds for annual plans", and repeat the last sentence of the chunk before it.
5. Store each chunk with its embedding and the page address.

Now a customer asks, "Can I get a refund on a yearly plan?" The search pulls back the "Refunds for annual plans" chunk first and the general "Refunds" chunk second. Because the overlap repeats the general rule's last line, and the label says which policy it belongs to, the model has what it needs to answer without guessing.

How to test whether your chunks work

You do not need special tools for a first test. You need a list of real questions and a spreadsheet.

1. Write 20 questions your users actually ask, and next to each, the page and section that holds the answer.
2. Run only the search step for each question and look at the top 3 chunks it returns.
3. Mark each question: did the right chunk come back in the top 3, and did that chunk contain the full answer?
4. Change one thing, such as chunk size or overlap, and run the same 20 questions again.

The first mark tells you whether retrieval finds the right place. The second tells you whether your cuts kept the answer whole. Checking these before you judge the model's final answers saves time, because a wrong final answer often starts with a wrong chunk. The Evals: Shipping AI That Works course goes further, with a lesson on building an eval set and judging outputs.

When chunking is not the problem

Better chunks will not fix everything. If the answer is not in your documents, retrieval cannot find it, and the model needs a clear instruction to say so instead of guessing. If the task is about tone or format rather than facts, retrieval may be the wrong tool; our comparison of RAG vs fine-tuning sets out when each fits.

Builders also differ on how much to invest in chunking. Some prefer simple heading-based cuts and spend their effort on testing. Others use methods that cut by meaning, or send larger chunks now that many models read more text at once. Each has costs: larger chunks mean more tokens per request, and cleverer cuts are harder to debug. The test above shows which trade works for your documents.

Frequently asked questions

What is a good chunk size for RAG?

There is no single right size. Start with chunks that each cover one section or idea, then run 20 real questions through your search and adjust the size based on which chunks come back whole.

Why add overlap between chunks?

Overlap repeats a sentence or two across each cut, so a rule or number that sits right at a boundary still appears whole in at least one chunk.

Should I store the title with each chunk?

Yes. A chunk on its own often loses what it is about. Putting the page title and heading in each chunk helps the search match it and helps the model read it correctly.

Does chunking matter if the model can read long documents?

It still matters for cost and focus. Sending only the matching chunks keeps each request smaller, and the model is not handed pages that have nothing to do with the question.

Get started

The AI Product Stack course on Age of AI Entrepreneur has three lessons: Model APIs, Tokens and What a Request Costs; Giving Models Your Data: Retrieval; and Choosing Models: Big, Small, Open, Closed. Each lesson takes about 9 minutes, lists its sources and checks you with questions as you go, and you can retake a lesson any time for a better grade. Sign in with your email and an emailed code to open the course.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.