RAG Chunking Strategies: Fixed, Semantic, or Structured?
Use fixed-size chunking as a reproducible baseline, structure-aware chunking when headings, lists, tables, or records carry meaning, and semantic chunking only when tests show that topic boundaries improve retrieval enough to justify the added processing.
Choose a strategy from document shape
Chunk boundaries decide which passages compete during retrieval. Begin with representative questions and the natural structure of the documents.
| Corpus and query shape | Start with | Preserve | Primary risk |
|---|---|---|---|
| Mixed prose with weak markup | Fixed size with sentence-safe boundaries | Local continuity and source identifier | A needed statement is split from its qualifier. |
| Manuals, policies, or technical documentation | Heading-aware sections with a size ceiling | Section path, title, and cross-reference | A large section becomes too broad to rank precisely. |
| Contracts, forms, and records | Structure-aware units | Clause, field group, row, or record identity | Layout parsing detaches a value from its label. |
| Narrative text with gradual topic changes | Semantic boundaries | Coherent topic transitions | Boundary detection is unstable or expensive to reproduce. |
| Tables plus explanatory prose | Typed chunks with shared parent metadata | Headers, units, row labels, and nearby explanation | Linearized cells lose their relationships. |
| Unknown or changing corpus | Simple fixed baseline | Deterministic versioned output | Complex preprocessing hides whether retrieval itself works. |
Microsoft's official Azure AI Search chunking guidance describes fixed, variable, semantic, and combined approaches and treats overlap as a corpus-dependent choice. That is a useful menu, not evidence that one preset will work for your documents.
Separate retrieval-stage effects from generation-stage effects
At retrieval, boundaries control candidate quality
A small chunk can match a narrow question precisely, but it may omit the definition, date, exception, or heading needed to interpret the match. A large chunk preserves context but can dilute the signal with unrelated text. Overlap can rescue facts near a boundary, yet it also creates near-duplicate candidates that consume ranking and context capacity.
Keep stable chunk IDs, parent document IDs, section paths, timestamps, and authorization fields outside the prose. Those fields support filtering, deduplication, citation, deletion, and incident replay. If a parser cannot reconstruct where a chunk came from, retrieval quality is only part of the problem.
At generation, context assembly is a second decision
The generator does not need to receive chunks exactly as the retriever stored them. After ranking, a context builder can merge adjacent passages, add a section title, remove overlap, or include a parent excerpt. Record that transformation. A faithful answer can still fail when the correct chunk was retrieved but assembled without its controlling exception.
Evaluate chunking with paired retrieval and answer tests
Build a labeled benchmark from real question shapes: lookup, comparison, multi-part procedure, exception, table query, and cross-section synthesis. For each question, identify the minimum evidence units that should be available. Run the same embedding model, query method, candidate budget, and reranker against each chunking variant so the boundary policy is the variable under test.
Measure whether required evidence appears and where it ranks, then evaluate whether the answer uses that evidence accurately and completely. The RAG evaluation metrics guide explains why retrieval relevance, groundedness, completeness, and task quality should remain separate. The RAG evaluation survey likewise distinguishes retrieval and generation components rather than collapsing the pipeline into one score.
Chunking failure cases worth forcing
- Orphaned qualifier: the rule is retrieved but “except for administrators” remains in the next chunk. Add boundary fixtures around negation, exceptions, and footnotes.
- Header loss: a passage says “retain for seven years” without the section title that names which record type it governs. Prepend or separately carry the section path.
- Duplicate crowding: high overlap returns several versions of the same sentence and displaces complementary evidence. Deduplicate by span and parent before context assembly.
- Parser fiction: a PDF extractor presents columns in the wrong order. No chunk size repairs incorrect source structure; fix extraction and preserve layout evidence.
- Authorization drift: child chunks lose the parent document's tenant or group labels. Inherit and verify permissions during ingestion and again at query time.
- Generator blame: an answer misses a fact that was never retrieved. Diagnose candidate evidence before changing the prompt or model.
A testable implementation sequence
- Inventory document types. Sample every parser path and record headings, tables, lists, images, repeated headers, authorization labels, and update semantics.
- Define the evidence unit. For each question class, state what text and metadata must stay together for a reviewer to interpret the answer.
- Create a deterministic baseline. Split on sentence-safe fixed boundaries, store offsets and parent IDs, and version the parser, splitter, and embedding model.
- Add one structure-aware variant. Respect headings or record boundaries while enforcing a maximum size. Handle oversized tables and sections explicitly instead of silently truncating them.
- Build both indexes from the same snapshot. Hold retrieval and ranking settings constant, then compare required-evidence recall, rank, duplicate rate, assembled context, and final answers.
- Inspect failures, not only aggregates. Classify boundary loss, parser errors, stale data, metadata loss, ranking misses, and generator misuse. Each class points to a different repair.
- Promote the simplest passing policy. Keep the losing variant and fixtures as regression evidence. Reopen the choice when corpus structure, question mix, or model limits change.
Decision summary
Start with structure-aware chunks when document boundaries carry meaning, fixed chunks for a measurable baseline, and semantic splitting only when its added complexity fixes observed retrieval failures. Choose with paired retrieval and answer evaluations, not chunk size folklore.
Primary and official sources
- Microsoft Learn: Chunk large documents for RAG and vector search — official guidance on fixed, variable, semantic, combined, and overlapping chunks.
- Yu et al., “Evaluation of Retrieval-Augmented Generation: A Survey” — a research survey separating retrieval and generation evaluation dimensions.