Chunking is the most underappreciated factor in RAG quality. A perfectly fine-tuned embedding model, a state-of-the-art LLM, and a well-configured vector database will all produce poor results if the chunking strategy is wrong. Bad chunking splits sentences mid-way, separates related content, and creates chunks that lack context. The result: retrieval returns irrelevant chunks, and the LLM generates incorrect answers. Here are the chunking strategies that improve retrieval quality 40%+ over naive approaches.
Why Chunking Matters
Each chunk becomes a single unit of retrieval. If a chunk contains a sentence split mid-way, the embedding loses meaning. If a chunk is too large, it dilutes the relevant content with irrelevant content. If a chunk is too small, it lacks the context needed for semantic similarity. The chunking strategy determines: what information each chunk contains, how well the chunk's embedding represents its content, and how many chunks are needed to answer a typical question. Get chunking right, and everything downstream improves.
Strategy 1: Semantic Chunking
Split at natural document boundaries: paragraphs, sections, headings, and list items. Each chunk represents a complete thought or topic. This preserves context and produces embeddings that accurately represent the chunk's content. Implementation: parse the document structure (Markdown, HTML, or plain text), split at heading boundaries and paragraph breaks, and merge small adjacent chunks to meet minimum size requirements. Semantic chunking achieves 20-40% better retrieval than fixed-size chunking.
Strategy 2: Hierarchical Chunking
Create chunks at multiple granularities: paragraph-level (for precise retrieval), section-level (for broader context), and document-level (for summarisation). At query time, retrieve at the paragraph level first, and if the context is insufficient, fall back to the section level. This gives you the precision of small chunks with the context of large chunks. The trade-off: 3x the storage and embedding cost, but significantly better retrieval quality.
Pro Tip
Hierarchical chunking with paragraph-level retrieval and section-level fallback achieves the best retrieval quality for complex questions that span multiple paragraphs.
Strategy 3: Contextual Chunking
Add context to each chunk before embedding: prepend the document title, section heading, and any relevant metadata. Example: instead of chunk 'We recommend PostgreSQL for vector storage', the chunk becomes 'Document: RAG Architecture Guide / Section: Database Selection / Content: We recommend PostgreSQL for vector storage'. The additional context improves embedding quality and retrieval precision, especially when chunks from different documents are semantically similar.
Conclusion
Chunking determines retrieval quality. Semantic chunking (split at natural boundaries) is the best default. Hierarchical chunking adds precision at the cost of complexity. Contextual chunking improves embedding quality by adding document context. Start with semantic chunking and iterate based on retrieval metrics.
Key Takeaways
- Chunking is the most underappreciated factor in RAG quality — get it right first
- Semantic chunking (paragraph/section boundaries) achieves 20-40% better retrieval than fixed-size
- Hierarchical chunking (multiple granularities) gives precision + context at 3x storage cost
- Contextual chunking (prepend document title/heading) improves embedding quality
- Start with semantic chunking, iterate based on retrieval precision and recall metrics