Large language models know what is on the public internet. They do not know what is in your company's knowledge base, your internal documentation, your customer support history, or your proprietary data. Retrieval-Augmented Generation (RAG) bridges this gap: it retrieves relevant information from your private data and provides it to the LLM as context for generating accurate, grounded responses. RAG has become the default architecture for enterprise AI because it solves the fundamental limitation of LLMs: they do not know what they were not trained on.
How RAG Works
The RAG pipeline has four steps: (1) Ingestion — documents are chunked, converted to embeddings (numerical vectors), and stored in a vector database. (2) Retrieval — when a user asks a question, the question is converted to an embedding, and the vector database returns the most semantically similar document chunks. (3) Augmentation — the retrieved chunks are injected into the LLM's prompt as context, alongside the user's question. (4) Generation — the LLM generates a response based on both the question and the retrieved context. The response is grounded in your data, not the LLM's training data.
Why RAG Over Fine-Tuning
Fine-tuning modifies the LLM's weights to include your data. RAG provides your data as context without modifying the model. RAG is almost always preferred because: it is cheaper (no training compute), faster to update (add documents, not retrain), more transparent (you can see exactly which documents the response is based on), more auditable (you can cite sources), and more flexible (change the knowledge base without changing the model). Fine-tuning is only preferable when you need the model to learn a new behaviour or style, not just new facts.
Pro Tip
Use RAG for new facts and knowledge. Use fine-tuning for new behaviours and styles. Most enterprise use cases need RAG, not fine-tuning.
The Chunking Strategy
How you chunk documents determines retrieval quality. Bad chunking = irrelevant retrieval = bad responses. Three chunking strategies: (1) Fixed-size chunks (500-1000 tokens) — simple, fast, but may split sentences and lose context. (2) Semantic chunks — split at natural boundaries (paragraphs, sections) — preserves context but requires document structure. (3) Hierarchical chunks — chunks at multiple granularities (paragraph, section, document) — enables retrieval at the right level of detail. Start with semantic chunks and adjust based on retrieval quality metrics.
When RAG Is the Right Choice
RAG is the right choice when: you need the LLM to reference private or proprietary data, your data changes frequently (RAG updates are instant, fine-tuning takes hours/days), you need source attribution (RAG can cite which documents the response is based on), accuracy is critical (RAG reduces hallucination by grounding responses in real data), or you have a large knowledge base that exceeds the LLM's context window.
Conclusion
RAG is the bridge between LLMs and private knowledge. It retrieves relevant documents, provides them as context, and generates grounded responses. For most enterprise AI use cases, RAG is the right architecture: cheaper, faster to update, more transparent, and more auditable than fine-tuning.
Key Takeaways
- RAG retrieves relevant documents and provides them as LLM context — it grounds responses in your data
- Four steps: ingestion (chunk + embed), retrieval (semantic search), augmentation (inject context), generation
- Use RAG for new facts; use fine-tuning for new behaviours — most enterprises need RAG
- Chunking strategy determines retrieval quality — start with semantic chunks
- RAG is cheaper, faster to update, more transparent, and more auditable than fine-tuning