Most RAG tutorials show a simple pipeline: chunk documents, embed them, store in a vector database, retrieve on query, generate a response. This works for demos. For production, you need: query rewriting, hybrid search, reranking, context compression, citation extraction, and feedback loops. Here is the complete production RAG architecture.
The Production RAG Pipeline
A production RAG pipeline has eight stages: (1) Query understanding — rewrite the user's query to improve retrieval (expand abbreviations, disambiguate terms, add context). (2) Query routing — decide which knowledge base to search (not all documents are relevant to all queries). (3) Retrieval — retrieve candidates using multiple strategies (semantic search, keyword search, metadata filtering). (4) Reranking — reorder candidates by relevance using a cross-encoder model. (5) Context compression — remove irrelevant portions from retrieved documents to fit the context window. (6) Generation — generate a response with the compressed context. (7) Citation extraction — identify which portions of the response are based on which documents. (8) Quality validation — check the response for hallucinations, completeness, and accuracy.
Note
Most production RAG failures happen at retrieval and reranking. If you retrieve the wrong documents, the best LLM in the world cannot generate a correct response.
Hybrid Search
Semantic search (vector similarity) is good at understanding intent but bad at exact matches. Keyword search (BM25) is good at exact matches but bad at understanding intent. Hybrid search combines both: run semantic and keyword search in parallel, merge the results, and rerank. This catches both: semantic matches ('how do I reset my password' matches a document about 'account recovery') and exact matches (a document containing the exact error code the user mentioned).
The Reranking Step
Initial retrieval (vector search) is fast but imprecise — it returns candidates that are semantically similar but may not be the most relevant. Reranking uses a cross-encoder model that reads the query and each candidate document together, producing a precise relevance score. Reranking is slower (it processes each candidate individually) but dramatically improves precision. The typical pipeline: retrieve 50 candidates via vector search, rerank to the top 10, compress to the top 5 for context. This pipeline achieves significantly higher accuracy than vector search alone.
Conclusion
Production RAG architecture has eight stages, not four. The stages that matter most for quality are query understanding, hybrid search, and reranking. Invest in retrieval quality before investing in LLM quality — the best LLM cannot compensate for retrieved documents that are not relevant.
Key Takeaways
- Production RAG has eight stages: query understanding, routing, retrieval, reranking, compression, generation, citation, validation
- Hybrid search (semantic + keyword) catches both intent and exact matches
- Reranking with cross-encoder models dramatically improves retrieval precision
- Most RAG failures happen at retrieval — invest in retrieval quality before LLM quality
- Retrieve 50 → rerank to 10 → compress to 5 for context