A RAG system can fail in two ways: retrieval can return the wrong documents, or generation can produce a wrong answer even with correct documents. Evaluating RAG requires measuring both: retrieval quality (did the system find the right information?) and generation quality (did the system produce a correct, grounded answer?). Most RAG evaluations focus only on generation. This misses the most common failure mode: bad retrieval.
Retrieval Evaluation Metrics
Retrieval quality is measured by: Precision@K (of the top K retrieved documents, how many are relevant?), Recall@K (of all relevant documents, how many did the system retrieve?), Mean Reciprocal Rank (how high is the first relevant document in the results?), and Normalised Discounted Cumulative Gain (ranking quality, accounting for position). These metrics require a ground-truth dataset: questions paired with the documents that should be retrieved. Build this dataset from your most common queries and their expected answers.
Generation Evaluation Metrics
Generation quality is measured by: Faithfulness (is the answer grounded in the retrieved documents, or is it hallucinated?), Relevance (does the answer address the question?), Completeness (does the answer cover all aspects of the question?), and Citation accuracy (are the cited documents actually the source of the answer?). Use LLM-as-judge evaluation: prompt a separate LLM to evaluate the answer against the retrieved documents and score each dimension.
Pro Tip
Faithfulness is the most important generation metric. An irrelevant answer is annoying; a hallucinated answer is dangerous.
The End-to-End Evaluation Pipeline
Build an end-to-end evaluation pipeline: (1) Create a test dataset of 100+ questions with expected answers and expected retrieved documents. (2) Run each question through the RAG pipeline. (3) Evaluate retrieval quality (precision, recall, MRR). (4) Evaluate generation quality (faithfulness, relevance, completeness). (5) Compare against quality thresholds. (6) Run the pipeline after every change to the RAG system (chunking, embedding, retrieval, generation). This pipeline catches regressions automatically and tracks quality over time.
Conclusion
RAG evaluation requires measuring both retrieval quality and generation quality. Retrieval metrics (precision, recall, MRR) catch the most common failures. Generation metrics (faithfulness, relevance) catch the remaining failures. Build the evaluation pipeline as a living system that runs continuously.
Key Takeaways
- Measure both retrieval quality (precision, recall, MRR) and generation quality (faithfulness, relevance)
- Faithfulness is the most important generation metric — hallucinated answers are dangerous
- Build a 100+ question test dataset with expected answers and expected retrieved documents
- Run the evaluation pipeline after every change — catch regressions automatically
- Retrieval failures are more common than generation failures — measure retrieval first