RAG costs are more predictable than general AI costs because the pipeline is well-defined: ingestion, embedding, storage, retrieval, and generation. Each component has clear cost drivers. Here is the complete cost breakdown for a production RAG system, with ranges from small-scale (100K documents) to large-scale (10M+ documents).
The Cost Components
RAG costs have five components: (1) Ingestion and embedding ($0-$5K one-time): processing documents into chunks and generating embeddings. Cost depends on document count and embedding model used. (2) Vector database ($100-$5,000/month): storage and retrieval of embeddings. Cost scales with document count and query volume. (3) LLM inference ($200-$5,000/month): generation cost per query. Cost depends on model choice, context length, and query volume. (4) Infrastructure ($200-$2,000/month): compute, networking, and monitoring. (5) Maintenance ($500-$5,000/quarter): re-embedding after document updates, index optimisation, quality monitoring.
Scale-Based Cost Ranges
Small scale (100K documents, 1K queries/day): $500-$1,500/month total. Medium scale (1M documents, 10K queries/day): $2,000-$8,000/month total. Large scale (10M+ documents, 100K+ queries/day): $10,000-$50,000/month total. The cost per query decreases significantly at scale: a small-scale system costs $0.05-$0.15 per query; a large-scale system costs $0.003-$0.01 per query. This economies-of-scale effect favours building over buying at higher volumes.
Note
The break-even point for RAG vs SaaS alternatives is typically at 5-10K queries/day. Below that, SaaS is cheaper. Above that, custom RAG becomes cost-effective.
Cost Optimisation Techniques
Five techniques reduce RAG costs: (1) Model routing: use smaller models for simple queries, larger models for complex queries. (2) Semantic caching: cache frequent query results to avoid redundant retrieval and generation. (3) Prompt compression: shorter contexts cost fewer tokens. (4) Retrieval optimisation: retrieve fewer, more relevant documents (improves quality and reduces generation cost). (5) Self-hosted embedding: run your own embedding model instead of using paid APIs — significantly cheaper at scale.
Conclusion
RAG costs are predictable and controllable. The five components (ingestion, vector DB, LLM inference, infrastructure, maintenance) have clear cost drivers. Cost optimisation techniques (model routing, caching, compression, retrieval optimisation, self-hosted embedding) can reduce costs 40-60% without quality degradation.
Key Takeaways
- RAG costs: ingestion, vector DB ($100-$5K/mo), LLM inference ($200-$5K/mo), infrastructure, maintenance
- Cost per query decreases at scale: $0.05-$0.15 (small) vs $0.003-$0.01 (large)
- Break-even vs SaaS is typically at 5-10K queries/day
- Five optimisation techniques: model routing, semantic caching, prompt compression, retrieval optimisation, self-hosted embedding
- Combined savings of 40-60% are typical without quality degradation