RAG costs have four components: embedding, storage, retrieval, and generation. Each component has optimisation opportunities. The generation step (LLM inference) is typically 60-80% of total RAG cost, so optimising generation has the highest impact. But storage, embedding, and retrieval also have significant optimisation potential. Here are the specific techniques for each RAG pipeline stage.
Embedding Cost Optimisation
Embedding costs scale with document count and embedding model choice. Optimise with: (1) Self-hosted embedding models: run BGE, E5, or GTE locally instead of using paid APIs. At 100K+ documents, self-hosted is 5-10x cheaper. (2) Batch embedding: process multiple documents in a single forward pass. 3-5x more efficient than one-by-one embedding. (3) Incremental embedding: only embed new or changed documents, not full re-embedding. Reduces ongoing embedding costs by 80-90%.
Retrieval Cost Optimisation
Retrieval costs scale with query volume and index size. Optimise with: (1) Index optimisation: HNSW indexes with tuned parameters (ef_construction, m) reduce query time and resource usage. (2) Query caching: cache frequent query results. 20-40% of queries are repetitive. (3) Reduced retrieval: retrieve fewer, more relevant documents (improve retrieval precision). Retrieving 5 instead of 20 documents reduces generation cost proportionally.
Generation Cost Optimisation
Generation (LLM inference) is 60-80% of RAG cost. Optimise with: (1) Model routing: use smaller models for simple RAG queries. (2) Context compression: remove irrelevant content from retrieved documents before injection. Reduces token count 30-50%. (3) Prompt optimisation: shorter system prompts and instructions. (4) Semantic caching: cache query-response pairs. (5) Self-hosted LLM: at high volumes, self-hosted models are 3-5x cheaper than cloud APIs.
Pro Tip
Context compression is the most underutilised optimisation. Removing irrelevant content from retrieved documents reduces generation cost 30-50% with no quality loss.
Conclusion
RAG costs are optimisable at every stage: embedding (self-hosted, batch, incremental), retrieval (index tuning, caching, reduced retrieval), and generation (routing, compression, caching, self-hosted). Generation is 60-80% of cost — optimise there first.
Key Takeaways
- RAG costs: embedding, storage, retrieval, generation — generation is 60-80% of total
- Embedding: self-hosted models, batch processing, incremental embedding
- Retrieval: HNSW tuning, query caching, reduced retrieval (fewer, more relevant documents)
- Generation: model routing, context compression (30-50% token reduction), semantic caching
- Context compression is the most underutilised optimisation — reduce irrelevant content before generation