Caching is the easiest AI cost optimisation to implement and one of the most effective. Many AI requests are repetitive: the same question asked in slightly different ways, the same classification applied to similar inputs, the same summarisation of the same document. Caching avoids redundant inference for these requests, reducing costs 20-40% with zero quality degradation.
Deterministic Caching
For identical inputs, deterministic caching is straightforward: hash the input (prompt + system prompt + parameters), store the output in a cache (Redis, Memcached), and return the cached output for identical subsequent requests. This works for: classification of the same document, summarisation of the same content, and translation of the same text. Cache hit rates of 10-20% are typical for deterministic caching. Limitation: only catches exact duplicates, not semantically similar requests.
Semantic Caching
Semantic caching goes further: cache inputs and outputs in a vector database, and for new inputs, check if a semantically similar input has been handled before. If a cached response meets quality thresholds (similarity score above a configurable threshold), return it instead of running inference. This catches semantically similar requests ('how do I reset my password' and 'how to change my password'). Cache hit rates of 20-40% are typical for semantic caching.
Pro Tip
Start with deterministic caching (simple, low risk), then add semantic caching for additional savings. Semantic caching requires quality threshold tuning.
Cache Invalidation
Cache invalidation is the hard problem: when should cached responses be discarded? Strategies: (1) TTL-based: cached responses expire after a configurable time (1 hour, 1 day, 1 week). Simple but may serve stale responses. (2) Content-based: invalidate when the underlying data changes (document updated, knowledge base modified). More accurate but requires change detection. (3) Quality-based: periodically re-evaluate cached responses against current quality standards. Catches model degradation. Use TTL-based for most cases; content-based for knowledge-intensive applications.
Conclusion
Caching is the easiest and one of the most effective AI cost optimisations. Start with deterministic caching (10-20% savings), then add semantic caching (20-40% savings). Implement cache invalidation based on your data freshness requirements.
Key Takeaways
- Deterministic caching: exact input matches — 10-20% cache hit rates
- Semantic caching: similar input matches — 20-40% cache hit rates
- Start with deterministic, add semantic for additional savings
- Cache invalidation: TTL-based (simple), content-based (accurate), quality-based (comprehensive)
- Caching is the easiest cost optimisation — implement first