Model routing is the practice of directing each AI request to the optimal model based on task complexity, cost constraints, and quality requirements. Instead of using one model for everything, you maintain a portfolio of models and route each request to the model that provides the best quality at the lowest cost. This is the single highest-leverage cost optimisation for AI systems.
The Routing Architecture
The model routing architecture has four components: (1) Router: a lightweight classifier that examines the request and assigns it to a model tier. (2) Model registry: a catalog of available models with their capabilities, costs, and latency characteristics. (3) Model pool: the actual model instances (cloud APIs or self-hosted). (4) Feedback loop: quality metrics that inform routing decisions. The router runs on every request, so it must be fast (<5ms) and cheap (<1% of inference cost).
Routing Strategies
Three routing strategies: (1) Rule-based: simple rules based on request length, task type, or user tier. Fast to implement, easy to understand. (2) ML-based: a lightweight classifier trained on request features to predict the optimal model. More accurate, adapts to patterns. (3) Confidence-based: try a cheap model first; if confidence is low, upgrade to a more expensive model. Maximises cost savings while maintaining quality. Start with rule-based, evolve to ML-based as routing data accumulates.
Measuring Routing Quality
Routing quality is measured by: cost per query (are you using cheaper models for appropriate tasks?), quality score (are routed responses meeting quality thresholds?), upgrade rate (how often does the confidence-based fallback trigger?), and latency distribution (are routed requests meeting latency SLAs?). These metrics should be tracked per-route (which model tier handles which task types) and per-tenant (which tenants benefit most from routing).
Note
The upgrade rate is the key metric for confidence-based routing. If it exceeds 20%, your cheap model tier is too aggressive — relax the confidence threshold.
Conclusion
Model routing is the highest-leverage cost optimisation for AI systems. Route by task complexity, tenant tier, and confidence. Start with rule-based routing, evolve to ML-based. Measure cost, quality, upgrade rate, and latency to ensure routing is effective.
Key Takeaways
- Model routing: direct each request to the optimal model — 40-60% cost reduction
- Architecture: router + model registry + model pool + feedback loop
- Three strategies: rule-based (simple), ML-based (adaptive), confidence-based (try cheap, upgrade)
- Key metrics: cost/query, quality score, upgrade rate, latency distribution
- Start with rule-based, evolve to ML-based as routing data accumulates