Not every AI request needs the most powerful model. A simple classification task does not need GPT-4. A complex reasoning task does not need a tiny model. Model routing — directing each request to the optimal model based on task complexity, tenant tier, and cost constraints — is the highest-leverage cost optimisation for AI SaaS. Done well, it reduces inference costs 40-60% with no measurable quality degradation.
The Routing Decision
The routing decision considers three factors: (1) Task complexity: simple tasks (classification, extraction) go to small models; complex tasks (reasoning, generation, planning) go to large models. (2) Tenant tier: free tier gets the cheapest model; premium tier gets the best model. (3) Cost budget: if the tenant's inference budget is exhausted, route to a cheaper model or return a cached response. The routing is typically implemented as a classifier that examines the request and assigns it to a model tier.
Routing Strategies
Three routing strategies: (1) Rule-based routing: simple rules based on request length, task type, or tenant tier. Fast to implement, easy to understand, but not adaptive. (2) ML-based routing: a lightweight classifier trained on request features to predict the optimal model. More accurate, adapts to patterns, but requires training data. (3) A/B routing: split traffic between models and measure quality metrics for each. Discovers optimal routing empirically but requires statistical rigor. Start with rule-based, evolve to ML-based as you accumulate routing data.
Pro Tip
The routing classifier itself should be cheap and fast. It runs on every request, so it should add <5ms of latency and <1% of inference cost.
Fallback and Quality Assurance
Model routing introduces a quality risk: routing to a cheaper model may degrade quality. Mitigate with: confidence thresholds (if the cheaper model's confidence is below a threshold, upgrade to a more powerful model), quality sampling (periodically compare outputs between model tiers to detect degradation), and user feedback (collect explicit quality ratings to identify routing failures). The routing system should be conservative: when in doubt, route to the more capable model. The cost of upgrading is small; the cost of a poor response is large.
Conclusion
Model routing is the highest-leverage cost optimisation for AI SaaS. Route by task complexity, tenant tier, and cost budget. Start with rule-based routing, evolve to ML-based. Implement confidence thresholds and quality sampling to prevent degradation.
Key Takeaways
- Route by: task complexity, tenant tier, and cost budget — 40-60% cost reduction typical
- Three strategies: rule-based (simple), ML-based (adaptive), A/B (empirical)
- Start with rule-based, evolve to ML-based as routing data accumulates
- Confidence thresholds: if cheap model is uncertain, upgrade to capable model
- Quality sampling: periodically compare outputs between model tiers to detect degradation