One Model For Everything Is A Pricing Decision You Didn't Make

That is the question I'd start with, because it exposes the same design flaw from the money end rather than the architecture end. Most production assistants ship with a single model behind a single endpoint, and then the traffic arrives. Some of it is "what time do you close on a Sunday." Some of it is "reconcile these two ledgers, explain the variance, and tell me which line items to query." Both go to the same place, at the same price, through the same prompt.

That arrangement can never be correct. If the model is strong enough for the reconciliation, you are massively overpaying for the opening hours. If it is cheap enough for the opening hours, the reconciliation is quietly wrong and nobody notices until a client does.

Routing is not an optimisation you bolt on later. It is the layer where your unit economics and your accuracy ceiling are both set.

Tier by what the work actually is

The instinct is to route on surface features — message length, word count, whether a question mark is present. This fails immediately. "Should I hedge?" is four characters longer than "Are you open?" and about four orders of magnitude harder.

Route on the shape of the work:

Tier 0 is the one teams skip, and it is usually the largest slice of live traffic. A support assistant with any real volume will see the same forty questions over and over. Those should never reach a model. A normalised-string lookup, then an embedding-similarity check above a tuned threshold, will absorb a large fraction of requests for effectively nothing. Every request you resolve here is one you never pay for again.

Make the decision itself cheap

The trap in routing is spending on the decision what you saved on the answer. If your router is a frontier-model call that reads the message and returns a tier, you have added latency and cost to every single request in order to avoid latency and cost on some of them.

Build it as a cascade, cheapest stage first, and let each stage terminate early:

1. Deterministic match. Cache and pattern rules. Microseconds, no model.
2. Semantic match. Embed the query, compare against known intents and prior resolved answers. Single-digit milliseconds.
3. Trained classifier. A small fine-tuned or distilled model returning {intent, tier, confidence} in tens of milliseconds.
4. Model-based routing. Only for the residual traffic that the first three stages genuinely can't place.

If stage four is handling more than a small minority of your requests, your intent taxonomy is wrong, not your router.

Design the escalation path before the happy path

The uncomfortable truth about any classifier is that it will misroute, and the two failure directions are not symmetrical. Sending a trivial question to the expensive tier costs you a few pennies. Sending a financial analysis to a small model costs you a confidently wrong number in front of a customer.

So build the cheap path to fail upward. Promote a request to the next tier when:

• Classifier confidence falls below threshold
• The cheap model declines, hedges, or returns a refusal
• Output validation fails — schema mismatch, missing citation, a number that no tool produced
• The user pushes back, rephrases, or repeats themselves

Escalating a minority of requests after the fact is dramatically cheaper than routing all of them to the top pre-emptively. And a system with an escalation path is one you can tune aggressively downward, because the cost of being wrong is a retry rather than a bad answer.

Every tier needs its own contract

Once you have tiers, resist the urge to share one prompt across them. A tier is not just a model choice — it is a full execution contract:

• Its own system prompt. The FAQ tier should be constrained to the point of being boring. The analysis tier needs room to plan.
• Its own tools. Analysis gets the calculator, the query engine, the market data feed. FAQ gets nothing. Tool access is the single largest lever on both blast radius and cost.
• Its own timeout and token budget. A 400ms budget on tier 1 is a feature. Applying it to tier 2 just guarantees truncated reasoning.
• Its own eval set. This is the part that gets skipped. A tier without an eval suite is a tier you cannot safely change models on.

For anything quantitative, one rule overrides the rest: figures come out of tools, never out of the model's parameters. The reasoning tier's job is to decide which calculation to run and to explain the result — not to produce the digits.

Put a ceiling on the expensive tier

An escalation path is also an attack surface. If a user can reliably trigger promotion to tier 2 by phrasing things a certain way, they can trigger it a thousand times.

Contain it: per-user and per-session rate limits on tier 2, a hard daily spend ceiling with defined degradation behaviour, and asynchronous handling for genuinely long analyses. Long jobs should be queued with streamed progress rather than held open on a synchronous request that will time out at the load balancer anyway.

Instrument the router, not just the models

Model-level metrics won't show you a routing problem. You need to watch the decision layer directly:

• Cache hit rate, and how it moves as your corpus and traffic shift
• Misroute rate in both directions — under-routed and over-routed, tracked separately, because they mean different things
• Escalation rate. A router that never escalates is not efficient; it is silently returning weak answers.
• Cost per resolved conversation, not cost per API call. The call-level number rewards you for cheap failures that cause three follow-ups.
• p95 latency per tier, since a blended average across tiers is meaningless.

Add a rolling review of what actually lands in the ambiguous bucket. That bucket is your roadmap — it tells you which new intent to promote into a rule, and which one deserves a tier of its own.

The point

A single model in front of mixed traffic is a decision to overpay for your easy requests, underserve your hard ones, or both. Routing is where you stop treating every question as if it costs the same to answer well — because it doesn't, and your customers can tell.

Related reading: Making Architectural Decisions to Optimise LLM Cost, 10 LLM Cost Explosion Traps, and AI Pricing Tips: How to Control AI Costs Effectively.