LLM Routing Cost Optimization: When Cheaper Models Cost More
LLM routing can reduce inference spending, but a cheaper first model does not guarantee a cheaper completed task. Failed attempts, context reconstruction, verification and operational maintenance all contribute to the final cost. The useful comparison is between workflows that meet the same quality standard, not simply between two advertised token rates.
Current frontier pricing illustrates the problem. Claude Fable 5.1 and GPT-6 Astra list matching base input and output rates in the categories below, yet their cache-read rates differ. That difference can matter in repeated-context workloads without establishing which model is cheaper for your application.
Identical Headline Prices, Different Billing Categories
The following snapshot uses Anthropic's model pricing and OpenAI's API pricing, checked September 10, 2026. These are published rates, not negotiated customer invoices.
| Billing category | Claude Fable 5.1 | GPT-6 Astra |
|---|---|---|
| Base input | $10.00 | $10.00 |
| Cache reads | $0.25 | $1.00 |
| Cache writes | $12.50 for 5-minute caching; $20.00 for 1-hour caching | $12.50 in the listed category |
| Output | $50.00 | $50.00 |
The cache-read rate is four times higher in the Astra column. That is a ratio between specific unit prices, not a fourfold difference in total task cost. Tokenization, generated output, cache eligibility and request configuration can change the comparison. Check the applicable context tier, residency settings and processing mode before estimating production costs.
Prompt Caching Does Not Make Retries Free
Prompt caching can reduce repeated input processing when requests reuse eligible content. OpenAI's prompt-caching documentation describes model-dependent behavior, matching rules and retention. A repeated request is not proof that a cache hit occurred; inspect reported usage.
A retry can still generate a complete new answer, call tools and consume execution time. Even with discounted input, those operations have costs. If an application trims history, summarizes earlier work or retrieves only selected documents, its input growth will also differ from a conversation that resends everything.
When changing models, budget for the destination to process the required context rather than assuming a transferable cache. Sending a concise handoff may reduce input, but the handoff must preserve requirements, evidence and unresolved issues. Losing a critical detail can turn a token saving into additional rework.
Calculate the Routing Break-Even Point
For a simple two-stage cascade, estimate the initial pass, its check and the incremental cost of escalation. The following is a planning model, not a measured vendor benchmark:
Expected routed cost = C_initial + C_check + p_escalate * C_escalation
Break-even escalation share =
(C_direct - C_initial - C_check) / C_escalationC_direct is the average direct-model cost per original task. C_escalation is the average additional cost conditional on escalation, including required context processing and subsequent work. Use comparable accounting boundaries and quality requirements for both paths.
Hypothetical example: direct processing costs $1.00, the initial pass costs $0.10, the check costs $0.05 and an escalation adds $1.50. Break-even is approximately 56.7%. At 30% escalation, routed cost is $0.60; at 70%, it is $1.20.
This simplified example excludes shared engineering overhead and assumes stable averages. Escalated jobs may be harder and longer than typical jobs, so estimate their conditional cost rather than substituting the all-task average. A lower cost is only useful if acceptance quality and latency remain adequate.
The Verification Step Determines Whether Routing Works
A cascade needs a defensible reason to accept a result or escalate it. Some coding tasks have executable checks, but passing a test suite does not establish that every requirement was covered. A plausible summary or customer response can be even harder to grade automatically.
Record false accepts as well as escalations. A system that appears inexpensive because it silently accepts incorrect work is not an optimization. Human review, rollback and customer-impact costs belong in the evaluation where relevant.
The same principle applies to evaluating AI agents in research workflows: completed, accepted outcomes matter more than raw activity. Compare retries and review effort across both the direct and routed paths.
What to Measure Before Adding a Router
- Cost per accepted task: include rejected attempts in the numerator.
- Input categories: separate fresh input, cache writes and cache reads.
- Output and tools: include billed reasoning, execution and external services where applicable.
- Escalation behavior: record frequency, cause and conditional cost.
- Quality and latency: measure incorrect accepts, rework and completion time.
- Maintenance: track changes to prompts, integrations and checks after model updates.
Before building a multi-model system, compare simpler alternatives: remove unnecessary context, stabilize reusable instructions, evaluate batch processing for work that can wait, or select a model by task type before execution begins. Each alternative needs its own quality check.
Optimize the Application, Not Just the Token Rate
Cognativ's view: routing is an architectural choice, not a universal discount. Start with one workload, a direct baseline and explicit acceptance criteria. Expand only when observed savings survive review, latency requirements and maintenance costs.
A software development assessment can identify whether the main constraint is model selection, excessive context, weak validation or integration design. Changing providers will not automatically solve all four.
To evaluate a specific workflow and its operating requirements, discuss your AI application with Cognativ.