Adaptive Model Routing in Fabric Gateway: Cutting LLM Costs 50–80% on Most Workloads

A gateway sees every kind of request: a one-line lookup, then a multi-step agent workflow, then a quick rewrite. Most teams send all of them to the same model, usually whichever one is set as the default, and that forces a bad choice. Pick the strongest and you overpay for the many requests that didn't need it. Pick a cheap one and you fail on the hard requests users notice most.

So we stopped picking one model, and built adaptive model routing into Fabric Gateway. It decides, per request, which model should answer. On public benchmarks it cuts cost by roughly 50–80% on most workloads for a 1–3 point quality drop.

Most gateways put a single router in front of every provider. Fabric Gateway runs two: one tuned for OpenAI, one for Anthropic. Routing well means judging how hard a request is for a specific provider's models and then calling that provider's own API, and a single shared router can't get both right for every provider. Our classifier's tiers were calibrated on OpenAI behavior, so reusing them for Claude would misjudge difficulty. Claude also needs the Messages API, where tools and extended thinking run together, which lets its cheaper tiers still reason through tool steps. The rest of this post covers how the routing works, what we found while testing it, and what the numbers show.

Adaptive routing results at a glance

Before the detail, here is every benchmark we ran across three model families. Each point is one benchmark, measured against sending every request to that lineup's strongest tier.

Most points sit in a band: half to four-fifths cheaper, a point or two worse. The ones that drift away from that band are where the interesting lessons are, and we'll get to each of them.

How adaptive model routing works in Fabric Gateway: classify once, route once

Our production strategy is called predict, and it's a single-pass router. For each request:

1. A small classifier runs once and labels the request along several dimensions, not just "how long is the answer." Our meta classifier tags capability type (code, reasoning, lookup/recall, translation, tool-use, …), domain, and complexity.
2. Those labels map to a model tier (cheap / medium / strong) via a profile: a capability-and-domain table with a complexity-based escalation for the hardest requests.
3. The tier also fixes the reasoning-effort for that one call, via a deterministic per-tier ladder.
4. The chosen model answers. Once. There is no "look at the answer and retry" loop.

The classifier isn't free, so its cost is counted into every savings figure in this post.

Finding 1: the cheap model should think harder

The obvious way to set reasoning effort is a ramp: low effort on the cheap tier, more as you climb. Our ladder deliberately isn't a simple low→high ramp. The cheap tier runs at high effort, medium at medium, and strong at high.

The reason is something we found while testing: a small model reasoning hard often beats a mid model reasoning lightly. So effort is derived from the tier, not a separate dial the classifier tunes.

Why we run two routers instead of one

Routing a request means two provider-specific things: classifying it, and speaking the provider's wire protocol. Those differ enough between OpenAI and Anthropic that we run two routers over a shared routing core.

Both share the same predict logic (classify once → tier + effort → answer). The split is about fit: a classifier tuned for one provider's models, and the right API for each. Running Anthropic benchmarks through the OpenAI-tuned router would measure the wrong thing, so we don't.

How we benchmarked the router

We route through the live Fabric Gateway, so the plugin makes the real per-request decision, and we ran it over several public benchmarks. The single-turn suites are domainbench, ra_freeform (free-form knowledge), ra_mcq, and RouterBench (general Q&A, reasoning, math, code). The agentic suites are APIFlow (multi-step API workflows) and Terminal-Bench (containerized coding tasks), graded by each suite's own validators with no LLM judge.

The comparison is always the same: the router against sending every request to the strongest tier, on identical tasks. Cost is the delivered, cache-aware cost plus the classifier probe. Single-turn quality is a 0–1 graded score; agentic quality is pass rate. The finer methodology notes are in the appendix at the end.

Benchmark results

OpenAI results: half to four-fifths cheaper

We ran two OpenAI lineups through Adaptive Model Routing Plugin: the shipped v2 (nano → gpt-5.4 → gpt-5.5) and the newer GPT-5.6 (luna → terra → sol). Relative cost is vs that lineup's all-strong baseline; lower = cheaper.

v2 (gpt-5.4 / gpt-5.5)

GPT-5.6 (luna / terra / sol)

On most workloads the router removes half to four-fifths of the cost for a point or two of quality.

Finding 2: the questions that look easy

One benchmark kept breaking the pattern: ra_freeform. It's made of short, knowledge-recall questions, and short questions look easy to the classifier. But they need a strong model's recall, so routing them down costs real quality: −5 to −8 pt on the OpenAI lineups.

That's a classifier-calibration gap, not a model limit, which makes it the clearest lever for the next round of tuning. As you'll see below, it shows up on Claude too.

Finding 3: the strongest model said no

For the Anthropic run we started with Opus 5 as the strong tier. Then its safety classifier refused ~46% of APIFlow's synthetic API-automation tasks, flagging them as "cyber." That crippled the all-Opus baseline before the router had done anything.

Opus 4.8 doesn't refuse them. We verified that with a direct A/B: same prompt, Opus 4.8 answers, Opus 5 refuses. So the whole Anthropic lineup uses Opus 4.8 as the strong tier, which keeps one consistent comparison across every suite.

Anthropic results: same router, a different story

The Claude lineup (Haiku 4.5 → Sonnet 5 → Opus 4.8) runs through the adaptive-model-router over the Messages API. Baseline is all-Opus.

Two things stand out. First, Anthropic saves least on agentic work. Claude's tier prices are compressed and the Opus baseline already uses relatively few output tokens, so there's little headroom. On Terminal-Bench the router leans heavily on Sonnet, so it trims only ~5% of cost for a 3-point quality drop. Of the three families this is the weakest agentic trade-off; OpenAI's lineups cut 45–55% there.

Second, the drop is on the middle, not the extremes. On Terminal-Bench the loss is concentrated on medium tasks (77% → 65%), where the classifier routes down to Sonnet/Haiku but the task still needs Opus. Easy tasks tie, and on hard ones the two are within noise. And ra_freeform shows the same recall gap as OpenAI, amplified.

The lineup matters as much as the router. Same routing logic, very different outcomes, because whether the cheaper tiers can hold quality on hard work is a property of the models, not the router.

Key takeaways for routing LLM traffic

1. Calibrate the classifier for your workload. Most of the remaining quality loss (ra_freeform, hard agentic) is the classifier under-rating difficulty, not a model ceiling.
2. Route for your provider. We run a provider-tuned classifier and the right wire protocol per provider for a reason: tools-plus-reasoning on Claude's Messages API is a different world from OpenAI's.
3. Count the overhead. Classifier (and any judge) calls are part of the bill; every number here includes them.
4. Know your headroom. Cost savings are bounded by the lineup's price spread; compressed tiers (Anthropic) cap the upside before the router does.

What's next for adaptive routing

The weakest results all point the same way. Questions that look easy but need recall, and medium agentic tasks routed one tier too low, are cases of the classifier under-rating difficulty. Calibrating it is the next round of tuning.

Appendix: how we measured

• Routing ran through the live Fabric Gateway; the plugin made the real per-request decision.
• Agentic suites were graded by each suite's own validators, with no LLM judge.
• Cost is the delivered, cache-aware cost plus the classifier probe. Relative cost is against each lineup's all-strong baseline.
• For Anthropic, single-turn is the stratified subset; Terminal-Bench is the 89-task set compared on tasks both arms ran.