# Adaptive Model Routing in Fabric Gateway: Cutting LLM Costs 50–80% on Most Workloads

> How adaptive model routing in Fabric Gateway classifies each request once and routes it to a cheap, medium or strong model tier, cutting LLM costs 50–80% for a 1–3 point quality drop, with benchmark results for OpenAI and Anthropic.

Fabric Gateway, from Postman. This is the Markdown version of https://fabricgateway.ai/blog/adaptive-model-routing-in-fabric-gateway/.

By [Arpit Kumar](https://www.linkedin.com/in/arpitsr/) and [Anand Keshavan](https://www.linkedin.com/in/anand-kumar-keshavan-18829a1/) · October 13, 2026 · 7 min read

A gateway sees every kind of request: a one-line lookup, then a multi-step agent workflow, then a quick rewrite. Most teams send all of them to the same model, usually whichever one is set as the default, and that forces a bad choice. Pick the strongest and you overpay for the many requests that didn't need it. Pick a cheap one and you fail on the hard requests users notice most.

So we stopped picking one model, and built **adaptive model routing** into **Fabric Gateway**. It decides, per request, which model should answer. On public benchmarks it cuts cost by **roughly 50–80%** on most workloads for a **1–3 point** quality drop.

Most gateways put a single router in front of every provider. Fabric Gateway runs **two**: one tuned for OpenAI, one for Anthropic. Routing well means judging how hard a request is for a specific provider's models and then calling that provider's own API, and a single shared router can't get both right for every provider. Our classifier's tiers were calibrated on OpenAI behavior, so reusing them for Claude would misjudge difficulty. Claude also needs the Messages API, where tools and extended thinking run together, which lets its cheaper tiers still reason through tool steps. The rest of this post covers how the routing works, what we found while testing it, and what the numbers show.

## Adaptive routing results at a glance

Before the detail, here is every benchmark we ran across three model families. Each point is one benchmark, measured against sending every request to that lineup's strongest tier.

Each point is one benchmark. Further right is cheaper; higher is closer to (or above) the all-strong baseline. Hover a point for its values; the tables below list every one.

| Benchmark | v2 (gpt-5.4 / gpt-5.5): quality change | v2 (gpt-5.4 / gpt-5.5): cost reduction | GPT-5.6 (luna / terra / sol): quality change | GPT-5.6 (luna / terra / sol): cost reduction | Anthropic (Claude): quality change | Anthropic (Claude): cost reduction |
| --- | --- | --- | --- | --- | --- | --- |
| domainbench | −1.5 pt | 73% | −2.9 pt | 68% | +0.9 pt | 58% |
| ra_freeform | −5 pt | 68% | −8 pt | 81% | −5.9 pt | 50% |
| ra_mcq | −1.3 pt | 72% | −2 pt | 60% | −1.5 pt | 66% |
| routerbench | −5.4 pt | 64% | −1.3 pt | 54% | −0.8 pt | 45% |
| APIFlow | −2 pt | 77% | −3 pt | 68% | −1 pt | 33% |
| Terminal-Bench | −2 pt | 46% | −6 pt | 53% | −3 pt | 5% |

Most points sit in a band: half to four-fifths cheaper, a point or two worse. The ones that drift away from that band are where the interesting lessons are, and we'll get to each of them.

## How adaptive model routing works in Fabric Gateway: classify once, route once

Our production strategy is called `predict`, and it's a single-pass router. For each request:

1. A small **classifier** runs once and labels the request along **several dimensions**, not just "how long is the answer." Our meta classifier tags **capability type** (code, reasoning, lookup/recall, translation, tool-use, …), **domain**, and **complexity**.
2. Those labels map to a **model tier** (cheap / medium / strong) via a profile: a capability-and-domain table with a complexity-based escalation for the hardest requests.
3. The tier also fixes the **reasoning-effort** for that one call, via a deterministic per-tier ladder.
4. The chosen model answers. **Once.** There is no "look at the answer and retry" loop.

The classifier isn't free, so its cost is counted into every savings figure in this post.

### Finding 1: the cheap model should think harder

The obvious way to set reasoning effort is a ramp: low effort on the cheap tier, more as you climb. Our ladder deliberately isn't a simple low→high ramp. The **cheap tier runs at high effort**, medium at medium, and strong at high.

The reason is something we found while testing: a small model reasoning hard often beats a mid model reasoning lightly. So effort is derived from the tier, not a separate dial the classifier tunes.

## Why we run two routers instead of one

Routing a request means two provider-specific things: *classifying* it, and *speaking the provider's wire protocol*. Those differ enough between OpenAI and Anthropic that we run two routers over a shared routing core.

|  | Adaptive Model Routing Plugin | adaptive-model-router |
| --- | --- | --- |
| Used for | OpenAI lineups (GPT-5.x) | Anthropic (Claude) |
| Classifier | the **meta** classifier, a taxonomy **tuned for OpenAI** routing | multi-provider classifier (OpenAI nano as the scorer) |
| Wire protocol | OpenAI **Chat Completions** and **Responses** APIs | Anthropic **Messages** API (`/v1/messages`) |
| Why it's separate | the meta taxonomy and its tier profiles were calibrated on OpenAI behavior | Claude needs the Messages protocol, where **tools + extended thinking run together**, so cheaper Claude tiers can still reason on tool steps; and its own cheap/medium/strong lineup |

Both share the same `predict` logic (classify once → tier + effort → answer). The split is about *fit*: a classifier tuned for one provider's models, and the right API for each. Running Anthropic benchmarks through the OpenAI-tuned router would measure the wrong thing, so we don't.

## How we benchmarked the router

We route through the live Fabric Gateway, so the plugin makes the real per-request decision, and we ran it over several public benchmarks. The single-turn suites are domainbench, ra_freeform (free-form knowledge), ra_mcq, and **RouterBench** (general Q&A, reasoning, math, code). The agentic suites are **APIFlow** (multi-step API workflows) and **Terminal-Bench** (containerized coding tasks), graded by each suite's own validators with no LLM judge.

The comparison is always the same: the router against **sending every request to the strongest tier**, on identical tasks. Cost is the delivered, cache-aware cost **plus** the classifier probe. Single-turn quality is a 0–1 graded score; agentic quality is pass rate. The finer methodology notes are in the appendix at the end.

## Benchmark results

### OpenAI results: half to four-fifths cheaper

We ran two OpenAI lineups through **Adaptive Model Routing Plugin**: the shipped **v2** (nano → gpt-5.4 → gpt-5.5) and the newer **GPT-5.6** (luna → terra → sol). Relative cost is vs that lineup's all-strong baseline; lower = cheaper.

#### v2 (gpt-5.4 / gpt-5.5)

| Benchmark | Baseline quality | Meta quality | Quality Δ | Cost reduction |
| --- | --- | --- | --- | --- |
| domainbench | 0.835 | 0.820 | −1.5 pt | 73% |
| ra_freeform | 0.590 | 0.540 | −5.0 pt | 68% |
| ra_mcq | 0.866 | 0.853 | −1.3 pt | 72% |
| routerbench | 0.950 | 0.896 | −5.4 pt | 64% |
| APIFlow | 89% | 87% | −2 pt | 77% |
| Terminal-Bench | 75% | 73% | −2 pt | 46% |

#### GPT-5.6 (luna / terra / sol)

| Benchmark | Baseline quality | Meta quality | Quality Δ | Cost reduction |
| --- | --- | --- | --- | --- |
| domainbench | 0.840 | 0.811 | −2.9 pt | 68% |
| ra_freeform | 0.604 | 0.524 | −8.0 pt | 81% |
| ra_mcq | 0.869 | 0.849 | −2.0 pt | 60% |
| routerbench | 0.930 | 0.917 | −1.3 pt | 54% |
| APIFlow | 88% | 85% | −3 pt | 68% |
| Terminal-Bench | 84% | 78% | −6 pt | 53% |

On most workloads the router removes half to four-fifths of the cost for a point or two of quality.

Quality change (left) against cost reduction (right) for each benchmark, both measured against that lineup's all-strong baseline.

| Benchmark | v2 (gpt-5.4 / gpt-5.5): quality change | v2 (gpt-5.4 / gpt-5.5): cost reduction | GPT-5.6 (luna / terra / sol): quality change | GPT-5.6 (luna / terra / sol): cost reduction |
| --- | --- | --- | --- | --- |
| domainbench | −1.5 pt | 73% | −2.9 pt | 68% |
| ra_freeform | −5 pt | 68% | −8 pt | 81% |
| ra_mcq | −1.3 pt | 72% | −2 pt | 60% |
| routerbench | −5.4 pt | 64% | −1.3 pt | 54% |
| APIFlow | −2 pt | 77% | −3 pt | 68% |
| Terminal-Bench | −2 pt | 46% | −6 pt | 53% |

### Finding 2: the questions that look easy

One benchmark kept breaking the pattern: **ra_freeform**. It's made of short, knowledge-recall questions, and short questions *look* easy to the classifier. But they need a strong model's recall, so routing them down costs real quality: −5 to −8 pt on the OpenAI lineups.

That's a classifier-calibration gap, not a model limit, which makes it the clearest lever for the next round of tuning. As you'll see below, it shows up on Claude too.

### Finding 3: the strongest model said no

For the Anthropic run we started with Opus 5 as the strong tier. Then its safety classifier **refused ~46% of APIFlow's synthetic API-automation tasks**, flagging them as "cyber." That crippled the all-Opus baseline before the router had done anything.

Opus 4.8 doesn't refuse them. We verified that with a direct A/B: same prompt, Opus 4.8 answers, Opus 5 refuses. So the whole Anthropic lineup uses Opus 4.8 as the strong tier, which keeps one consistent comparison across every suite.

### Anthropic results: same router, a different story

The Claude lineup (Haiku 4.5 → Sonnet 5 → **Opus 4.8**) runs through the **adaptive-model-router** over the Messages API. Baseline is all-Opus.

| Benchmark | Baseline | Meta | Quality Δ | Cost reduction |
| --- | --- | --- | --- | --- |
| domainbench | 0.788 | 0.797 | +0.9 pt | 58% |
| ra_freeform | 0.485 | 0.426 | −5.9 pt | 50% |
| ra_mcq | 0.855 | 0.840 | −1.5 pt | 66% |
| routerbench | 0.947 | 0.939 | −0.8 pt | 45% |
| Terminal-Bench | 66% | 63% | −3 pt | 5% |
| APIFlow | 88% | 87% | −1 pt | 33% |

Two things stand out. First, **Anthropic saves least on agentic work.** Claude's tier prices are compressed and the Opus baseline already uses relatively few output tokens, so there's little headroom. On Terminal-Bench the router leans heavily on Sonnet, so it trims only ~5% of cost for a 3-point quality drop. Of the three families this is the weakest agentic trade-off; OpenAI's lineups cut 45–55% there.

Second, **the drop is on the middle, not the extremes.** On Terminal-Bench the loss is concentrated on *medium* tasks (77% → 65%), where the classifier routes down to Sonnet/Haiku but the task still needs Opus. Easy tasks tie, and on hard ones the two are within noise. And ra_freeform shows the same recall gap as OpenAI, amplified.

> **The lineup matters as much as the router.** Same routing logic, very different outcomes, because whether the cheaper tiers can hold quality on hard work is a property of the models, not the router.

## Key takeaways for routing LLM traffic

1. **Calibrate the classifier for your workload.** Most of the remaining quality loss (ra_freeform, hard agentic) is the classifier under-rating difficulty, not a model ceiling.
2. **Route for your provider.** We run a provider-tuned classifier and the right wire protocol per provider for a reason: tools-plus-reasoning on Claude's Messages API is a different world from OpenAI's.
3. **Count the overhead.** Classifier (and any judge) calls are part of the bill; every number here includes them.
4. **Know your headroom.** Cost savings are bounded by the lineup's price spread; compressed tiers (Anthropic) cap the upside before the router does.

## What's next for adaptive routing

The weakest results all point the same way. Questions that look easy but need recall, and medium agentic tasks routed one tier too low, are cases of the classifier under-rating difficulty. Calibrating it is the next round of tuning.

## Appendix: how we measured

- Routing ran through the live Fabric Gateway; the plugin made the real per-request decision.
- Agentic suites were graded by each suite's own validators, with no LLM judge.
- Cost is the delivered, cache-aware cost plus the classifier probe. Relative cost is against each lineup's all-strong baseline.
- For Anthropic, single-turn is the stratified subset; Terminal-Bench is the 89-task set compared on tasks both arms ran.

---

- [Homepage](https://fabricgateway.ai/)
- [Pricing](https://fabricgateway.ai/pricing/)
- [Book a demo](https://fabricgateway.ai/book-demo/)
- [Documentation](https://docs.fabricgateway.ai)
- [Sign up and log in](https://app.fabricgateway.ai)
- [Site summary for AI assistants](https://fabricgateway.ai/llms.txt)