How to Evaluate an LLM Router: The Buyer's Guide (2026)

Emissary

Home Resources

Playground

13th Aug

5 mins

GUIDE

Executive Summary (TL;DR)

What it is: An LLM router dynamically directs queries to optimal models at runtime based on cost, latency, and capability (unlike AI gateways, which merely unify API access).
Key Metrics: Target <500ms routing overhead, a cost tax of <5% of default inference spend, and track Deflection Rate against output quality.
Core Differentiators: Evaluate prompt cache awareness, task-aware (not just complexity-based) routing, custom pricing support, and model update cadence.

Model routing has quietly become one of the highest-leverage decisions in an AI stack. Every request you send to a frontier model that a mid-tier model could have handled, is margin left on the table; every request sent down-tier that shouldn't have a quality incident. But with high leverage comes noise - "we route to the best model for each query" is the new 'we're an AI company'.

This guide lays out how to evaluate model routers beyond the hype, organizing around four questions every AI buyer (and builder) is thinking through: does it actually work, does it fit your stack, what genuinely sets it apart, and will it be a boon or burden in 6 months?

What Is an Model Router? (And What Isn't One)

An LLM or model router is a service that sits between your application and your model providers. It decides during runtime, which model should handle each request - balancing output quality with cost and latency. An effective model router makes that decision thousands of times a minute: cheaply, invisibly, and correctly.

It's worth separating a router from its most common lookalike: the AI gateway. A gateway unifies access - one API key, one endpoint, shared plumbing for rate limits, retries, and observability across providers, but you still pick the model (usually at compile time). A router flips that decision layer: it picks the model for you. The market blurs the two constantly, and the naming doesn't help. (OpenRouter, perhaps the most inaptly named product, is primarily a gateway).

A quick heuristic:

If the pitch is…	What it actually is
"One API for every model"	Gateway
"We choose the model"	Router

This guide focuses on the router, although both may ship with features of the other.

The Core Question: Does the Routing Work?

Cost vs. Performance Delta: Measuring Model Router Quality

Most companies start at peak provisioned state - everything to the frontier model, the quality ceiling and the cost ceiling at once.

The value of an LLM router is captured in two conflicting numbers:

Metric	Definition
Cost delta	Gross savings on inference spend compared to peak provisioning
Performance delta	Difference in output quality between peak and routed provisioning

A core metric to monitor here is the Deflection Rate, the percentage of tokens routed away from the default peak intelligence model. Routing is a controlled descent from peak-provisioning towards optimal inference allocation.

A naive router posts an impressive deflection rate while hiding a large performance delta; a timid one posts a near-zero cost delta by barely deflecting anything. Look for a high deflection rate (resulting in lower gross spend) at a negligible, or even positive, performance delta - this shows that the router knows when to downsize the intelligence, not just trade quality for cost.

Model Routing Overhead: Latency and Cost

A router is a tax on every single request, paid in two currencies: latency and cost.

If the routing decision adds 2 seconds to a request that ultimately goes to a faster, cheaper model, you've eroded much of the point of routing it there. Ask vendors for p50 and p99 routing latency under production-like load, and ask how the decision itself is made - a small classifier running at the edge has a very different overhead profile than a full LLM call scoring each query. The optimal routers operate at <500ms (p50 200ms) routing overhead.

On cost, the routing decision should be a rounding error relative to the inference it's steering. A good rule of thumb is <5% of default model spend (~$0.15/M tokens). If the router charges takes a percentage of spend or savings, model that against your traffic mix before you're impressed by the headline savings numbers.

The Fit: Will This Router Work in Your Stack?

These are the unglamorous requirements that kill router deployments in practice, and they deserve as much scrutiny as the routing intelligence itself.

Supported Models: Check for Specific Overlap, Not the Logo Wall

A router can only route among models it supports, and the intersection of the vendor's supported list with your approved list is what you're actually buying. If your organization is committed to two providers for compliance reasons and the router's strength is arbitraging across eight, most of its intelligence is inaccessible to you. Many routers also incorporate models at a significant lag from their release, rendering the routing significantly less valuable to mature end users.

Check the overlap explicitly - including specific model versions, regions, and deployment types (public API vs. your cloud provider's hosted endpoints).

Prompt Cache Awareness

Prompt caching is now a first-class cost lever - cached input tokens can cost an order of magnitude less than uncached ones. A router that ignores this will happily "save" you money by moving a request from Model A, where you have a warm cache on a 50k-token system prompt, to Model B, where you'll pay full freight to rebuild it.

A cache-aware router treats cache state as part of the cost function: it knows what's warm where, weighs the true marginal cost of each candidate model, and exhibits stickiness for conversations and agents with long shared prefixes. If a vendor can't explain how their router interacts with prompt caching, assume it doesn't, and discount the savings accordingly.

Custom Pricing and Private Models

Almost no serious buyer pays list price. You may have committed-spend discounts, provisioned throughput, negotiated enterprise rates, or fine-tuned and self-hosted models with an entirely different cost structure.

A router optimizing against public list prices will systematically make the wrong call for you, it might route away from capacity you've already paid for. The router needs to accept your actual price sheet as an input, and ideally support registering custom or private model endpoints as first-class routing targets with their own cost and quality profiles.

Security and Data Privacy

An LLM router sees everything: every prompt, every response, every document and tool call your product sends to a model. That makes it one of the most sensitive components in your stack, and it needs to clear security review before its routing intelligence matters at all.

Ensure that you have clarity on vendor compliance with your security profile, and set clear expectations around data handling. Instead of seeking blanket ZDRs, it is worth noting that data retention may actually be an advantage in improving the routing over time. Where relevant, in-VPC deployments may be worth exploring.

The Edge: Beyond "Easy Query, Cheap Model"

Most first-generation routers share the same mental model: estimate query difficulty, send hard queries to big models and easy queries to small ones. That leaves value on the table.

Task-Aware Routing and the Jagged Frontier

Model capability isn't a single ladder. The frontier is jagged: a mid-sized model may beat a frontier model on SQL generation while trailing badly on legal reasoning; one provider's model may be exceptional at structured extraction and mediocre at creative work.

Complexity-based routing collapses all of this into one difficulty axis and misses it entirely. A router that learns task-conditional performance - which model is genuinely best at this kind of work, not just whether the work is hard - can effectively route to a model that is simultaneously cheaper and better for a given request.

Ask vendors directly whether their routing is purely complexity-based or task-aware, and how they measure per-task model performance. This is where the real headroom in routing lives.

Modality-Aware and Context-Aware Routing

Requests aren't just texts. They can carry images, audio, documents, tool definitions, and wildly varying context lengths, and models differ sharply in how well and how cheaply they handle each.

If your system requires it, ensure that your router is capable of multi-modal routing. At the simplest level, this involves modality sensitive routing: routing only to VLMs for multi-modal queries. At the advanced level - the router itself may be multi-modal to account for the inherent complexity variation in images.

The Horizon: Will It Still Be Right in Six Months?

The model landscape turns over every few months. A new release can invert cost-quality tradeoffs overnight: yesterday's optimal routing policy becomes tomorrow's money leak. That makes a router less like a library you integrate once and more like a service whose value depends on continuous maintenance.

Two checks tell you most of what you need to know:

Model freshness. Compare the supported model list against the newest releases from major providers. How many days behind are they?
Policy freshness. When was the routing logic last retrained or re-benchmarked, and does that happen on a schedule or ad hoc?

A router whose model list is three releases stale, or whose routing policy was last calibrated a model generation ago, is optimizing for a world that no longer exists. Ask how new models get incorporated: is there an evaluation pipeline that characterizes a new model's cost-quality profile within days of release, or does support arrive when a customer complains?

LLM Router Evaluation Checklist

Bring these to every vendor call:

Frequently Asked Questions

What does an LLM router do?

It sits between your application and your model providers and selects, per request, the model that best balances cost, latency, and quality, sending easy or specialized work to cheaper or better-suited models.

What is deflection rate?

The percentage of traffic a router sends to cheaper or smaller models instead of your default frontier model. It only means something alongside a quality measurement: a high deflection rate with degraded outputs is a liability, not a saving.

Does model routing reduce quality?

It can, if the router is naive. That's why evaluation should pair deflection rate with total spend at a quality bar you define, verified via a shadow-mode trial on your own traffic rather than public benchmarks.

How is a router different from an AI gateway?

A gateway typically handles keys, rate limits, retries, and observability across providers; a router adds the decision layer that chooses which model each request should go to. Many products blur the line - evaluate the routing intelligence on its own merits.

The Bottom Line

Evaluate an LLM router the way you'd evaluate any cost-optimization infrastructure: demand measurement over marketing. Insist on a shadow-mode trial against your real traffic, judged by your evals. Verify the overhead is negligible, the deflection and total-spend numbers hold at your quality bar, the model universe and pricing match your reality, and caching is part of the cost model rather than a casualty of it.

Then weigh the differentiators: modality sensitivity and jagged-frontier routing, as signals of where the product is headed, and the update cadence as a signal of whether it will still be pointed at the right target six months from now. The routers worth buying will welcome exactly this scrutiny.