The Routing Pitch Is Seductive
Send cheap questions to gpt-4o-mini. Route hard reasoning to claude-opus-4. Use gemini-1.5-pro for multimodal. A simple classifier at the front door, costs drop 60%, performance stays flat. Ship it.
Except IBM Research just published what happens when you actually try this, and the reality is messier than the pitch deck. They built routing into CodeAct agents running on the AppWorld benchmark and hit three hard truths that most routing discussions quietly ignore.
Routing isn't a model-selection problem. It's a systems optimization problem. And the difference matters.
Cost Isn't What the Pricing Sheet Says
Here's the kicker: across 417 tasks, claude-sonnet-4.6 cost $79 total ($0.19/task). gpt-4.1 cost $155 ($0.37/task)—nearly double.
On paper, this makes zero sense. GPT-4.1's token pricing is lower on both input and output. Sonnet takes roughly three times as many reasoning steps to finish the same tasks. By sticker price alone, GPT should win easily.
The explanation? Caching. Agent workloads reuse massive chunks of context across reasoning steps. When cache hit rates are high, effective input costs collapse. Sonnet's lower cache-read pricing meant it benefited disproportionately from this pattern—enough to overcome both its higher base pricing and its longer trajectories.
Most routing papers treat cost as a static lookup table. In production, actual cost depends on the interaction between the model, the workload, and the serving infrastructure. A router that only looks at pricing sheets is optimizing against the wrong numbers.
Task Difficulty Is Invisible at Routing Time
The intuitive strategy: estimate how hard a task is, send harder tasks to stronger models. Sounds reasonable. Breaks down immediately.
First problem: difficulty is often invisible at the front door. A request like "summarize this contract" looks trivial but might trigger retrieval, compliance checks, tool use, and multiple refinement rounds before it's done. Meanwhile, a highly technical prompt might be handled efficiently by a smaller specialized model. You often don't know how hard a task actually is until execution is underway.
Second problem: even if you could perfectly estimate difficulty, it's only one signal among many. In production, routers need to balance:
- Cost
- Latency
- Model specialization
- Reliability
- Compliance requirements
- Data residency rules
- Privacy constraints
- Approved model lists
Enterprise deployments pile on even more. A task that would ideally go to one model might need to go elsewhere because of governance—and the router has to handle that gracefully, without blowing up the agent's execution plan.
Routers aren't solving one problem. They're constantly juggling cost, quality, latency, compliance, and reliability all at once.
Latency Isn't Just Model Size
The naive take: bigger models are slower, smaller ones are faster. Route accordingly.
What the user actually experiences depends on much more than parameter count.
Routing itself adds overhead. Every decision point introduces latency. Infrastructure factors—which hardware a model is running on, whether the cache is warm, how busy the endpoint is—often dominate end-to-end response times. A theoretically faster model can still produce a slower experience if the serving conditions aren't right.
Then there's routing granularity. Routing once per task adds minimal overhead. But routing at every step—which gives you more flexibility to adapt mid-execution—means every additional decision point introduces latency and operational complexity.
You can't optimize latency without understanding the serving system. A router that ignores infrastructure is optimizing against the wrong reality.
What Actually Works: Optimization, Not Classification
IBM's solution: stop asking "which model is best for this task?" and start asking "what's the best operating point for the entire system?"
They built a router that optimizes across cost, quality, and latency simultaneously—while staying lightweight enough (roughly 6 ms and 2 kB of memory per task) to avoid becoming the bottleneck.
The results on AppWorld with CodeAct agents tell the story. Their router traces out a cost-accuracy frontier—a range of operating points you can choose depending on whether you prioritize cost, latency, or accuracy.
Configuration 1: Latency-Optimized
- 84% accuracy for $93 and 83s
- 21% cost reduction compared to running
opusalone - 9% latency reduction
- Only 4% accuracy drop
Configuration 2: Cost-Optimized
Pushes cost even lower while maintaining reasonable accuracy.
Meanwhile, a standard difficulty-based router lands in a similar accuracy range but at higher cost—it doesn't explore the full tradeoff space the way an optimization-based approach can.
The Real Lesson
Routing isn't really about choosing models. It's about optimizing systems.
Models are one variable—an important one, but just one among caching behavior, infrastructure state, compliance constraints, and workload patterns. When routing works well, it's rarely because it found the "best" model for a given task. It's because it found the best operating point for the entire system.
That's a harder problem than classification. But it's the one worth solving.
What This Means for Builders
If you're building routing into agentic systems:
-
Measure actual cost, not sticker price. Cache hit rates, trajectory length, and serving infrastructure matter more than token pricing tables.
-
Don't route on difficulty alone. You're balancing multiple constraints simultaneously. Difficulty is one signal, not the signal.
-
Model latency ≠ user latency. Routing overhead, infrastructure state, and serving conditions dominate end-to-end response times.
-
Treat routing as optimization, not classification. You want a range of operating points, not a single "best" answer.
IBM says they'll be sharing more technical details in a follow-up post. In the meantime, the takeaway is clear: routing is a systems problem masquerading as a model problem. The sooner you treat it that way, the better your agents will perform in production.