TwinRouterBench: Dual-Track LLM Routing Benchmark

• TwinRouterBench is a dual-track benchmark that evaluates step-level routing policies in agentic large language model systems across both static and dynamic tracks.
• It leverages execution-verified labels and strict arithmetic cost formulas to assess model performance and cost-saving efficiency over diverse workloads.
• The framework supports reproducibility and extensibility by enabling rapid offline iterations alongside live, end-to-end system validation with real API cost tracking.

TwinRouterBench is a comprehensive, dual-track benchmark designed to evaluate step-level routing policies in agentic LLM systems operating over long-horizon trajectories. It addresses the limitations of prior one-shot routing benchmarks by providing both deterministic, offline static evaluation across diverse agent benchmarks and dynamic live harnesses measuring real API usage and task resolution in coding agents. TwinRouterBench anchors policy evaluation in execution-verified labels and strict arithmetic cost formulas rather than LLM- or human-in-the-loop judging, supporting reproducibility and extensibility across models and domains (Yang et al., 14 May 2026).

1. Motivation and Rationale

Contemporary agentic systems involve long, multi-turn workflows in which LLMs interact with evolving, stateful prefixes comprised of chat messages, tool outputs, code edits, and retrieval results. Routing each LLM call to the cheapest sufficient model is essential for cost-sensitive applications, especially where agent trajectories are lengthy and heterogeneous in complexity. Existing router benchmarks typically restrict evaluation to single-turn, one-shot prompts, neglecting (1) the need to expose the actual intermediate state visible to routers at each step, (2) failure propagation across steps, and (3) the necessity of matching downstream trajectory success rather than single-step correctness.

TwinRouterBench resolves these deficits by supplying two evaluation tracks:

• A static track featuring 970 agentic prefixes from 520 instances across five representative workloads (SWE-bench, BFCL, mtRAG, QMSum, PinchBench), each paired with an execution-verified label identifying the cheapest model tier sufficient for successful trajectory completion.
• A dynamic track with a live agent harness running the SWE-bench Verified suite, wherein routers select models per step, and the system logs realized API cost and end-to-end task success using deterministic predicates.

This dual-track design enables both rapid, LLM-independent offline routing policy development and genuine end-to-end system validation.

2. Mathematical Formulation and Label Construction

Let a trajectory be $\tau = (x_1, x_2, \ldots, x_N)$, where $x_i$ encodes the router-visible prefix up to the $i$-th LLM call. The step-level routing policy is formalized as

$\pi : x_i \rightarrow t_i \in \mathcal{T},$

with $$\mathcal{T} = \{\text{low}, \text{mid}, \text{mid\_high}, \text{high}\}$$ ordered by increasing cost and capability, and each tier $t$ mapped to a pool of models $\mathcal{M}_t$.

The ideal but unobservable tier for step $i$ is

$$t_i^* = \min \{ t \in \mathcal{T} : \exists m \in \mathcal{M}_t \ \text{such that} \ V_i(m; x_i) = 1 \},$$

where $V_i(m; x_i)=1$ if model $x_i$0 succeeds on $x_i$1 (approximated for multi-turn tasks by trajectory completion without step count inflation). Since $x_i$2 cannot be computed directly, TwinRouterBench releases an execution-verified estimate $x_i$3 via the downgrade-and-cascade protocol.

Downgrade-and-Cascade Labeling:

For each step, starting from the highest tier, the protocol prunes implausible candidates with strong-model hints and attempts to lock steps to lower tiers while maintaining full trajectory success metrics. For non-low tiers, it tests up to three models per pool, accepting $x_i$4 if any succeed. Manual audits confirm label “tightness,” with only one out of 64 reviewed cases found further downgradeable and subsequently corrected.

3. Static Track: Dataset, Metrics, and Cost Scoring

Data Coverage

The static track comprises 970 annotated rows from successful strong-model executions distributed as follows:

Workload	Instances	Source/License
SWE-bench	40	Code repair (Apache-2.0)
BFCL	130	Function-calling (CC BY 4.0)
mtRAG	193	RAG QA (IBM Research)
QMSum	145	Meeting summarization (MIT)
PinchBench	12	General agent tasks (MIT)

Target tier distribution: 689 low, 62 mid, 49 mid_high, 170 high (Table A.2).

Scoring Formulas and Metrics

Let $x_i$5 be the label, $x_i$6 the router’s prediction, and $x_i$7 the computed tier cost for row $x_i$8.

• Pass/Exact Metrics:
  • $x_i$9
  • $i$0
• Trajectory Metrics:
  • $i$1 if all steps in $i$2 have $i$3.
  • Always-high trajectory cost: $i$4
• CostSave (failure-penalized):

$i$5

$i$6

Combined score: average of RowPass, RowExact, TrajPass, and CostSave.

• Token Billing Formula:

$i$7

where $i$8, $i$9, $\pi : x_i \rightarrow t_i \in \mathcal{T},$0, $\pi : x_i \rightarrow t_i \in \mathcal{T},$1 are input, cache_read, cache_write, output tokens; $\pi : x_i \rightarrow t_i \in \mathcal{T},$2 are per-tier costs. Cache misses on tier switches accrue cache_write costs at the new tier’s rate.

4. Dynamic Track: Live Agent Harness and Evaluation

The dynamic track runs the full 500-case SWE-bench Verified suite, with evaluation reported on a 100-case held-out split disjoint from static supervision examples. At each agent step, the router observes $\pi : x_i \rightarrow t_i \in \mathcal{T},$3 (the prefix), possible models, cache state, and budget, selecting $\pi : x_i \rightarrow t_i \in \mathcal{T},$4 from a locked pool of 11 models (Table A3).

API usage is bucketed by token type (input, cache_read, cache_write, output) per provider logs.

• Per-instance cost:

$\pi : x_i \rightarrow t_i \in \mathcal{T},$5

• Task resolution:

$\pi : x_i \rightarrow t_i \in \mathcal{T},$6, per SWE-bench predicate.

• Penalty for unresolved cases:

$\pi : x_i \rightarrow t_i \in \mathcal{T},$70.60$$(approximate perfect solver cost).</p> <ul> <li><strong>Final bill:</strong></li> </ul> <p>$$\pi : x_i \rightarrow t_i \in \mathcal{T},$8

Leaderboards rank by $\pi : x_i \rightarrow t_i \in \mathcal{T},$9 (lower is better), reporting API and penalty costs separately.</p> <h2 class='paper-heading' id='experimental-results'>5. Experimental Results</h2><h3 class='paper-heading' id='static-track-performance'>Static Track Performance</h3><div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Method</th> <th>RowPass</th> <th>RowExact</th> <th>TrajPass</th> <th>CostSave</th> <th>Combined</th> </tr> </thead><tbody><tr> <td>SR-KNN (in-sample upper)</td> <td>91.9%</td> <td>78.8%</td> <td>84.7%</td> <td>56.2%</td> <td>77.9%</td> </tr> <tr> <td>ClawRouter (rule-based)</td> <td>–</td> <td>–</td> <td>–</td> <td>–</td> <td>52.8%</td> </tr> <tr> <td>UncommonRoute (rule-based)</td> <td>–</td> <td>–</td> <td>–</td> <td>–</td> <td>57.0%</td> </tr> <tr> <td>Always-high (<a href="https://www.emergentmind.com/topics/opus-4-6" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Opus 4.6</a>)</td> <td>–</td> <td>–</td> <td>–</td> <td>–</td> <td>54.4%</td> </tr> </tbody></table></div> <p>SWE-bench accounts for 34.6% of total static score weight and is the major driver of failures and negative CostSave (penalized for failed trajectories). On non-SWE workloads, cost savings frequently exceed 85%.</p> <h3 class='paper-heading' id='dynamic-track-results'>Dynamic Track Results</h3><div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Method</th> <th>Resolved/100</th> <th>API/case</th> <th>Penalty</th> <th>Bill</th> </tr> </thead><tbody><tr> <td>Unrouted Opus 4.6</td> <td>74</td> <td>\$\mathcal{T} = \{\text{low}, \text{mid}, \text{mid\_high}, \text{high}\}$015.60</td> <td>\$70.33 SR-KNN router 75 \$\mathcal{T} = \{\text{low}, \text{mid}, \text{mid\_high}, \text{high}\}$170.61</td> <td></td> <td></td> </tr> <tr> <td>UncommonRoute (trained on static labels)</td> <td>75</td> <td>\$\mathcal{T} = \{\text{low}, \text{mid}, \text{mid\_high}, \text{high}\}$240.66</td> <td></td> <td></td> </tr> <tr> <td>UncommonRoute (rule-based)</td> <td>73</td> <td>\$\mathcal{T} = \{\text{low}, \text{mid}, \text{mid\_high}, \text{high}\}$3188.76

A logistic policy trained on static labels achieves a 53% real-world cost reduction at matched resolution rate compared to the always-high baseline.

6. Implications for Research and System Development

TwinRouterBench’s dual-track design enables rapid, reproducible benchmarking of routing policies with:

• Fast offline iteration: Static scoring requires only milliseconds per policy, supporting massive, LLM-free router search and rapid ablation.
• Live validation: The dynamic SWE harness assesses entire agent behaviors, including tool and cache effects, tracking API spend and actual task resolution rather than proxy measures. This design uncovers phenomena such as mixed-model out-of-distribution drift, which static proxies may overlook.
• Reproducibility and extensibility: The benchmark versions all model pools, pricing constants, and scoring logic. Any model or pricing update can trigger protocol-consistent re-labeling, preserving results comparability.

Exposing the full per-step epistemic state and execution-verified cost/label information sets a rigorous standard for routing evaluation in agentic LLM systems.

7. Prospects for Future Work

TwinRouterBench paves avenues for advancing LLM routing research by:

• Extending to broader tool suites and languages beyond English.
• Jointly optimizing routing and caching policies for further cost-effectiveness.
• Developing evaluation protocols that allocate partial credit for subtasks (e.g., code test coverage).

A plausible implication is that systematic, execution-grounded routing benchmarks like TwinRouterBench will facilitate both efficient LLM utilization and improved transparency in agentic system development. All code, datasets, and detailed methodologies are openly available at https://github.com/CommonstackAI/TwinRouterBench, supporting ongoing experimentation and community contributions (Yang et al., 14 May 2026).