CodeRouterBench: Adaptive Routing in Coding Tasks

• CodeRouterBench is a benchmark that formalizes model routing as an online decision problem, selecting cost-effective LLMs per task.
• It evaluates diverse routing policies across 10,111 coding tasks using both execution-verified and proxy metrics over single-turn and agentic programming dimensions.
• Empirical findings show that adaptive routers with online feedback, like ACRouter, significantly reduce cumulative regret compared to static policies.

to=arxiv_search เงินไทยฟรี 2^^^^2query2query2^^^^ 72 {"^^^^^^^^2query2 OR \2^^^^^^^^"Agent-as-a-Router: Agentic Model Routing for Coding Tasks\"","max_results":5,"sort_by":"submittedDate"} to=arxiv_search 天天爱彩票网站 凤凰大参考 2^^^^2query2query2^^^^ 72 {"^^^^^^^^2query2 routing LLM","max_results":5,"sort_by":"relevance"} CodeRouterBench is a streaming evaluation environment for model routing in coding tasks, introduced alongside the Agent-as-a-Router framework in "Agent-as-a-Router: Agentic Model Routing for Coding Tasks" (&&&^^^^2query2^^^^&&&). It is designed to measure how well a routing policy assigns each coding task to the most cost-effective LLM in a multi-model setting where performance and pricing vary by task dimension. The benchmark is motivated by three observations: real-world users often have access to multiple LLMs from different providers; no single LLM dominates all coding dimensions; and existing routers typically treat routing as a static, one-shot classification problem rather than a sequential decision process that can exploit execution-grounded feedback (&&&^^^^2query2^^^^&&&).

^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^. Problem setting and motivation

CodeRouterBench formalizes model routing for coding as an online decision problem over a stream of tasks. Its core target is not raw model quality in isolation, but the quality of the routing policy that selects among models with heterogeneous capabilities and costs. The benchmark therefore focuses on a practical regime in which users can choose from multiple frontier LLMs, including open-source and API-based systems, whose relative advantages depend on the task dimension, such as code generation, bug fixing, or test generation (&&&^^^^2query2^^^^&&&).

A central premise is that routing performance is constrained by information deficit. In the underlying study, simply augmenting a vanilla LLM router with performance statistics at the task-dimension level yields a ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^5.3% relative gain and surpasses a heuristic router built on the same dimension-level priors (&&&^^^^2query2^^^^&&&). This motivates the broader agentic formulation in which routing improves by accumulating deployment experience rather than relying solely on static priors or frozen training data.

The benchmark is also explicitly designed to show that an oracle selecting the best model per task can outperform any fixed-model strategy by tens of percent. This establishes a nontrivial headroom for routing and justifies a regret-based protocol in which routers are compared against a per-task oracle rather than against a single model baseline alone (&&&^^^^2query2^^^^&&&).

2. Dataset composition and task taxonomy

CodeRouterBench comprises ^^^^2(Zhou et al., 22 Jun 2026) OR \2query2^^^^,^^^^2(Zhou et al., 22 Jun 2026) OR \2(Zhou et al., 22 Jun 2026) OR \2(Zhou et al., 22 Jun 2026) OR \2^^^^ tasks. It is partitioned into a probing set for development, an in-distribution test set, and an out-of-distribution agentic programming test set. The probing set contains 7,^^^^2query2 routing LLM2query2^^^^ tasks across 9 single-turn coding dimensions. The in-distribution test set contains 2,9^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^9 held-out tasks, with approximately 3^^^^2query2^^^^6 to 347 tasks per dimension. The out-of-distribution test set contains ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^76 long-horizon, multi-file tasks requiring planning and iterative debugging (&&&^^^^2query2^^^^&&&).

Split	Size	Scope
Probing set (development)	7,^^^^2query2 routing LLM2query2^^^^	9 single-turn coding dimensions
In-distribution test	2,9^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^9	Held-out tasks, ≈3^^^^2query2^^^^6–347 per dimension
OOD agentic programming test	^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^76	Long-horizon, multi-file tasks

The benchmark covers 9 single-turn dimensions and ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^ out-of-distribution dimension:

^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^. Code Generation: HumanEval+, MBPP+, BigCodeBench—execution-based pass@^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^

1. Algorithm Design: LiveCodeBench, BigCodeBench—execution
2. Bug Fixing: DebugBench, SWE-bench Lite—execution
3. Code Completion: CRUXEval, HumanEval+ variants—execution
4. Code Refactoring: Bugs2Fix, CanItEdit—proxy metrics + LLM-as-Judge
5. Data Science: DS-^^^^2(Zhou et al., 22 Jun 2026) OR \2query2query2query2^^^^, BigCodeBench—execution
6. Multi-Language: HumanEval-X, MultiPL-E—execution
7. Code Understanding: CodeXGLUE summarization—proxy + LLM-as-Judge
8. Test Generation: LiveCodeBench, HumanEval+ variants—proxy + LLM-as-Judge ^^^^2(Zhou et al., 22 Jun 2026) OR \2query2^^^^. Agentic Programming: SWE-bench Verified, LongCLI-Bench, FeatureBench, SWE-CI—Docker harness execution (&&&^^^^2query2^^^^&&&)

Task verification differs by dimension. For execution-based dimensions, each model output is run in a sandboxed Docker environment and judged as pass or fail, with pass@^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^ recorded. For proxy dimensions, outputs are scored by an LLM-as-Judge and/or AST-based checks. The OOD agentic tasks use the SWE-bench Docker harness with mini-swe-agent capped at 4^^^^2query2^^^^ steps; success requires passing the repository’s test suite (&&&^^^^2query2^^^^&&&).

This mixture of execution-verified and proxy-scored tasks is methodologically significant because it broadens coverage beyond narrowly defined code synthesis benchmarks while preserving verifiability where possible. A plausible implication is that the benchmark is intended to approximate deployment-relevant routing conditions rather than only standardized single-function completion.

3. Formal evaluation protocol

CodeRouterBench models routing as a contextual multi-armed bandit over a stream of PRESERVED_PLACEHOLDER_^^^^2query2^^^^ tasks PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^, with a model pool $\mathcal{M} = \{m_1,\dots,m_M\}$, where $M=8$ in the reported experiments (&&&^^^^2query2^^^^&&&). At step $i$, the context $c_i$ includes the prompt $p_i$, optional metadata $d_i$ such as dimension, difficulty, or language, and past history $\mathcal{H}_{<i}$. The action $a_i \in [M]$ is the selected model index. The feedback is PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2query2^^^^, where PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2(Zhou et al., 22 Jun 2026) OR \2^^^^ is verifier-observed performance and PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \22^^^^ is the incurred cost in USD (&&&^^^^2query2^^^^&&&).

The per-task reward is defined as a cost-aware quantity: PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^3 where PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^4 is the ground-truth score of model PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^5 on task PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^6, PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^7 is the monetary cost in USD, and typically PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^8, PRESERVED_PLACEHOLDER_^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^9 (&&&^^^^2query2^^^^&&&).

The benchmark pre-computes the full outcome matrix

$\mathcal{M} = \{m_1,\dots,m_M\}$^^^^2query2^^^^

The per-task oracle then selects

$\mathcal{M} = \{m_1,\dots,m_M\}$^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^

with average reward

$\mathcal{M} = \{m_1,\dots,m_M\}$2

A routing policy $\mathcal{M} = \{m_1,\dots,m_M\}$3 is evaluated by cumulative regret: $\mathcal{M} = \{m_1,\dots,m_M\}$4 where

$\mathcal{M} = \{m_1,\dots,m_M\}$5

Lower cumulative regret indicates routing behavior closer to the per-task oracle (&&&^^^^2query2^^^^&&&).

The streaming protocol is sequential. Routers process tasks one by one, update any internal memory or bandit statistics online, and are evaluated against the pre-collected reward matrix $\mathcal{M} = \{m_1,\dots,m_M\}$6. This design is essential to the benchmark’s stated purpose: static routers and adaptive routers can be compared under the same verified outcomes while preserving the asymmetry between policies that do and do not learn from feedback (&&&^^^^2query2^^^^&&&).

4. Model pool and routing policies

The benchmark uses a model pool of eight frontier LLMs:

• Claude Opus 4.6 (premium)
• Claude Sonnet 4.6 (high)
• GPT-5.4 (high)
• Qwen3-Max (mid)
• Qwen3.5-Plus (low)
• GLM-5 (mid)
• Kimi-K2.5 (mid)
• MiniMax-M2.7 (low) (&&&^^^^2query2^^^^&&&)

Routing policies are grouped into several categories. The single-model baselines are Always-$\mathcal{M} = \{m_1,\dots,m_M\}$7 and Random. Static heuristic routers without feedback include DimensionBest, which uses the pre-computed best model per dimension, and kNN Retrieval, which performs nearest-neighbor lookup in a frozen development memory. Static trained policies without memory or feedback include LogReg with TF-IDF features, TF-IDF+MLP, RouteLLM-MF, RouteLLM-BERT, and a Qwen3.5-^^^^2query2^^^^.8B LoRA-finetuned router. Dynamic online bandits with reward-only verification but no reasoning include LinUCB and LinTS. The full Agent-as-a-Router instantiation, ACRouter, uses a C-A-F loop and consists of an Orchestrator, a Verifier, and a Memory module (&&&^^^^2query2^^^^&&&).

ACRouter’s Orchestrator is a fine-tuned Qwen3.5-^^^^2query2^^^^.8B combined with heuristics. Its Verifier uses sandbox, AST, and LLM-as-Judge signals. Its Memory performs online kNN over task embeddings with capacity $\mathcal{M} = \{m_1,\dots,m_M\}$8 (&&&^^^^2query2^^^^&&&). These components correspond to the proposed Context $\mathcal{M} = \{m_1,\dots,m_M\}$9 Action $M=8$^^^^2query2^^^^ Feedback $M=8$^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^ Context loop, which is intended to close the information gap by accumulating execution-grounded experience during deployment.

A common misconception is that model routing for coding can be treated as a purely static classification problem. CodeRouterBench is explicitly constructed to test the opposite hypothesis: if verification signals and memory are available online, routing may improve materially over time, especially when tasks are heterogeneous or distributionally shifted (&&&^^^^2query2^^^^&&&).

5. Empirical findings on in-distribution tasks

On the in-distribution test set of $M=8$2, the oracle attains AvgPerf 57.^^^^2query2query2^^^^% and CumReg ^^^^2query2^^^^. ACRouter attains AvgPerf 49.98%, CumReg 2^^^^2query2^^^^5.5, and Perf/\$M=8$$3 3.69. The bandit baselines LinUCB and LinTS attain AvgPerf 46.8–46.5$$M=8$4 6.^{^{^{^{2(<a href="/papers/2606.22902" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Zhou et al., 22 Jun 2026</a>)}}}} OR \2^{^{^{^–6.8.}}} Always-Opus attains 43.83%, CumReg 387.^{^{^{^{2(<a href="/papers/2606.22902" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Zhou et al., 22 Jun 2026</a>)}}}} OR \2^{^{^{^,}}} and Perf/\$ ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^.29 (&&&^^^^2query2^^^^&&&).

These results establish several benchmark-specific conclusions. First, ACRouter achieves the lowest cumulative regret on in-distribution tasks by accumulating execution feedback. Second, static classifiers approximately match DimensionBest on in-distribution performance while operating at about half the cost (&&&^^^^2query2^^^^&&&). Third, Figure 1 in the source study shows that static routers without memory accumulate regret nearly linearly at a high slope, bandits improve slightly but plateau, and ACRouter exhibits a substantially lower slope, indicating continual improvement (&&&^^^^2query2^^^^&&&).

Figure 2 further places ACRouter above the in-distribution Pareto frontier, delivering higher AvgPerf at lower cost than any single-model or static policy (&&&^^^^2query2^^^^&&&). Within the benchmark’s cost-aware reward definition, this is important because high raw task performance alone is not sufficient; the benchmark explicitly values efficient model selection rather than maximal spending.

This suggests that CodeRouterBench is not merely a leaderboard for selecting the strongest model, but a testbed for adaptive allocation under economic constraints. The distinction matters because a routing policy may dominate in cumulative regret while not maximizing Perf/\$ or raw accuracy under every alternative operational criterion.

6. Out-of-distribution generalization and interpretive significance

The out-of-distribution test set contains $M=8$5 agentic programming tasks. On this split, the oracle achieves AvgPerf 75.89%. ACRouter attains AvgPerf 62.5^^^^2query2^^^^%, CumReg ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^7.^^^^2query2^^^^, and Perf/\$M=8$6 ^^^^2query2^^^^.64. The bandit baselines attain 46.4–49.8% with CumReg 3^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^.^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^–35.9. Static classifiers collapse to 8.9–2^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^.4%, and kNN Retrieval attains ^^^^2(Zhou et al., 22 Jun 2026) OR \2^^^^4.3% (&&&^^^^2query2^^^^&&&).

The benchmark’s interpretation is explicit: only routers with an active feedback loop, combining Memory and Verifier, adapt to fundamentally new agentic tasks (&&&^^^^2query2^^^^&&&). Figure 1 on the OOD split shows that static policies accumulate huge regret immediately, whereas ACRouter’s regret remains low, indicating online self-adaptation (&&&^^^^2query2^^^^&&&).

This OOD behavior is one of CodeRouterBench’s most consequential features. In-distribution routing can often be approximated by static priors over dimensions or by retrieval from a frozen development set. The OOD results show that such approaches can overfit probing-set patterns and fail sharply when confronted with long-horizon, multi-file, repository-level tasks. A plausible implication is that benchmarking only single-turn or stationary settings would systematically overestimate the practical adequacy of static routers.

The benchmark therefore links routing quality to the ability to integrate verification signals during deployment. That emphasis also distinguishes OOD agentic programming from the benchmark’s single-turn dimensions: the former requires planning and iterative debugging, and success is evaluated at repository level via test-suite passing rather than local code quality proxies (&&&^^^^2query2^^^^&&&).

7. Reproducibility, pricing, and benchmark role

CodeRouterBench uses an MD5-based deterministic split of 6^^^^2query2^^^^% train plus ^^^^2(Zhou et al., 22 Jun 2026) OR \2query2^^^^% validation for development and 3^^^^2query2^^^^% for in-distribution test. OOD tasks are withheld entirely from training (&&&^^^^2query2^^^^&&&). Pricing is computed using per-model USD/M-token official API rates, augmented by \$^^^^2query2^^^^.^^^^2query2 for self-hosted router tokens (&&&^^^^2query2^^^^&&&). Code and benchmark data are released at the project repository specified in the source paper (&&&^^^^2query2^^^^&&&).

As an evaluation framework, CodeRouterBench combines several design choices: verified per-model outcomes, a cost-aware reward, an explicit oracle baseline, streaming sequential interaction, and both in-distribution and out-of-distribution splits. Together, these elements allow regret-based comparison of routers under a unified protocol (&&&^^^^2query2^^^^&&&).

Its broader significance lies in how it reframes coding-task routing. Rather than treating routing as a frozen prediction problem, it treats routing as an adaptive process in which execution results feed back into future decisions. The benchmark’s reported results indicate that this distinction is not merely conceptual: routers with memory and verification improve cumulative regret materially and retain performance under distribution shift, whereas static classifiers can fail abruptly in novel agentic settings (&&&^^^^2query2^^^^&&&).