RouterArena: Benchmarking LLM Routers

• RouterArena is an end-to-end benchmarking platform that systematically evaluates LLM routers based on accuracy, cost, and latency.
• It employs standardized datasets with balanced category coverage and Bloom's Taxonomy-based difficulty stratification for reproducible metrics.
• Empirical results highlight cost-accuracy tradeoffs and guide future adaptive, robust routing innovations in multi-model selection.

RouterArena is an open, end-to-end benchmarking platform for the comprehensive and systematic comparison of LLM routers. These routers dynamically select the most appropriate model from a candidate pool for each user query, directly addressing the heterogeneity and rapidly growing diversity in public and commercial LLM offerings. RouterArena standardizes the evaluation, reporting, and leaderboard publication of router algorithms, facilitating an empirical, reproducible, and quantitative science of multi-model selection strategies (Lu et al., 30 Sep 2025).

1. Motivation and Conceptual Foundations

The fragmentation of the LLM ecosystem, marked by a proliferation of distinct models with wide-ranging capabilities, costs, and specializations, catalyzed the emergence of router algorithms. A router’s function is, for each input $q$, to select a model $m^*(q)$ from a pool, targeting cost, accuracy, and performance tradeoffs that no single model can uniformly achieve. However, prior to RouterArena, there existed no public infrastructure for apples-to-apples router comparison—benchmarks, metrics, and leaderboards were model-centric rather than router-centric (Lu et al., 30 Sep 2025).

Prior foundational efforts such as RouterEval (Huang et al., 8 Mar 2025) highlighted the “model-level scaling up” effect: with larger or more heterogeneous pools and an effective router, the composite system's accuracy can substantially exceed that of any single model, including state-of-the-art baselines such as GPT-4. RouterArena operationalizes these concepts into an automated and extensible platform.

2. Dataset Construction and Categorization

RouterArena’s dataset consists of 8,400 multiple-choice questions, curated to span nine Dewey Decimal top-level domains (excluding only “Religion”) and 44 sub-categories. Construction uses recursive deficit redistribution for uniform category coverage (science:humanities at 2:1, with further splits at the leaf category level). The initial raw pool is derived from two existing LLM benchmarks and 21 domain datasets, systematically deduplicated via cosine similarity in sentence-transformer space (all-MiniLM-L6-v2).

Difficulty stratification aligns with Bloom’s Taxonomy: each question receives an LLM-as-judge annotation in one of six Bloom levels (Remember, Understand, Apply, Analyze, Evaluate, Create). “Create” is excluded due to excessive open-endedness. The levels are collapsed as follows:

• Easy: Remember ∪ Understand
• Medium: Apply
• Hard: Analyze ∪ Evaluate

The final dataset is balanced across category and difficulty, yielding approximately 45% easy, 35% medium, and 20% hard examples (Lu et al., 30 Sep 2025).

3. Evaluation Metrics

RouterArena implements a suite of five principal metrics, each precisely defined to compare routers along orthogonal axes (Lu et al., 30 Sep 2025):

Metric	Formula/Definition	Purpose
Query Accuracy	$A_i = \frac{1}{|D|} \sum_{q \in D} A(m_i(q),q)$	Fraction of correctly routed queries
Query Cost	$$cost_i = \sum_{q \in D} [c_m \cdot (N_{in}(q) + N_{out}(q))]$$	Operational cost of router’s selections
Routing Optimality	$\text{OSR}_i, \text{OAR}_i, \text{OCR}_i$ (see below)	Efficiency versus “cheapest correct”
Robustness	$$R_i = \frac{1}{|D|} \sum_{q \in D} 1\{ m_i(q) = m_i(q') \}$$	Consistency under query perturbations
Latency	$L_i$, mean overhead on A100 GPU	Routing overhead time

Optimality is decomposed as follows:

• Optimal Selection Ratio (OSR): Fraction of queries for which the router selects the cheapest correct model, $m^*(q) = \arg\min_{m: A(m,q)=1} c_m$.
• Optimal Accuracy Ratio (OAR): Measured accuracy over the best-in-pool possible.
• Optimal Cost Ratio (OCR): Actual router cost divided by cost of an optimal router.

Cost is log-normalized for leaderboard reporting:

$$C_i = \frac{\log_2(c_{max}) - \log_2(cost_i)}{\log_2(c_{max}) - \log_2(c_{min})} \in [0,1]$$

Composite Arena Score is a $\beta$-weighted tradeoff:

$m^*(q)$0

All metrics are computed under a reproducible, containerized pipeline with strict inference caching and time-stamped latency instrumentation.

4. Leaderboard Automation and Evaluation Pipeline

The RouterArena workflow supports plug-and-play evaluation for new routers. After API registration:

1. The benchmark queries are streamed to the router.
2. Model selections and final outputs are recorded.
3. Inference results are cached—responses are only charged cost and accuracy once per unique (model, query) pair.
4. All metrics and composite Arena scores are computed and leaderboard rankings are refreshed automatically.

Robustness evaluation involves perturbed queries (e.g., typos, paraphrases); the platform quantifies the “pick-flip” rate in router selections.

Each leaderboard entry details all principal metrics as well as averaged rank-position across axes; no router dominates every dimension.

5. Empirical Findings and Model-Routing Insights

Empirical evaluation initially covered twelve routers, including three commercial systems (e.g., NotDiamond, Azure Model Router, GPT-5) and nine open-source baselines (e.g., GraphRouter, CARROT, RouteLLM, vLLM Semantic Router). Key findings:

• Deferral Efficiency: Commercial routers led in accuracy (up to $m^*(q)$187%) but incurred high cost; open-source routers such as CARROT and GraphRouter achieved 2–5% less accuracy at a fraction of cost.
• Cost-accuracy Tradeoff: vLLM Semantic Router and CARROT demonstrated that cost could be reduced by $m^*(q)$235% for less than 2% absolute drop in accuracy.
• Optimality Gaps: Multi-model routers showed a spread in OSR and OCR; high-OSR routers did not always maximize overall accuracy.
• Robustness: Embedding-based routers showed greater than 90% pick consistency on query perturbations; explicit-feature and KNN routers were less reliable.
• Latency: Most academic routers added under 100ms overhead; routers with external API dependencies exceeded 200ms.

Notably, the accuracy/cost Pareto-frontier remained unconquered: GPT-5 ranked only seventh overall, penalized by operational expense despite top accuracy, whereas no router ranked first in all axes (Lu et al., 30 Sep 2025).

6. Limitations, Evaluation Artifacts, and Protocol Recommendations

Subsequent research revealed artifacts empirically inflating the perceived “routing headroom” and “unsolvability ceiling.” These include systematic biases from LLM-as-judge scoring, truncation under token budgets, and output-format misalignment. For example, judge scoring deviates from exact-match by up to 10–24 percentage points on knowledge tasks (MMLU, MedQA), leading to over- or underestimation of unsolvability (Garg et al., 8 May 2026). Truncation affected up to 65% of responses in certain settings, and format mismatch errors further exaggerated the gap between small and large models.

To address these, best-practice recommendations for RouterArena include:

• Dual-judge validation: Combine LLM-as-judge metrics with exact-match groundings wherever gold answers exist.
• Truncation-aware evaluation: Select generation budgets so truncation remains below 5%, and publish per-tier truncation rates.
• Balanced and cost-sensitive objectives: Incorporate class weighting or focal loss in router training to mitigate majority-class collapse.
• Domain-based pre-routing: For highly domain-skewed tasks, pre-filter and route to specialized models to avoid underutilization of high-capacity tiers.

Implementing these protocols improves fidelity and comparability of router evaluation results, as major artifacts can otherwise induce 13–17 percentage point opportunity cost by distorting training signals (Garg et al., 8 May 2026).

7. Router Innovations and Future Directions

Production-focused routing algorithms (e.g., OrcaRouter) leverage hybrid offline–online learning methods, such as LinUCB-based contextual bandit approaches over lexical and embedding features, to adaptively balance reward (accuracy), cost, and latency. OrcaRouter’s architecture (feature-dimension $m^*(q)$3) demonstrates competitive Arena scores (72.08) with 75.54% accuracy at USD 1.00 per 1K queries, ranking second on the leaderboard and approaching the diagnostic oracle’s upper bound (80.72) (Bao et al., 29 May 2026). Ablation studies confirm the necessity of feature conditioning for effective routing: removing context features collapses performance to static single-model selection.

Future enhancements outlined for RouterArena include:

• Expanded dataset coverage (open-ended tasks, finer-grained Bloom levels)
• Extended metrics (fairness, privacy overhead, multi-turn and retrieval-aware routing)
• Robustness to adversarial, multilingual, and embedding-drift scenarios
• Open submission pipeline for newly released routers and custom evaluation tasks

A plausible implication is that router methodology and platform improvements—facilitated by standardized, high-fidelity infrastructure—will accelerate innovation in cost- and quality-efficient multi-model LLM systems.

---

References:

(Huang et al., 8 Mar 2025, Lu et al., 30 Sep 2025, Garg et al., 8 May 2026, Bao et al., 29 May 2026)