RouteJudge: LLM Routing Evaluation Framework

Updated 5 July 2026

RouteJudge is an online pairwise preference evaluation framework that assesses LLM routing decisions by attributing user preferences to router-level outputs.
It integrates with the ORBIT toolbox to standardize, reproduce, and benchmark diverse routing strategies under budget and latency constraints.
The framework employs advanced metrics—including BTL aggregation, Elo ratings, and cost/latency analysis—to drive practical, preference-aware routing optimization.

Searching arXiv for the primary paper and closely related routing/judging works to ground the article. RouteJudge is an online pairwise preference evaluation framework for LLM routing systems that evaluates router-level decision quality rather than model-level response quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints; the selected model responses are then presented through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Together with its companion toolbox ORBIT (Optimal Routing and Budgeted Inference Toolbox), RouteJudge forms an open, end-to-end ecosystem for developing, reproducing, and evaluating LLM routing strategies under real user preferences and practical deployment constraints such as cost and latency (Lai et al., 17 Jun 2026).

1. Problem formulation and motivation

LLM routing is the problem of selecting, for each incoming query, one model from a heterogeneous pool—proprietary or open-source models with different capabilities, costs, latencies, context lengths, and modalities—subject to budget and service-level constraints, to maximize user-facing utility (Lai et al., 17 Jun 2026). In deployment, “quality” is not absolute: for creative writing, dialogue, translation, summarization, tutoring, and analysis, multiple answers may be valid, and users have pluralistic preferences about detail, tone, reasoning style, formatting, and speed/cost trade-offs.

This framing motivates a distinction between response evaluation and routing evaluation. Offline routing evaluations based on static benchmarks, golden answers, automated judges, or accuracy against an assumed best model lock evaluation into a fixed target and usually treat cost and latency as secondary statistics. A router can therefore appear “accurate” offline yet route to models whose outputs users do not prefer online. RouteJudge explicitly evaluates whether a router chooses models that generate responses preferred by real users, given the query, task context, and budget constraint (Lai et al., 17 Jun 2026).

A plausible implication is that RouteJudge shifts the optimization target from correctness against a fixed reference to preference-grounded decision quality under deployment constraints. This contrasts with model-level evaluation platforms and offline routing benchmarks such as RouterBench, RouterEval, and LiveBench for models, which the paper characterizes as optimizing for fixed correctness or automated scores while reporting cost as ancillary (Lai et al., 17 Jun 2026).

2. System architecture and routing-centered records

RouteJudge’s workflow is centered on the router rather than on any individual model. Users submit a text or multimodal query and a cost budget $C$ . The platform builds the budget-feasible model set $M_C$ by estimating per-query model costs. Each registered router $r_i$ receives the same inputs—query/context and $M_C$ —and recommends one model $m_i \in M_C$ . RouteJudge then aggregates these recommendations as model-level votes

$v(m) = |\{i : m_i = m\}|.$

The platform selects a duel pair $(m_A, m_B)$ from the top-voted distinct models, with ties broken to improve coverage and then randomly, queries both models in parallel, and presents their outputs side by side in randomized, anonymous order. Users label the comparison as A Win, B Win, Tie, or Both Bad. After the vote, RouteJudge reveals model identities, vote distribution, cost, and latency, and attributes preference back to the routers behind the compared models (Lai et al., 17 Jun 2026).

Preference scores for the two compared models are mapped to

$(s_A, s_B) \in \{(1,0),(0,1),(0.5,0.5),(0,0)\}.$

Each router $r_i$ then receives

$S_i = \begin{cases} s_A, & m_i = m_A \ s_B, & m_i = m_B \ \varnothing, & \text{otherwise.} \end{cases}$

The $M_C$ 0 case records non-participation when the router’s selected model did not enter the duel.

Each interaction is stored as a routing-centered evaluation record

$M_C$ 1

where query/context/multimodal inputs $M_C$ 2, budget $M_C$ 3, feasible set $M_C$ 4, all router decisions $M_C$ 5, model votes $M_C$ 6, the compared pair $M_C$ 7, user label $M_C$ 8, router-level attribution $M_C$ 9, actual costs $r_i$ 0, latencies $r_i$ 1, task label $r_i$ 2, and metadata $r_i$ 3 are logged (Lai et al., 17 Jun 2026). The task label may be Coding, Math, Translation, Creative Writing, Analysis, or Other.

This record design enables preference-aware analysis, cost-aware and latency-aware analysis, and task-conditioned diagnostics. The paper also notes that post hoc diagnostics such as entropy, agreement, and consensus strength are supported by the logged interaction structure (Lai et al., 17 Jun 2026). This suggests that RouteJudge is not only an online leaderboard mechanism but also a measurement substrate for auditing router behavior.

3. Preference modeling, statistical aggregation, and metrics

RouteJudge treats each user judgment as a pairwise preference. Beyond raw win, tie, and loss outcomes, the platform supports Bradley–Terry–Luce (BTL) aggregation for routers or models:

$r_i$ 4

with log-likelihood

$r_i$ 5

Scores can be estimated by maximum likelihood with optional regularization, and bootstrap resampling of comparisons yields confidence intervals. For significance testing, the paper specifies binomial tests on win rates and McNemar’s test on head-to-head shared duels:

$r_i$ 6

where $r_i$ 7 is the count of duels won by $r_i$ 8 but lost by $r_i$ 9, and $M_C$ 0 is the reverse count (Lai et al., 17 Jun 2026).

RouteJudge also reports Elo ratings when enough duels are available. Elo is presented as complementary to BTL by providing a widely used ordinal comparison metric with uncertainty from the number of matches. The paper cautions, however, that participation differences complicate single-score interpretations (Lai et al., 17 Jun 2026).

The platform defines participation and preference metrics as follows:

$M_C$ 1

$M_C$ 2

where $M_C$ 3 is the participating set. Restricting to binary outcomes yields

$M_C$ 4

Cost- and latency-aware metrics are equally central. Average participating cost is

$M_C$ 5

Preference-per-dollar and preference-per-second are defined as

$M_C$ 6

$M_C$ 7

Pareto analysis marks a router as dominant if it has equal-or-higher preference scores and equal-or-lower cost or latency than another router, with at least one strict improvement (Lai et al., 17 Jun 2026).

These metrics are unusual in combining preference, coverage, cost, and latency in a single protocol. A plausible implication is that RouteJudge treats routing as a deployment decision problem rather than a pure ranking problem.

4. Routing objective, router families, and offline development in ORBIT

The offline budgeted routing objective reproduced in ORBIT is

$M_C$ 8

With expected cost and latency constraints $M_C$ 9 and $m_i \in M_C$ 0, the paper gives a Lagrangian relaxation

$m_i \in M_C$ 1

A practical gating rule for a single test-time decision is

$m_i \in M_C$ 2

where $m_i \in M_C$ 3 is a learned utility score and $m_i \in M_C$ 4 trade off cost and latency (Lai et al., 17 Jun 2026). The paper explicitly states that this unifies heuristic thresholds, reward-model gating, and multi-objective selection under one view.

RouteJudge supports a broad family of routers under one protocol and one model pool. The paper enumerates heuristics such as similarity and kNN; retrieval and embedding-based scorers; learned classifiers, regressors, and reward-model routers; matrix factorization; graph-based and structured routers; cascade, uncertainty, and two-tier routers; ensemble and aggregation methods; and bandit-style or knapsack/budget formulations (Lai et al., 17 Jun 2026). Regardless of internal design, each router exposes the same prediction interface

$m_i \in M_C$ 5

ORBIT standardizes development and reproduction of such routers. It provides benchmark loading, query representation, embedding layers, router training and validation, budget-aware evaluation, and method comparison. The paper identifies several key properties:

Property	Description
Composability	Datasets, encoders, and routers are swappable via configuration
Reproducibility	A single entry point runs a method under a fixed dataset config, model pool, budget grid, split protocol, and logging
Unified router interface	Every router is a class inheriting `BaseRouter`, implements training and `predict(q, M_C)`, and logs configs, seeds, and checkpoints
Benchmark suite	Supports RouterEval, RouterBench, MixInstruct-like sources, and MMR-Bench
Offline evaluation	Sweeps budgets and outputs trade-off curves and metrics

ORBIT’s offline curve metrics include Normalized AUC over performance–cost curve, Peak Score, Quality Neutral Cost (QNC and rQNC), and Routing Collapse Index (RCI) (Lai et al., 17 Jun 2026). These are used for development before online preference tests.

A plausible interpretation is that ORBIT functions as the reproducibility layer and RouteJudge as the online external-validity layer. The paper makes this relationship explicit by describing ORBIT as both a modular toolbox and the submission and integration layer for RouteJudge (Lai et al., 17 Jun 2026).

5. Evaluation protocol, platform operation, and empirical status

The submission workflow has three stages. First, a researcher implements and validates a router in ORBIT, verifying reproducibility through the standard CLI, exemplified by $m_i \in M_C$ 7 Second, the ORBIT-compatible router is submitted via pull request, after which the RouteJudge team runs historical replay on past preference records to sanity-check alignment. Replay attribution follows the same scoring rule as online duels:

$m_i \in M_C$ 6

Third, the router is deployed to online evaluation and joins the live committee, with results such as PrefScore, WinRate, Elo, PPD/PPS, participation, and task-conditioned scores shown on routejudge.cn (Lai et al., 17 Jun 2026). Typical integration delay is reported as 2–7 days depending on training/runtime constraints.

The paper reports both offline and online empirical observations. Offline, representative routers were tested on RouterEval under a shared configuration consisting of encoder all-MiniLM-L6-v2, 2:8 train:test split, and a low-data setting; budget sweeps produced performance–cost curves, and ORBIT logged curve metrics and ablations such as encoder swaps with minimal code changes (Lai et al., 17 Jun 2026).

Online, as of 2026-06-08 04:00 AoE, the platform recorded 261 matches and 109 user votes. Preliminary rankings by Elo and win rate showed differentiation among routers, with RouterLLM-MF and NIRT-Router leading early (Lai et al., 17 Jun 2026). The platform also maintained a shared pool of 17 models spanning families and price tiers and logged per-interaction latency and cost.

The paper notes that plotting observed model win rate versus average cost yields a Pareto frontier, and that several low-cost models are competitive. This implies that routers should adapt to budget and task rather than always pick the strongest or most expensive model (Lai et al., 17 Jun 2026). Because participation differs across routers, the paper cautions against interpreting any one scalar metric in isolation.

6. Reproducibility, fairness, privacy, and limitations

RouteJudge’s protocol emphasizes consistency and anonymity. All routers see identical inputs—query/context, budget, and feasible model set—and operate under the same cost model and serving arrangements. Duel selection is based on vote aggregation with coverage-aware tie-breaks. Users judge blinded outputs with randomized ordering, while model identities and router votes are revealed only after a preference label is submitted (Lai et al., 17 Jun 2026).

The logging policy is correspondingly detailed. Each interaction stores the full routing-centered record, enabling post hoc audits, diagnostics, and replays. Historical replay supports new methods before live deployment. Continuous expansion is supported because ORBIT lowers the barrier to adding datasets, encoders, and routers, and RouteJudge can incorporate new routers as they pass reproducibility checks (Lai et al., 17 Jun 2026).

The paper explicitly states several limitations. First, the system is in an early-stage sample regime: initial statistics such as 109 votes are informative but not definitive, and more data are needed to stabilize Elo and BTL estimates and tighten confidence intervals. Second, user preference bias and heterogeneity remain intrinsic: the framework embraces pluralism, but observed preferences may reflect current user cohorts. The paper suggests that stratification, task conditioning, and future incorporation of demographic or intent metadata may help, as could personalized routing (Lai et al., 17 Jun 2026).

Third, duel selection among top-voted models improves efficiency but can bias participation. RouteJudge records non-participation explicitly, and the paper identifies future active-dueling policies as a direction for optimizing fairness and exploration. Fourth, current task labels are coarse, and expansion to richer taxonomies and multimodal tasks—images, tables, code execution, and tool use—is in scope. Fifth, cost and latency measurement vary across providers and load conditions; further standardization, throughput constraints, and multi-turn or session-level routing are proposed future directions (Lai et al., 17 Jun 2026).

On privacy and governance, the paper does not specify data licensing. It states that, in practice, queries are stored with task metadata and cost/latency, and judgments are attributed at router level with anonymized interfaces (Lai et al., 17 Jun 2026). This suggests an audit-oriented design but not a complete data-governance specification.

RouteJudge is part of a broader shift from static, monolithic evaluation toward routed, preference-aware, and context-sensitive judgment. The closest conceptual contrast in the paper is between router-level evaluation and model-level response evaluation. RouteJudge’s distinctive contribution is an online, preference-based protocol that measures whether a router’s decisions lead to user-preferred responses under real budgets and latencies (Lai et al., 17 Jun 2026).

Several adjacent works illuminate this space from other directions. “Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge” studies adaptive routing between reasoning and non-reasoning judges under budget and distribution shift, framing routing as constrained distributionally robust optimization (Zhang et al., 11 May 2026). “AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling” replaces static pooling with adaptive routing among multiple judging views, making routing internal to the evaluator itself (Miao et al., 13 Jan 2026). “Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases” proposes PlanJudge, a two-stage planning-based evaluation strategy for mitigating bias in LLM judges (Huang et al., 7 Jan 2026). These works concern judging architectures or cost-efficient judge selection rather than online router evaluation, but they share RouteJudge’s concern with adaptive selection under heterogeneous instances.

Other “judge” systems with routing semantics are domain-specific rather than general-purpose router evaluation frameworks. JURE routes instruction-based image-editing evaluation to specialized atomic experts and aggregates their feedback into a transparent verdict (Sun et al., 10 Apr 2025). DriveJudge combines VLM reasoning with deterministic rule invocation for context-aware autonomous-driving evaluation (Sun et al., 15 Jun 2026). RideJudge aligns route-map semantics with juridical logic for ride-hailing adjudication (Wu et al., 18 Mar 2026). SRR-Judge provides step-level rating and refinement for search-integrated reasoning trajectories (Zhang et al., 8 Feb 2026). RoPoLL studies robust aggregation of multiple LLM judges under biased contamination, replacing arithmetic averaging with a geometric median (Acharya et al., 29 Jun 2026).

These systems are not substitutes for RouteJudge, because they evaluate outputs, trajectories, or disputes in specific domains, whereas RouteJudge evaluates the quality of model-selection decisions made by routers under shared pools and budgets. A plausible implication is that RouteJudge occupies a more infrastructural layer: it is concerned with who should answer a query, not only with how answers or actions should be judged.

In the paper’s own practical summary, ORBIT provides the standardized substrate to build, reproduce, and compare routers, while RouteJudge serves as the online, preference-grounded test of whether a router’s decisions actually satisfy users within real budget and latency constraints (Lai et al., 17 Jun 2026).