RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

Abstract: We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/AIGNLAI/LAMDA-ORBIT.

• The paper presents an open platform that enables reproducible, preference-aware LLM routing evaluation through anonymized, pairwise user comparisons.
• It introduces ORBIT, a unified toolbox that standardizes diverse routing methods and supports modular experiments for cost-quality trade-off analyses.
• Empirical results reveal that offline-optimized routers may underperform online, emphasizing the need for dynamic, user-centric, multi-objective router learning.

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

Motivation and Objectives

Scaling and cost-effective deployment of LLMs is increasingly operationalized via model pools composed of heterogeneous models varying in cost, latency, and performance. The key challenge is LLM routing: dynamically selecting an optimal model per query, optimizing for both user experience and resource constraints. While the field has seen numerous routing strategies—ranging from simple rule-based policies to advanced learned selectors—evaluation is largely limited to offline, static benchmarks (often single-metric or target-anchored). This approach fails to capture the complex, pluralistic and context-specific preferences of real users, especially for open-ended tasks where notion of response quality is non-unique and user utility is multi-faceted.

"RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing" (2606.18774) introduces a comprehensive, open infrastructure that focuses on closing this evaluation gap. The platform enables robust, reproducible, and preference-aware comparison of routing decisions, leveraging direct user preference signals and offering modular extensibility for integrating new routing methods and model pools.

The RouteJudge Evaluation Framework

RouteJudge operationalizes human-centric evaluation of LLM routing via an online platform, shifting attribution from model-level correctness to router-level decision quality. The system's pipeline is summarized as follows:

• User Query and Budgeting: The user submits a (potentially multimodal) query and specifies a cost budget, pruning the candidate model set to those feasible within the constraint.
• Router Recommendation: Competing routing strategies, executed in parallel, recommend models given the query and budget.
• Duel Model Selection: From routers' votes, the two most-selected models are chosen for direct comparison.
• Anonymous Pairwise Judgment: The user is shown both responses in random order, with no exposure to underlying model identities, routing logic, or cost (Figure 1).
• Preference Attribution: User preference feedback is attributed back to the routers driving each response, enabling aggregate analysis of router quality in terms of preference win rate, Elo rating, cost-adjusted quality, and other diagnostics.
• Full-Record Storage: Each interaction's contextual data is logged, supporting comprehensive post hoc analyses by task, cost, router participation, and more.

  Figure 2: The RouteJudge evaluation framework shifts attribution from models to routers, enabling preference- and cost-aware statistics based on anonymized user pairwise comparisons.

  

Figure 1: The RouteJudge interface ensures unbiased user preference feedback by anonymizing responses and routing strategies.

This framework not only resolves the mismatch between benchmark-anchored correctness and open-ended user preference ("pluralistic preference alignment problem") but also enables fine-grained diagnostics (task- or budget-conditioned analyses), richer router-to-router comparisons, and robust participation tracking.

ORBIT: A Unified Development and Integration Layer

A core barrier to scalable evaluation is the heterogeneity of routing method implementations, model pools, and data pipelines. RouteJudge introduces ORBIT (Optimal Routing and Budgeted Inference Toolbox) as the standardized substrate for router development, benchmark loading, training, validation, and RouteJudge integration. ORBIT is architected for maximal composability:

• Modular Experimentation: Datasets, query encoders, routers, and evaluation scripts are interchangeable via configuration files.
• Unified Interfaces: All routers, whether rule-based, neural, cascade, or embedding-driven, are required to follow a unified prediction interface, supporting seamless evaluation.
• Submission Pipeline: New methods can be tested offline in ORBIT and promoted for online exposure via a pull-request protocol, first undergoing automatic replay-based evaluation on historical RouteJudge records before deployment in live user evaluation.

  Figure 3: ORBIT provides a standardized, modular pipeline for developing, validating, and integrating LLM routing strategies.

Through ORBIT, researchers benefit from reproducible results, rapid ablations, cross-method comparisons under controlled protocols, and convenient submission to the RouteJudge online system.

Router and Evaluation Diversity

The platform supports a varied candidate model pool, including both proprietary and open-source LLMs at a range of cost/quality levels. Routers span a wide methodological spectrum:

• Statistical Methods: Matrix factorization, item-response-theory (IRT) based routers, kNN retrieval, SVM and MLP classifiers.
• Learned and Neural Approaches: Embedding-based selection (e.g., using BERT or specialized encoders), regression to cost/quality metrics, and reward-model based classifiers.
• Graph-based and Cascade Methods: Exploiting relations between models or using staged fallback according to uncertainty/confidence.

Comprehensive router diagnostics are supported:

• Model selection distributions across routers.
• Pairwise router win rates and consensus/entropy measures.
• Detailed cost/preference trade-off curves and task-conditional statistics.

  Figure 4: Routers exhibit diverse model selection distributions, revealing differing strategies and adaptation to the model pool and routing objective.

  

  Figure 5: The router head-to-head win-rate matrix exposes fine-grained differences in routing strategy competitiveness.

Empirical Results and Numerical Findings

Experiments highlight that offline benchmark-optimized routers do not always transfer to superior online, preference-based performance—a crucial finding supported by explicit win-rate, Elo, and participation statistics. Specifically, some learned or regression-based selectors, effective on static metrics, underperform matrix factorization or IRT-based methods when evaluated by actual user preferences. Peak observed win rates in preliminary user studies approach 80%, with competitive router clusters differentiated by only a few percentage points on both Elo and win rate metrics.

Cost-aware preference analyses reveal a strongly non-trivial Pareto frontier: certain low-cost models achieve competitive preference win rates for a subset of tasks, invalidating any naive "strongest-only" routing policy and emphasizing the necessity of fine-grained, context-sensitive routing.

Figure 6: Performance–cost trade-off curves for representative routers demonstrate substantial differences in efficiency and cost-quality balance.

Implications, Limitations, and Future Directions

RouteJudge and ORBIT collectively represent an open, extensible ecosystem for scalable, user-centered LLM routing research:

• Methodological Rigor: By decoupling routers from implementation artifacts (data splits, feature encoders), RouteJudge enables reproducible, robust, and fair evaluation.
• Cross-Disciplinary Utility: As deployed LLM systems expand into increasingly multimodal, multi-objective, and user-facing domains, robust preference-grounded routing selection becomes central to cost and quality management.
• Research Directions: The findings suggest that preference-based and multi-objective router learning is necessary; future methods may explicitly model user heterogeneity, incorporate dynamic cost/latency signals, or optimize for deployment objectives aligned with real-world utility functions. Additionally, further theoretical work on understanding failure/degeneracy modes (routing collapse, overfit to distributional artifacts) is warranted.
• Platform Extensions: Planned expansions include richer multimodal evaluation, online active learning, adversarial and robustness analyses, and deeper task-conditioned diagnostics.

Conclusion

"RouteJudge" establishes a rigorous, scalable foundation for evaluating and improving LLM routing under realistic, user-centered constraints and objectives. By formalizing preference-aware, cost-conditioned router evaluation and providing an open, reproducible development ecosystem, the work enables deeper understanding of routing trade-offs, methodological benchmarks, and robust progress towards deployable, context-sensitive LLM orchestration.

(2606.18774)