User-Centric Subjective Leaderboard (USL)

• User-Centric Subjective Leaderboard (USL) is an evaluation paradigm that ranks models based on user needs, subjective criteria, and diverse real-world scenarios.
• It employs customizable reward models and prompt-conditioned pairwise comparisons to capture heterogeneous user preferences and dynamic trade-offs.
• USL shifts from static, global scoring to tailored rankings that factor in accuracy, cost, fairness, and task fit for actionable model selection.

User-Centric Subjective Leaderboard (USL) denotes an evaluation paradigm in which model rankings are conditioned on user needs, subjective criteria, and heterogeneous real-world scenarios rather than collapsed into a single global score. In the LLM literature, the term is most specifically associated with the system introduced in "User-centric Subjective Leaderboard by Customizable Reward Modeling," which provides a preference-driven, dynamic ranking of LLMs across diverse real-world scenarios and is built on more than 10K subjective queries (Jia et al., 13 Aug 2025). More broadly, USL names a shift away from static, accuracy-centric, or crowd-averaged leaderboards toward ranking mechanisms that expose user-dependent trade-offs in utility, cost, task fit, and preference heterogeneity (Ethayarajh et al., 2020).

1. Conceptual basis and historical emergence

USL emerged from sustained criticism of conventional leaderboards in NLP and adjacent areas. A central critique is that leaderboard utility and practitioner utility are not the same object. "Utility is in the Eye of the User: A Critique of NLP Leaderboards" formalizes this divergence with a microeconomic lens: leaderboard utility depends on rank, whereas practitioner utility depends on multiple desiderata such as accuracy, compactness, fairness, and energy efficiency (Ethayarajh et al., 2020). In that framing, the conventional leaderboard is a poor proxy for the utility of the NLP community at large because prediction cost and other deployment-relevant properties are borne by practitioners but ignored by the ranking mechanism.

This critique was extended by proposals for real-world-centric evaluation. "Self-Improving-Leaderboard(SIL): A Call for Real-World Centric Natural Language Processing Leaderboards" identified three structural problems in prevailing leaderboard systems: the use of a single, static test set, the discrepancy between testing and real-world application, and leaderboard-centric bias toward the test set (Park et al., 2023). SIL proposed an evolving test dataset, periodic ranking, and robustness-oriented evaluation, thereby shifting attention from one-time test-set optimization to continued performance under changing conditions.

Within this lineage, USL can be understood as a stronger claim: not merely that benchmarks should better approximate reality, but that reality itself contains irreducible variation in user goals and judgments. The specific USL system of 2025 operationalizes this claim through dynamic, preference-conditioned ranking rather than a single universal leaderboard (Jia et al., 13 Aug 2025). A separate and unrelated use of the acronym appears in dialogue evaluation, where USL-H stands for "Understandability, Sensibleness, and Likability in Hierarchy"; despite the acronym overlap, that work concerns a configurable response-quality metric rather than a leaderboard infrastructure (Phy et al., 2020).

2. Empirical motivations: heterogeneous preferences and pragmatic skepticism

The empirical case for USL rests on two related findings: human preferences are diverse and often contradictory, and researchers continue to use leaderboards even while distrusting them. The USL paper analyzed more than 10,794 real subjective user query instances from LMArena and reported significant diversity and contradictions in human preferences (Jia et al., 13 Aug 2025). Its analysis found that 78.8% of samples have a winner, yet 92% of re-annotated samples judged both responses acceptable. It also reported no clear clustering separation between criteria favoring winners and losers, with Adjusted Rand Index approximately $0.001$. These results were used to argue that subjective tasks do not admit a universal preference function.

Prompt dependence supplies a second empirical motivation. "Prompt-to-Leaderboard" argues that aggregate leaderboards average across users and prompts in ways that obscure prompt-specific performance; its prompt-conditional Bradley–Terry formulation was designed precisely because the marginal leaderboard masks rare but important prompt types and user-specific variations (Frick et al., 20 Feb 2025). This observation is closely aligned with the USL premise that model quality is conditional rather than absolute.

A complementary perspective comes from researcher practice. "The Trust Paradox: How CS Researchers Engage LLM Leaderboards" interviewed eight researchers across four computer science subfields and found a near-universal paradox of pragmatic skepticism: participants expressed deep distrust of leaderboard rankings while continuing to use them as rough decision-making aids (Sadeghi et al., 27 May 2026). Peer recommendations were the primary basis for LLM selection for 7 out of 8 participants, 5 out of 8 had never checked a leaderboard, and arena-based human-voting leaderboards were preferred unanimously. The same study reported that disciplinary culture mediated leaderboard influence, with NLP researchers facing state-of-the-art comparison pressure while HCI and Systems/Privacy researchers reported none. Across these differences, seven of eight participants identified cost transparency as the most important missing feature. Taken together, these findings undermine the idea of a universally trusted, universally relevant leaderboard and instead motivate rankings that are explicitly conditional on user context.

3. Core computational mechanisms

The canonical USL implementation is based on Customizable Reward Models (CRMs). In the formulation used by the 2025 USL paper, each preference instance is represented as

$(c, q, o^{A}, o^{B}, y_c),$

where $c$ is a criterion, $q$ is the query, $o^A$ and $o^B$ are candidate responses, and $y_c$ is the winner under that criterion (Jia et al., 13 Aug 2025). CRMs are built on Qwen3 backbones ranging from 0.6B to 8B parameters and take as input a criterion, a query, and two responses. The paper reports that a pairwise classification objective outperformed a pairwise ranking loss: $$\hat{y}_c = \sigma(r_{\theta}(c, q, o^{A}, o^{B})),$$

$$\mathcal{L}_{\text{cls}} = - y_c \log \hat{y}_c - (1-y_c)\log(1-\hat{y}_c).$$

The same paper states that CRM-4B achieved 97.0% accuracy on topic generalization and 97.6% on criteria generalization, surpassing GPT-4.1 at 95.7% and 94.7%, respectively (Jia et al., 13 Aug 2025).

USL ranking is then produced by conditioning evaluation on a user’s topic preference and explicit criteria. The reported win-rate computation relative to a baseline model $B$ is

$(c, q, o^{A}, o^{B}, y_c),$0

where $(c, q, o^{A}, o^{B}, y_c),$1 is the topic-filtered set and $(c, q, o^{A}, o^{B}, y_c),$2 is the user-specified criterion set (Jia et al., 13 Aug 2025). This mechanism replaces the question “Which model wins on average?” with “Which model wins under these topics and these evaluative preferences?”

Prompt-conditioned ranking provides a parallel mechanism. "Prompt-to-Leaderboard" learns a function from natural-language prompts to a vector of Bradley–Terry coefficients: $(c, q, o^{A}, o^{B}, y_c),$3 so that each prompt induces its own leaderboard (Frick et al., 20 Feb 2025). The paper further shows how prompt-specific coefficients can be aggregated over a user’s prompt distribution, enabling personalization and optimal routing. This is not identical to CRM-based USL, but it is methodologically adjacent: both reject unconditional global ranking in favor of conditional evaluation.

4. Aggregation, pluralism, and representation

A central issue for any USL is how subjective judgments should be aggregated. Standard arena-style leaderboards often fit a single Bradley–Terry model to pairwise comparisons, which implies homogeneous preferences. "Pluralistic Leaderboards" argues that this is incompatible with heterogeneous user populations because a single latent quality score can systematically misrepresent large, cohesive minorities (Haghtalab et al., 1 Jun 2026). To address this, it adapts the social-choice notion of local stability. For a top-$(c, q, o^{A}, o^{B}, y_c),$4 set $(c, q, o^{A}, o^{B}, y_c),$5, local stability requires that no model outside $(c, q, o^{A}, o^{B}, y_c),$6 is collectively preferred to the top-$(c, q, o^{A}, o^{B}, y_c),$7 set by more than $(c, q, o^{A}, o^{B}, y_c),$8 of users. The paper’s formal statement is

$(c, q, o^{A}, o^{B}, y_c),$9

It then designs an alternative leaderboard mechanism that satisfies local stability while eliciting only $c$0 pairwise comparisons per user (Haghtalab et al., 1 Jun 2026).

This pluralistic view changes the purpose of aggregation. Instead of compressing disagreement into one scalar, aggregation is used to preserve representation across user segments. That perspective is consonant with USL’s explicit handling of contradictory criteria. The USL paper reports that rankings under “long” versus “short” criteria were strongly negatively correlated, with Kendall’s $c$1 and $c$2, and interprets this as evidence that contradictory preferences should produce divergent leaderboards rather than be averaged away (Jia et al., 13 Aug 2025). This suggests that a mature USL is not merely a personalized scorecard but a ranking system whose aggregation rule is itself user-aware.

Customization can also be implemented at the interface level. "How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation" proposed task-agnostic sample weighting, alternative evaluation metrics, and a visual analytics tool that lets users reweight samples by difficulty, spurious bias, out-of-distribution hardness, or confidence (Mishra et al., 2021). In a user study with commercial product development teams from five focus areas, the prototype reportedly reduced pre-deployment development and testing effort by 41% on average. Although that work predates the explicit USL terminology, it established a practical model of leaderboard customization around deployment priorities rather than fixed benchmark averages (Mishra et al., 2021).

5. Evaluation dimensions and interface features

USL is defined not only by conditional ranking algorithms but also by the dimensions it exposes to users. The trust-paradox study provides a concrete set of demanded features: cost and environmental metrics, task-specific breakdowns instead of a single aggregate score, voter demographics, model provenance, institutional affiliations or conflicts, confidence intervals, failure modes, and onboarding or user manuals for researchers outside NLP (Sadeghi et al., 27 May 2026). Arena-based evaluation was preferred unanimously in that study, but participants also asked for safeguards against vote manipulation and stronger statistical reporting.

Related user-centric benchmarks illustrate how these dimensions can be operationalized. "A User-Centric Multi-Intent Benchmark for Evaluating LLMs" collected 1,846 real-world use cases from 712 participants across 23 countries and benchmarked 10 LLM services on user intents such as Factual Question Answering, Solve Professional Problems, Text Assistant, Ask for Advice, Seek Creativity, and Leisure (Wang et al., 2024). Its scores aligned with human preference in real-world experience and pair-wise annotations, with Pearson correlations of 0.95 and 0.94, respectively. This provides one route for grounding USL-style evaluation in authentic scenario distributions rather than synthetic benchmark tasks.

In personalized interaction settings, "LikeBench: Evaluating Subjective Likability in LLMs for Personalization" decomposes likability into seven diagnostic metrics: emotional adaptation, formality matching, knowledge adaptation, reference understanding, conversation length fit, humor fit, and callback (Rahman et al., 15 Dec 2025). It reports that strong memory performance does not guarantee high likability, noting that DeepSeek R1, with lower memory accuracy at 86% and 17 facts per profile, outperformed Qwen3 by 28% on likability score despite Qwen3’s higher memory accuracy at 93% and 43 facts per profile. The significance for USL is methodological: user-centric ranking may require multi-session, multi-dimensional subjective axes that are not reducible to factual recall.

A similar principle appears in multimodal evaluation. LOTUS, a leaderboard for detailed image captioning, supports preference-oriented evaluation by allowing users to select or weight criteria such as alignment, descriptiveness, side effects, gender bias, and skin tone bias (Hirota et al., 25 Jul 2025). Its analysis reports that no single model excels across all criteria and that optimal model selection depends on user priorities. This is structurally close to USL even though it targets LVLM captioning rather than LLM chat.

6. Reliability, controversies, and methodological constraints

USL does not eliminate the classical problems of evaluation; it relocates them. A major methodological constraint is the reliability of subjective judgments. "On the Reliability of User-Centric Evaluation of Conversational Recommender Systems" studied 1,053 annotations from 124 crowd workers on 200 ReDial dialogues using the 18-dimensional CRS-Que framework (Müller et al., 19 Feb 2026). It found that utilitarian and outcome-oriented dimensions such as accuracy, usefulness, and satisfaction achieved moderate reliability under aggregation, whereas socially grounded constructs such as humanness and rapport were substantially less reliable. It also reported a strong halo effect in third-party judgments, with many dimensions collapsing into a single global quality signal. The paper therefore argues for multi-rater aggregation, dimension reduction, and caution in treating single annotators or LLM judges as ground truth (Müller et al., 19 Feb 2026).

Manipulability remains a second concern. The trust-paradox interviews emphasized that fixed leaderboards can be learned and that arena-based systems can also be subject to vote manipulation; participants preferred human-voting leaderboards, but only with stronger transparency and statistical safeguards (Sadeghi et al., 27 May 2026). Earlier work on leaderboard customization likewise showed that rankings can change dramatically when one reweights hard, out-of-distribution, or bias-sensitive samples, and argued that top-performing models may not always be the best models for deployment (Mishra et al., 2021). These findings caution against interpreting any USL output as an oracle. A plausible implication is that a robust USL requires not only customizable ranking but also explicit uncertainty reporting, provenance disclosure, and visibility into the ranking rule itself.

A final misconception concerns the meaning of “user-centric.” USL does not imply that evaluation becomes purely anecdotal or anti-quantitative. On the contrary, the literature repeatedly moves toward more explicit formalization: criterion-conditioned reward models, prompt-conditional Bradley–Terry coefficients, local-stability guarantees, intent-aware scoring rubrics, and reliability models are all attempts to make subjective evaluation auditable rather than opaque (Jia et al., 13 Aug 2025, Frick et al., 20 Feb 2025, Haghtalab et al., 1 Jun 2026, Wang et al., 2024). What changes is the target of measurement: not a universal best model, but the conditional fit between a model, a user, a task distribution, and a set of evaluative priorities.