Scorio: LLM Ranking Under Test-Time Scaling

• Scorio is an open-source Python library and evaluation framework designed for ranking large language models via repeated stochastic decoding.
• It implements diverse ranking methods—including pointwise, paired-comparison, and voting techniques—that transform binary response tensors into statistically derived scores.
• Empirical analyses demonstrate how ranking stability and convergence vary with trial budgets, revealing a trade-off between bias reduction and estimation variance.

Searching arXiv for “Scorio” and related variants to ground the article in the current literature. Scorio is an open-source Python library and evaluation framework for ranking reasoning LLMs under test-time scaling, the regime in which each model is sampled multiple times per prompt under stochastic decoding rather than being evaluated by a single output alone. In this setting, ranking is treated as a statistical estimation problem over repeated trials, not as a deterministic leaderboard construction. Scorio formalizes “dense benchmark ranking under test-time scaling,” implements a broad range of ranking procedures in a unified interface, and empirically studies how those procedures behave as the trial budget varies from the single-trial regime to high-budget repeated sampling (Hariri et al., 11 Mar 2026).

1. Concept and problem setting

Scorio is designed for benchmarks with $L$ models, $M$ questions, and $N$ i.i.d. stochastic trials per model–question pair. Its primitive object is a binary response tensor

$\mathbf{R}\in\{0,1\}^{L\times M\times N},$

with $R_{lmn}=1$ when model $l$ solves question $m$ on trial $n$. The additional trial axis is the defining feature of the framework. When $N=1$, the setting reduces to ordinary single-run benchmarking; when $N>1$, each prompt yields a multiset of sampled outcomes, and ranking methods can exploit repeated-trial structure in different ways (Hariri et al., 11 Mar 2026).

The framework is motivated by the observation that test-time scaling changes the semantics of evaluation. In repeated stochastic decoding, a model no longer has a single benchmark result per item, but rather a distribution of outcomes. A ranking therefore depends on what statistical object is being estimated. Accuracy-style methods target marginal solve probabilities, whereas paired-comparison methods target latent strengths inferred from decisive wins, and these targets need not agree even asymptotically. The paper explicitly states that average-accuracy ranking and Bradley–Terry ranking can disagree even with infinite data, so there is no universal ranking independent of modeling choice (Hariri et al., 11 Mar 2026).

Scorio therefore separates two questions. The first is low-budget stability: whether a ranking computed with small $M$0, especially $M$1, agrees with a high-budget reference. The second is convergence: how quickly a ranking method approaches its own or a reference full-trial ranking as the available number of trials increases from $M$2 to $M$3. This formulation makes model ranking under stochastic sampling a repeated-estimation problem.

2. Statistical representations and reference rankings

Scorio organizes repeated-trial benchmark data through several derived representations. The pointwise representation computes per-question solve rates

$M$4

together with the sufficient statistic $M$5. This is the natural input for accuracy-based and IRT-style procedures (Hariri et al., 11 Mar 2026).

The pairwise representation reduces the tensor to head-to-head counts between models:

$M$6

$M$7

Here $M$8 counts decisive wins of model $M$9 over $N$0, while $N$1 counts ties, so that $N$2 for $N$3. This representation underlies Bradley–Terry, Davidson, Elo, Glicko, HodgeRank, PageRank, Rank Centrality, $N$4-Rank, and related methods (Hariri et al., 11 Mar 2026).

The setwise/listwise representation associates each question–trial pair $N$5 with a winner set

$N$6

which induces a two-level partial order in which all winners tie above all losers. This supports models such as Plackett–Luce and Davidson–Luce (Hariri et al., 11 Mar 2026).

The principal empirical gold standard is $N$7, the uniform-prior Bayesian ranking at $N$8. For binary outcomes it is order-equivalent to avg@80, but the Bayesian form is preferred because it supports priors and uncertainty estimates. Scorio also studies $N$9, an empirical-prior variant in which one greedy-decoding output per question is incorporated as pseudo-count information. The paper emphasizes that this changes the effective evaluation target: the ranking becomes a compromise between stochastic-sampling evidence and greedy-decoding prior information (Hariri et al., 11 Mar 2026).

A central diagnostic is the greedy–sampling alignment $\mathbf{R}\in\{0,1\}^{L\times M\times N},$0, the Kendall $\mathbf{R}\in\{0,1\}^{L\times M\times N},$1 between rankings induced by greedy decoding and by stochastic sampling at $\mathbf{R}\in\{0,1\}^{L\times M\times N},$2. This quantity explains when the empirical prior is beneficial and when it introduces bias.

3. Method families implemented in Scorio

Scorio implements 72 ranking methods spanning pointwise metrics, paired-comparison models, voting rules, IRT models, graph-based methods, and spectral or Hodge-theoretic procedures (Hariri et al., 11 Mar 2026).

Family	Representative methods	Estimated object
Pointwise / metric	avg, inverse-difficulty, Pass@$\mathbf{R}\in\{0,1\}^{L\times M\times N},$3, G-Pass@$\mathbf{R}\in\{0,1\}^{L\times M\times N},$4, Bayes@$\mathbf{R}\in\{0,1\}^{L\times M\times N},$5	Marginal success or posterior utility
Paired-comparison / rating	Bradley–Terry, Davidson, Rao–Kupper, Elo, Glicko, TrueSkill, Bayesian Bradley–Terry	Latent model strength
Voting and listwise	Borda, Copeland, Schulze, Ranked Pairs, Kemeny–Young, Nanson, Baldwin, Majority Judgment, Plackett–Luce	Consensus order from question-wise preferences
IRT / graph / spectral	Rasch, 2PL, 3PL, MML, dynamic IRT, PageRank, Rank Centrality, $\mathbf{R}\in\{0,1\}^{L\times M\times N},$6-Rank, SerialRank, HodgeRank	Latent ability, centrality, or global flow score

Pointwise methods use per-question or aggregate correctness frequencies. The simplest score is mean accuracy,

$\mathbf{R}\in\{0,1\}^{L\times M\times N},$7

Bayes@$\mathbf{R}\in\{0,1\}^{L\times M\times N},$8 generalizes this with posterior means $\mathbf{R}\in\{0,1\}^{L\times M\times N},$9 and uncertainty $R_{lmn}=1$0, with ranking either by $R_{lmn}=1$1 or by the conservative quantile score $R_{lmn}=1$2. This family makes relatively mild assumptions and is the paper’s preferred default class (Hariri et al., 11 Mar 2026).

Paired-comparison models assume that head-to-head behavior is governed by latent strengths. The canonical Bradley–Terry form is

$R_{lmn}=1$3

with Davidson and Rao–Kupper extending the formulation to ties. Sequential rating systems such as Elo and Glicko instead process the stream of pairwise outcomes induced by every question–trial pair. These methods target a notion of comparative strength rather than marginal correctness (Hariri et al., 11 Mar 2026).

Voting rules treat each question as a voter. For each question, the trial count $R_{lmn}=1$4 ranks models by solve frequency on that item. Borda aggregates question-level ranks, Copeland aggregates pairwise victories, and Condorcet-style procedures such as Schulze, Ranked Pairs, and Kemeny–Young attempt to construct a consensus order from majority relations. Majority Judgment interprets $R_{lmn}=1$5 as grades rather than strict wins and losses (Hariri et al., 11 Mar 2026).

IRT models regard models as examinees with latent ability $R_{lmn}=1$6 and questions as items with difficulty and, depending on the variant, discrimination and guessing parameters. In the Rasch model,

$R_{lmn}=1$7

while 2PL and 3PL extend this with $R_{lmn}=1$8 and $R_{lmn}=1$9. These methods estimate ability, sometimes with posterior or credible-bound rankings, and therefore impose a psychometric structure on the benchmark (Hariri et al., 11 Mar 2026).

Graph-based and spectral methods interpret pairwise outcomes as a directed comparison graph. PageRank, Rank Centrality, and $l$0-Rank use stationary distributions or evolutionary dynamics; SerialRank uses Fiedler-vector seriation; HodgeRank solves a least-squares decomposition of pairwise edge flows into a global score. This suggests that Scorio is not tied to a single evaluative ontology, but rather exposes multiple mathematically distinct ranking targets under a common interface.

4. Experimental study and empirical findings

The empirical study evaluates 20 reasoning model configurations on four Olympiad-style math benchmarks—AIME’24, AIME’25, HMMT’25, and BrUMO’25—with $l$1 questions each. For every model–question pair, the experiments collect $l$2 independent stochastic trials using top-$l$3 sampling with temperature $l$4, $l$5, and seeds 1234 through 1313; one greedy-decoding output per question is also collected as $l$6. The computation reported is 192,000 inference runs, 7,445 GPU-hours, and 2.96B tokens (Hariri et al., 11 Mar 2026).

Ranking quality is measured primarily with Kendall’s $l$7, which accounts for ties. The high-budget result is that most full-trial rankings agree closely with the gold standard $l$8. Mean Kendall’s $l$9 between $m$0 and the other 71 methods is $m$1 on AIME’24, $m$2 on AIME’25, $m$3 on HMMT’25, $m$4 on BrUMO’25, and $m$5 when pooled. The number of methods that recover exactly the same ordering as $m$6 is 20, 19, 34, 26, and 22, respectively (Hariri et al., 11 Mar 2026).

The low-budget regime is more discriminative. At $m$7, the best methods reach approximately $m$8 against the full-budget reference. For gold-standard agreement, $m$9 is best on AIME’24, AIME’25, and BrUMO’25, while HMMT’25 has a 21-method tie class represented by $n$0. On the pooled benchmark, a 21-method tie class reaches $n$1, whereas $n$2 falls to $n$3 (Hariri et al., 11 Mar 2026).

The study distinguishes gold-standard agreement from self-consistency. Rasch MML with lower credible bound scoring is best for self-consistency on AIME’24, AIME’25, and HMMT’25, while BrUMO’25 again favors $n$4. On the pooled benchmark, Nanson’s rule with average-tie handling is most self-consistent at $n$5, followed by Rasch MML LCB at $n$6. The paper explicitly warns that self-consistency and closeness to the chosen gold standard are different objectives (Hariri et al., 11 Mar 2026).

Using greedy decoding as an empirical prior reduces variance at $n$7 by 42% on AIME’24, 17% on AIME’25, 16% on HMMT’25, and 52% on BrUMO’25. The mean effect, however, depends on greedy–sampling alignment: $n$8 on AIME’24, $n$9 on AIME’25, $N=1$0 on HMMT’25, and $N=1$1 on BrUMO’25. The paper’s interpretation is that greedy decoding may under-explore hard problems, whereas stochastic sampling can discover alternative successful reasoning paths, so $N=1$2 is a shrinkage estimator with a bias–variance trade-off rather than a uniformly superior rule (Hariri et al., 11 Mar 2026).

5. Library design and usage

Scorio is released as an open-source Python library whose purpose is to map a response tensor $N=1$3 to rankings and scores across multiple ranking families. Its API is intentionally uniform: the tensor is provided as a NumPy array of shape $N=1$4, each ranking function takes $N=1$5 as its first argument, and rankings are returned as arrays of shape $N=1$6. Rankings are 1-indexed, with lower rank values denoting better models (Hariri et al., 11 Mar 2026).

The paper presents the interface through direct examples:

$N>1$4

For categorical outcomes and conservative quantile ranking, the interface also supports:

$N>1$5

This interface exposes evaluation-metric methods such as avg, Pass@$N=1$7, G-Pass@$N=1$8, and Bayes@$N=1$9; paired-comparison and rating systems such as Bradley–Terry, Davidson, Rao–Kupper, Elo, Glicko, and TrueSkill; voting rules including Borda, Copeland, Schulze, Ranked Pairs, Kemeny–Young, Nanson, Baldwin, and Majority Judgment; listwise and setwise models such as Plackett–Luce and Davidson–Luce; IRT procedures including Rasch, 2PL, 3PL, MML, and dynamic IRT; and graph or spectral methods including PageRank, Rank Centrality, spectral eigenvector ranking, $N>1$0-Rank, Nash, SerialRank, and HodgeRank (Hariri et al., 11 Mar 2026).

The paper’s practical recommendation is conservative. At high budgets, the preferred default is $N>1$1, which is simple, interpretable, and uncertainty-aware. At low budgets, $N>1$2 can be attractive if greedy and stochastic rankings are aligned, but otherwise $N>1$3 remains the safer default. This suggests that Scorio is as much a benchmarking methodology as a software package.

6. Scope, limitations, and disambiguation

Scorio’s conceptual contribution is the claim that ranking under test-time scaling is not an automatic extension of single-run benchmarking. Different ranking rules summarize repeated-trial outcomes differently and can converge to different asymptotic targets. This has two consequences. First, leaderboard comparisons must specify the target notion of ranking rather than assuming that all reasonable procedures estimate the same object. Second, empirical priors such as greedy decoding should be treated as modeling choices that alter the target, not merely as harmless variance reduction devices (Hariri et al., 11 Mar 2026).

The framework’s principal limitation, as stated in the paper, is therefore not lack of methods but multiplicity of legitimate targets. Accuracy-like rankings, pairwise-strength rankings, voting-based consensus orders, psychometric latent-ability scores, and graph-centrality measures may all be internally coherent while disagreeing with one another. A plausible implication is that future benchmark reports using repeated stochastic decoding may need to distinguish “evaluation target” from “ranking algorithm” more explicitly than single-run leaderboards have typically done.

The name should also be distinguished from several unrelated arXiv topics. SCORPIO is the name of an open-source Python package for retrieving, combining, and visualizing survey images of galaxy pairs from sky coordinates (Benavides et al., 2021). SCORPIO is also the name of a deep Galactic-plane radio survey, “Stellar Continuum Originating from Radio Physics In Ourgalaxy,” designed to study stellar radio emission through blind observations rather than pre-selected targeted samples (Umana et al., 2015). In contemporary LLM evaluation, however, Scorio denotes the library for dense benchmark ranking under test-time scaling (Hariri et al., 11 Mar 2026).