---
title: 'Scorio: LLM Ranking Under Test-Time Scaling'
url: https://www.emergentmind.com/topics/scorio
type: topic
---

# Scorio: LLM Ranking Under Test-Time Scaling

Searching arXiv for “Scorio” and related variants to ground the article in the current literature.
Scorio is an open-source Python library and evaluation framework for ranking reasoning large language models under test-time scaling, the regime in which each model is sampled multiple times per prompt under stochastic decoding rather than being evaluated by a single output alone. In this setting, ranking is treated as a statistical estimation problem over repeated trials, not as a deterministic leaderboard construction. Scorio formalizes “dense benchmark ranking under test-time scaling,” implements a broad range of ranking procedures in a unified interface, and empirically studies how those procedures behave as the trial budget varies from the single-trial regime to high-budget repeated sampling [2603.10960].

## 1. Concept and problem setting

Scorio is designed for benchmarks with \(L\) models, \(M\) questions, and \(N\) i.i.d. stochastic trials per model–question pair. Its primitive object is a binary response tensor

\[
\mathbf{R}\in\{0,1\}^{L\times M\times N},
\]

with \(R_{lmn}=1\) when model \(l\) solves question \(m\) on trial \(n\). The additional trial axis is the defining feature of the framework. When \(N=1\), the setting reduces to ordinary single-run benchmarking; when \(N>1\), each prompt yields a multiset of sampled outcomes, and ranking methods can exploit repeated-trial structure in different ways [2603.10960].

The framework is motivated by the observation that test-time scaling changes the semantics of evaluation. In repeated stochastic decoding, a model no longer has a single benchmark result per item, but rather a distribution of outcomes. A ranking therefore depends on what statistical object is being estimated. Accuracy-style methods target marginal solve probabilities, whereas paired-comparison methods target latent strengths inferred from decisive wins, and these targets need not agree even asymptotically. The paper explicitly states that average-accuracy ranking and Bradley–Terry ranking can disagree even with infinite data, so there is no universal ranking independent of modeling choice [2603.10960].

Scorio therefore separates two questions. The first is **low-budget stability**: whether a ranking computed with small \(N\), especially \(N=1\), agrees with a high-budget reference. The second is **convergence**: how quickly a ranking method approaches its own or a reference full-trial ranking as the available number of trials increases from \(1\) to \(N\). This formulation makes model ranking under stochastic sampling a repeated-estimation problem.

## 2. Statistical representations and reference rankings

Scorio organizes repeated-trial benchmark data through several derived representations. The **pointwise** representation computes per-question solve rates

\[
\widehat{p}_{lm} := \frac{1}{N}\sum_{n=1}^N R_{lmn},
\qquad
\widehat{p}_{l} := \frac{1}{M}\sum_{m=1}^M \widehat{p}_{lm},
\]

together with the sufficient statistic \(k_{lm}:=\sum_n R_{lmn}\). This is the natural input for accuracy-based and IRT-style procedures [2603.10960].

The **pairwise** representation reduces the tensor to head-to-head counts between models:

\[
W_{ij} := \sum_{m=1}^M\sum_{n=1}^N \mathbf{1}\{R_{imn}=1,\; R_{jmn}=0\},
\]

\[
T_{ij} := \sum_{m=1}^M\sum_{n=1}^N \mathbf{1}\{R_{imn}=R_{jmn}\}.
\]

Here \(W_{ij}\) counts decisive wins of model \(i\) over \(j\), while \(T_{ij}\) counts ties, so that \(W_{ij}+W_{ji}+T_{ij}=MN\) for \(i\neq j\). This representation underlies Bradley–Terry, Davidson, Elo, Glicko, HodgeRank, PageRank, Rank Centrality, \(\alpha\)-Rank, and related methods [2603.10960].

The **setwise/listwise** representation associates each question–trial pair \((m,n)\) with a winner set

\[
U_{mn}:=\{l\in\mathcal{L}:R_{lmn}=1\},
\]

which induces a two-level partial order in which all winners tie above all losers. This supports models such as Plackett–Luce and Davidson–Luce [2603.10960].

The principal empirical gold standard is \(\mathrm{Bayes}_{\mathcal U}@80\), the uniform-prior Bayesian ranking at \(N=80\). For binary outcomes it is order-equivalent to avg@80, but the Bayesian form is preferred because it supports priors and uncertainty estimates. Scorio also studies \(\mathrm{Bayes}_{\mathbf{R}_0}@N\), an empirical-prior variant in which one greedy-decoding output per question is incorporated as pseudo-count information. The paper emphasizes that this changes the effective evaluation target: the ranking becomes a compromise between stochastic-sampling evidence and greedy-decoding prior information [2603.10960].

A central diagnostic is the **greedy–sampling alignment** \(\tau_{\text{G-S}}\), the Kendall \(\tau_b\) between rankings induced by greedy decoding and by stochastic sampling at \(N=80\). This quantity explains when the empirical prior is beneficial and when it introduces bias.

## 3. Method families implemented in Scorio

Scorio implements 72 ranking methods spanning pointwise metrics, paired-comparison models, voting rules, IRT models, graph-based methods, and spectral or Hodge-theoretic procedures [2603.10960].

| Family | Representative methods | Estimated object |
|---|---|---|
| Pointwise / metric | avg, inverse-difficulty, Pass@\({k}\), G-Pass@\({k}\), Bayes@\({N}\) | Marginal success or posterior utility |
| Paired-comparison / rating | Bradley–Terry, Davidson, Rao–Kupper, Elo, Glicko, TrueSkill, Bayesian Bradley–Terry | Latent model strength |
| Voting and listwise | Borda, Copeland, Schulze, Ranked Pairs, Kemeny–Young, Nanson, Baldwin, Majority Judgment, Plackett–Luce | Consensus order from question-wise preferences |
| IRT / graph / spectral | Rasch, 2PL, 3PL, MML, dynamic IRT, PageRank, Rank Centrality, \(\alpha\)-Rank, SerialRank, HodgeRank | Latent ability, centrality, or global flow score |

Pointwise methods use per-question or aggregate correctness frequencies. The simplest score is mean accuracy,

\[
s_l^{\mathrm{mean}} := \frac{1}{M}\sum_{m=1}^M \widehat{p}_{lm}.
\]

Bayes@\({N}\) generalizes this with posterior means \(\mu_l\) and uncertainty \(\sigma_l\), with ranking either by \(\mu_l\) or by the conservative quantile score \(\mu_l+\Phi^{-1}(q)\sigma_l\). This family makes relatively mild assumptions and is the paper’s preferred default class [2603.10960].

Paired-comparison models assume that head-to-head behavior is governed by latent strengths. The canonical Bradley–Terry form is

\[
\Pr(i\succ j)=\frac{\pi_i}{\pi_i+\pi_j},
\]

with Davidson and Rao–Kupper extending the formulation to ties. Sequential rating systems such as Elo and Glicko instead process the stream of pairwise outcomes induced by every question–trial pair. These methods target a notion of comparative strength rather than marginal correctness [2603.10960].

Voting rules treat each question as a voter. For each question, the trial count \(k_{lm}\) ranks models by solve frequency on that item. Borda aggregates question-level ranks, Copeland aggregates pairwise victories, and Condorcet-style procedures such as Schulze, Ranked Pairs, and Kemeny–Young attempt to construct a consensus order from majority relations. Majority Judgment interprets \(k_{lm}\in\{0,\dots,N\}\) as grades rather than strict wins and losses [2603.10960].

IRT models regard models as examinees with latent ability \(\theta_l\) and questions as items with difficulty and, depending on the variant, discrimination and guessing parameters. In the Rasch model,

\[
k_{lm} \sim \mathrm{Binomial}\!\left(N,\sigma(\theta_l-b_m)\right),
\]

while 2PL and 3PL extend this with \(a_m\) and \(c_m\). These methods estimate ability, sometimes with posterior or credible-bound rankings, and therefore impose a psychometric structure on the benchmark [2603.10960].

Graph-based and spectral methods interpret pairwise outcomes as a directed comparison graph. PageRank, Rank Centrality, and \(\alpha\)-Rank use stationary distributions or evolutionary dynamics; SerialRank uses Fiedler-vector seriation; HodgeRank solves a least-squares decomposition of pairwise edge flows into a global score. This suggests that Scorio is not tied to a single evaluative ontology, but rather exposes multiple mathematically distinct ranking targets under a common interface.

## 4. Experimental study and empirical findings

The empirical study evaluates 20 reasoning model configurations on four Olympiad-style math benchmarks—AIME’24, AIME’25, HMMT’25, and BrUMO’25—with \(M=30\) questions each. For every model–question pair, the experiments collect \(N=80\) independent stochastic trials using top-\(p\) sampling with temperature \(0.6\), \(p=0.95\), and seeds 1234 through 1313; one greedy-decoding output per question is also collected as \(\mathbf{R}_0\). The computation reported is 192,000 inference runs, 7,445 GPU-hours, and 2.96B tokens [2603.10960].

Ranking quality is measured primarily with Kendall’s \(\tau_b\), which accounts for ties. The high-budget result is that most full-trial rankings agree closely with the gold standard \(\mathrm{Bayes}_{\mathcal U}@80\). Mean Kendall’s \(\tau_b\) between \(\mathrm{Bayes}_{\mathcal U}@80\) and the other 71 methods is \(0.941\) on AIME’24, \(0.934\) on AIME’25, \(0.950\) on HMMT’25, \(0.954\) on BrUMO’25, and \(0.962\) when pooled. The number of methods that recover exactly the same ordering as \(\mathrm{Bayes}_{\mathcal U}@80\) is 20, 19, 34, 26, and 22, respectively [2603.10960].

The low-budget regime is more discriminative. At \(N=1\), the best methods reach approximately \(\tau_b\approx 0.86\) against the full-budget reference. For gold-standard agreement, \(\mathrm{Bayes}_{\mathbf{R}_0}@1\) is best on AIME’24, AIME’25, and BrUMO’25, while HMMT’25 has a 21-method tie class represented by \(\mathrm{Bayes}_{\mathcal U}@1\). On the pooled benchmark, a 21-method tie class reaches \(0.865\pm 0.049\), whereas \(\mathrm{Bayes}_{\mathbf{R}_0}@1\) falls to \(0.786\pm 0.031\) [2603.10960].

The study distinguishes **gold-standard agreement** from **self-consistency**. Rasch MML with lower credible bound scoring is best for self-consistency on AIME’24, AIME’25, and HMMT’25, while BrUMO’25 again favors \(\mathrm{Bayes}_{\mathbf{R}_0}@1\). On the pooled benchmark, Nanson’s rule with average-tie handling is most self-consistent at \(0.892\pm 0.050\), followed by Rasch MML LCB at \(0.883\pm 0.037\). The paper explicitly warns that self-consistency and closeness to the chosen gold standard are different objectives [2603.10960].

Using greedy decoding as an empirical prior reduces variance at \(N=1\) by 42% on AIME’24, 17% on AIME’25, 16% on HMMT’25, and 52% on BrUMO’25. The mean effect, however, depends on greedy–sampling alignment: \(\Delta\tau=+0.020\) on AIME’24, \(+0.008\) on AIME’25, \(-0.022\) on HMMT’25, and \(+0.049\) on BrUMO’25. The paper’s interpretation is that greedy decoding may under-explore hard problems, whereas stochastic sampling can discover alternative successful reasoning paths, so \(\mathrm{Bayes}_{\mathbf{R}_0}@N\) is a shrinkage estimator with a bias–variance trade-off rather than a uniformly superior rule [2603.10960].

## 5. Library design and usage

Scorio is released as an open-source Python library whose purpose is to map a response tensor \(\mathbf{R}\) to rankings and scores across multiple ranking families. Its API is intentionally uniform: the tensor is provided as a NumPy array of shape \((L,M,N)\), each ranking function takes \(R\) as its first argument, and rankings are returned as arrays of shape \((L,)\). Rankings are 1-indexed, with lower rank values denoting better models [2603.10960].

The paper presents the interface through direct examples:

```python
from scorio import rank
rankings = rank.avg(R)
rankings, scores = rank.bradley_terry(R, return_scores=True)
rankings = rank.bayes(R, R0=R0_greedy)
rankings = rank.pagerank(R, damping=0.85)
```

For categorical outcomes and conservative quantile ranking, the interface also supports:

```python
rankings = rank.bayes(R_cat, w=w, R0=R0_greedy, quantile=0.05)
```

This interface exposes evaluation-metric methods such as avg, Pass@\({k}\), G-Pass@\({k}\), and Bayes@\({N}\); paired-comparison and rating systems such as Bradley–Terry, Davidson, Rao–Kupper, Elo, Glicko, and TrueSkill; voting rules including Borda, Copeland, Schulze, Ranked Pairs, Kemeny–Young, Nanson, Baldwin, and Majority Judgment; listwise and setwise models such as Plackett–Luce and Davidson–Luce; IRT procedures including Rasch, 2PL, 3PL, MML, and dynamic IRT; and graph or spectral methods including PageRank, Rank Centrality, spectral eigenvector ranking, \(\alpha\)-Rank, Nash, SerialRank, and HodgeRank [2603.10960].

The paper’s practical recommendation is conservative. At high budgets, the preferred default is \(\mathrm{Bayes}_{\mathcal U}@N\), which is simple, interpretable, and uncertainty-aware. At low budgets, \(\mathrm{Bayes}_{\mathbf{R}_0}@N\) can be attractive if greedy and stochastic rankings are aligned, but otherwise \(\mathrm{Bayes}_{\mathcal U}@N\) remains the safer default. This suggests that Scorio is as much a benchmarking methodology as a software package.

## 6. Scope, limitations, and disambiguation

Scorio’s conceptual contribution is the claim that ranking under test-time scaling is not an automatic extension of single-run benchmarking. Different ranking rules summarize repeated-trial outcomes differently and can converge to different asymptotic targets. This has two consequences. First, leaderboard comparisons must specify the target notion of ranking rather than assuming that all reasonable procedures estimate the same object. Second, empirical priors such as greedy decoding should be treated as modeling choices that alter the target, not merely as harmless variance reduction devices [2603.10960].

The framework’s principal limitation, as stated in the paper, is therefore not lack of methods but multiplicity of legitimate targets. Accuracy-like rankings, pairwise-strength rankings, voting-based consensus orders, psychometric latent-ability scores, and graph-centrality measures may all be internally coherent while disagreeing with one another. A plausible implication is that future benchmark reports using repeated stochastic decoding may need to distinguish “evaluation target” from “ranking algorithm” more explicitly than single-run leaderboards have typically done.

The name should also be distinguished from several unrelated arXiv topics. **SCORPIO** is the name of an open-source Python package for retrieving, combining, and visualizing survey images of galaxy pairs from sky coordinates [2111.01260]. **SCORPIO** is also the name of a deep Galactic-plane radio survey, “Stellar Continuum Originating from Radio Physics In Ourgalaxy,” designed to study stellar radio emission through blind observations rather than pre-selected targeted samples [1506.04531]. In contemporary LLM evaluation, however, **Scorio** denotes the library for dense benchmark ranking under test-time scaling [2603.10960].

Source: https://www.emergentmind.com/topics/scorio