ATLAScore: Aggregating Long-Context Model Performance

• ATLAScore is a benchmark metric defined to aggregate long-context performance as a length-dependent capability profile, integrating both foundational and application dimensions.
• It computes length-aware AUC scores over a fixed context grid (8K to 1M tokens) and aggregates them using a harmonic mean to penalize imbalance across evaluation categories.
• The method employs end-to-end uncertainty propagation, allowing confidence intervals to reflect statistical variance from component-level scores to the final aggregate.

ATLAScore is the top-level aggregate metric introduced in the long-context benchmark “ATLAS: All-round Testing of Long-context Abilities across Scales” to summarize a model’s long-context ability as a length-dependent capability profile, rather than as a single task score or as performance at one arbitrarily chosen context length (Huang et al., 27 May 2026). It combines two design commitments: first, long-context competence should be measured over a fixed $8\mathrm{K}$–$1\mathrm{M}$ length grid rather than at a single context window; second, benchmark aggregation should respect a layered taxonomy separating foundational operations from application workloads and holistic assessment. The resulting score is a harmonic-mean aggregate over category-level summaries, with end-to-end uncertainty propagation from subset-level estimates through the nonlinear final score.

1. Benchmark setting and conceptual role

ATLAScore is defined within a benchmark that evaluates eight capability dimensions instantiated by nine auditable components over 6,438 instances and 26 models. The benchmark organizes those dimensions into a layered 3+5 taxonomy. The foundational layer consists of retrieval, aggregation, and multi-step reasoning. The application layer consists of question answering, in-context learning, code understanding, long-range memory, and holistic assessment. This structure is central to the score: ATLAScore is intended to measure not merely whether a model can retrieve from long context, but whether it can sustain balanced performance across foundational and downstream uses as context length grows (Huang et al., 27 May 2026).

The paper’s motivation is that long-context evaluation has two recurrent failure modes. A model can perform well at moderate lengths and then collapse as length increases, and strong retrieval performance can fail to transfer to realistic downstream workloads. ATLAScore is the benchmark’s mechanism for encoding both concerns into a single conservative aggregate. In this sense, it is not a raw benchmark average. It is a structured summary of breadth, robustness over length, and cross-category balance.

2. Length-aware scoring before final aggregation

ATLAScore is not computed directly from per-task accuracy. For the first seven dimensions, the benchmark first forms length-aware AUC summaries over a fixed geometric grid:

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$

The two reporting scopes are cumulative rather than pointwise:

$$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$

For a reporting scope $L^\star$, with ordered slices $$\mathcal{L}_{\leq L^\star}=\{\ell_0,\ell_1,\ldots,\ell_n\}$$ and normalized dimension score $s_d(\ell)\in[0,100]$, the paper defines a normalized trapezoidal AUC by

$$\frac{\sum_{i=0}^{n-1}\Delta_i\,[s_d(\ell_i) + s_d(\ell_{i+1})]/2}{\ell_n - \ell_0}, \qquad \Delta_i=\ell_{i+1}-\ell_i.$$

Because the length grid is geometric and the default slice weights are uniform, later and wider intervals naturally receive more weight in token-space. The paper states that this is intentional. Appendix C also rewrites the same quantity as a linear combination of per-slice scores with effective trapezoidal weights $\alpha_i$, which is important for later variance propagation (Huang et al., 27 May 2026).

The holistic assessment dimension is treated differently. It is evaluated at original benchmark lengths rather than over the ATLAS length grid because those tasks “cannot be meaningfully length-sliced.” This produces an asymmetry in the scoring pipeline: seven dimensions contribute through length-aware AUC, while holistic assessment contributes as a fixed original-length score.

3. Exact definition of ATLAScore

After dimension-level AUCs are computed, ATLAS forms three category aggregates. Using the paper’s notation, $\mathcal{D}_{\mathrm{base}}$ denotes the three foundational dimensions, $1\mathrm{M}$0 the four length-sliced application dimensions, and $1\mathrm{M}$1 the holistic assessment dimensions. The category aggregates are arithmetic means:

$1\mathrm{M}$2

$1\mathrm{M}$3

$1\mathrm{M}$4

The paper notes that $1\mathrm{M}$5 is fixed across reporting scopes because holistic assessment is not length-sliced.

The final ATLAScore is the harmonic mean of these three category aggregates:

$1\mathrm{M}$6

or equivalently,

$1\mathrm{M}$7

This harmonic aggregation is explicitly justified as a way to prevent a model from receiving a high overall score by excelling in only one category. The paper gives a concrete illustration: a profile $1\mathrm{M}$8 would score $1\mathrm{M}$9 under an arithmetic mean but only $$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$0 under the harmonic mean. The intention is therefore not merely compression of many numbers into one, but deliberate penalization of imbalance across foundational, application, and holistic performance (Huang et al., 27 May 2026).

A common misunderstanding is to treat ATLAScore as a generic “overall long-context accuracy.” The formal definition is narrower and more structured: it is a harmonic aggregate over three category-level summaries, and two of those summaries already encode degradation across length through AUC. The score is therefore a compound object whose semantics depend on both taxonomy and reporting scope.

4. Uncertainty propagation and statistical semantics

One of the paper’s distinctive methodological claims is end-to-end uncertainty propagation from subset scores to the final ATLAScore. All confidence intervals are reported at the 95% level with $$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$1. The appendix defines the half-width $$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$2 of an estimator $$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$3 and the corresponding variance estimate

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$4

At the subset level, the benchmark uses three variance estimators depending on component structure. For approximately independent instances it uses the CLT estimate

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$5

For clustered components such as MRCR and AA-LCR, it uses a cluster-robust estimator:

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$6

For weighted composite components such as LongCodeBench, it uses weighted-combination variance:

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$7

Because the dimension-level AUC is linear in per-slice scores under the trapezoidal rule, its variance propagates linearly. With $$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$8,

$$\mathcal{L} = \{8\mathrm{K}, 16\mathrm{K}, 32\mathrm{K}, 64\mathrm{K}, 128\mathrm{K}, \ldots, 1\mathrm{M}\}.$$9

and

$$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$0

The final ATLAScore is nonlinear, so the paper uses a first-order delta-method approximation. Let

$$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$1

Then

$$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$2

and, under the paper’s independence approximation across $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$3, $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$4, and $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$5,

$$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$6

The reported 95% confidence interval is then $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$7. The authors additionally validate this approximation with $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$8 Monte Carlo simulations per model and report that the ratio of delta-method CI to empirical Monte Carlo CI lies in $$\mathrm{ATLAScore}@8\mathrm{K}\text{-}128\mathrm{K} \qquad\text{and}\qquad \mathrm{ATLAScore}@8\mathrm{K}\text{-}1\mathrm{M}.$$9, indicating that the approximation is effectively exact in their regime (Huang et al., 27 May 2026).

This statistical pipeline matters for interpretation. In the paper’s framing, ATLAScore is not only a nonlinear benchmark aggregate; it is a benchmark aggregate with formally propagated uncertainty, so rank changes are meant to be interpreted against score intervals rather than as raw point-estimate noise.

5. Empirical behavior and what the score reveals

The paper argues that ATLAScore changes model comparisons because it is simultaneously length-aware and taxonomy-aware. Between $L^\star$0 and $L^\star$1, 20 of 26 models change position, and 7 models move by at least two ranks. Gemini-3.1-Pro-Preview leads at 128K, whereas Claude-Opus-4.6 leads at 1M. The paper defines relative decay as

$L^\star$2

and reports a mean relative decay of 24.3\% across 26 models, with Claude-Opus-4.6 at 8.5\% and GLM-4.7 (Non-reasoning) at 60.5\%. This is the main empirical rationale for using AUC over score-length curves rather than a single-length leaderboard (Huang et al., 27 May 2026).

The taxonomy layer matters independently. At 128K, the foundational aggregate $L^\star$3 and the application aggregate $L^\star$4 share only $L^\star$5 of cross-model variance, with $L^\star$6 and Spearman $L^\star$7. 15 of 26 models shift by at least four positions between foundational and application rankings, and the maximum layer-rank shift is 12 positions. The paper gives Kimi-K2.6 as an example of an application-heavy profile and GPT-5.2 as an example of a foundational-heavy profile. This means that ATLAScore’s harmonic aggregation is not collapsing redundant information; it is aggregating categories that are empirically non-redundant.

Capability-specific degradation is also uneven. The paper reports that retrieval and question answering are the most decay-prone dimensions between 128K and 1M, whereas code understanding degrades less and more uniformly. A plausible implication is that long-context competence is not a monolithic scaling property. Instead, ATLAScore is attempting to summarize a family of degradation curves whose shapes differ by capability.

6. Disambiguation, caveats, and reporting practice

ATLAScore is specific to the long-context benchmark “ATLAS: All-round Testing of Long-context Abilities across Scales.” Several other arXiv papers titled “ATLAS” explicitly state that they do not define an “ATLAScore,” including the long-horizon robotic annotation tool “ATLAS: An Annotation Tool for Long-horizon Robotic Action Segmentation” (Stanovcic et al., 29 Apr 2026), the scientific reasoning benchmark “ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning” (Liu et al., 18 Nov 2025), and “ATLAS: Agentic Test-time Learning-to-Allocate Scaling” (Qin et al., 1 Jun 2026). This suggests that ATLAScore is a benchmark-specific term rather than a generic ATLAS nomenclature.

The long-context paper itself is explicit that ATLAScore should not be treated as a universal standalone ranking. It recommends reporting the evaluated length scope, component scores, and known limitations alongside any headline ATLAScore (Huang et al., 27 May 2026). The stated limitations include an English-only benchmark, holistic assessment scored at original lengths rather than on the ATLAS grid, AMemBench-ACU relying on model-generated transcripts, LongCodeBench beginning at 32K, and the exclusion of open-ended summarization due to unreliable evaluator-independent scoring. Another important comparability rule is that when a model’s advertised context length is below a target slice, the harness uses middle truncation, so the resulting ATLAScore reflects actual behavior under that constraint rather than assumed support.

Two misconceptions follow from these design choices. First, ATLAScore is not a point score “at 128K” or “at 1M”; it is a cumulative summary over all slices from $L^\star$8 up to the stated upper bound. Second, ATLAScore is not intended to replace the underlying profile. In the paper’s own framing, long-context quality should be reported by capability and by length, with the harmonic-mean scalar functioning as a conservative summary rather than a complete description.