ATLAS: All-round Testing of Long-context Abilities across Scales

Abstract: Long-context LLMs now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task family, masking two failure modes: performance can collapse as length grows, and strong retrieval need not transfer to downstream use. We present ATLAS, a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling. ATLAS contributes three methodological principles:(i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles, with end-to-end uncertainty propagation from subset scores through the nonlinear final aggregate. We instantiate the framework across eight capability dimensions with nine auditable components and 6,438 instances, and evaluate 26 models. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 leads at 1M. Rankings reshuffle substantially between ATLASscore@8K-128K and ATLASscore@8K-1M: 7 models move by at least two ranks, and the two taxonomy layers share only 61% of cross-model variance, with individual rank gaps up to 12 positions. These results support reporting long-context quality by capability and length, not by a single headline score.

• The paper introduces ATLAS, a comprehensive framework that benchmarks long-context LLM performance across eight distinct capability dimensions.
• It employs methodological innovations like length-aware scoring with normalized AUC and harmonic mean aggregation to capture model degradation and ranking instability.
• Empirical findings reveal heterogeneous decay patterns and significant rank shifts, providing actionable insights for both model development and practical deployment.

ATLAS: A Comprehensive Framework for Long-Context LLM Evaluation

Motivation and Benchmarking Landscape

As LLMs propagate increasing context windows—ranging from thousands to millions of tokens—the critical gap between nominal context length and effective long-context capabilities has become more pronounced. Mainstream reporting conventionally focuses on a single operating length or a narrow task suite, often failing to characterize performance collapse at higher lengths or the weak transfer from synthetic retrieval probes to real-world application workloads. Existing benchmarks are limited either by task breadth at moderate length (e.g., SCROLLS, LongBench [bai-etal-2024-longbench], HELMET [yen2025helmet]) or by ultra-long regime coverage with narrow diagnostic scope ($\infty$Bench [zhang2024inftybench], LOFT [Lee2024LongContext], LongCodeBench [rando2025longcodebench]), as illustrated by the positioning of ATLAS in the benchmarking landscape.

Figure 1: ATLAS is situated in the upper-right corner, combining broad capability coverage and ultra-long context evaluation up to 1M tokens with length-aware AUC scoring.

Methodological Innovations

Layered Capability Taxonomy

ATLAS formalizes long-context evaluation as a layered, diagnostic taxonomy comprising eight capability dimensions partitioned into foundational and application layers. The foundational layer includes retrieval, aggregation, and multi-step reasoning, each fully determined by input structure. The application layer encompasses QA, in-context learning, code understanding, memory, and holistic assessment. This taxonomy ensures that performance can be attributed to distinct operations and distinguishes synthetic competence from downstream application robustness.

Length-Aware Scoring and ATLAScore

ATLAS introduces length-aware scoring, evaluating models across geometrically-spaced slices ($\mathcal{L}=\{8\mathrm{K},\,\ldots,\,1\mathrm{M}\}$) and integrating dimension-level score trajectories via a normalized AUC (trapezoidal rule), rather than relying on single-point evaluation. Aggregation across taxonomy categories uses a harmonic mean (ATLAScore), penalizing imbalanced profiles so models cannot mask major weaknesses in specific categories.

Figure 2: Length-aware scoring captures score degradation trajectories, while harmonic mean aggregation penalizes uneven category profiles.

Uncertainty Quantification

ATLAS provides the first end-to-end confidence interval propagation pipeline for long-context benchmarks, propagating subset-level variance through AUC and harmonic aggregation using a delta method. This ensures statistically robust reporting across all model evaluations.

Benchmark Design and Component Selection

Components and Coverage

ATLAS comprises nine auditable components instantiating eight taxonomy dimensions, spanning foundational probes (MRCR-8 needle, OOLong-Synth, GraphWalks), application workloads (LOFT-Text Retrieval, Helmet-ICL, LongCodeBench, AMemBench-ACU), and holistic assessment (LongBench-v2, AA-LCR). Component selection adheres to three strict criteria: extensibility across length, deterministic scoring, and strong cross-model discrimination.

Empirical validation confirms inter-dimension non-redundancy ($\bar{\rho}$=0.64, six pairs $<$0.5 at 128K; Code is notably uncorrelated with Retrieval/Reasoning), and discriminability ($\sigma\geq8.9$ at 128K). Leave-one-dimension-out analyses demonstrate that the overall ranking remains robust ($\rho\geq0.96$ at 1M).

Empirical Findings

Length-Dependent Rank Instability

ATLAScore comparisons between the 128K and 1M regimes reveal substantial rank reshuffling: 20 of 26 models shift rank, with seven moving by at least two positions. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 at 1M—a shift primarily driven by heterogenous decay profiles rather than uniform shrinkage.

Figure 3: ATLAScore comparison for representative models at 128K vs 1M context, exposing dramatic rank movements as context extends.

Figure 4: Sankey diagram summarizing model migration in ATLAScore rank from 128K to 1M; green/red indicate relative robustness/weakness at ultra-long scale.

Heterogeneous Degradation Patterns

Capability decay heatmaps show severe and dimension-specific degradation, with retrieval and QA most fragile (up to 70–80% decay in several models). Some models, such as GPT-5.2, retain high ICL performance at 1M but collapse on retrieval/QA, causing prominent overall ATLAScore drops.

Figure 5: Heatmap of capability-specific decay, highlighting significant heterogeneity and exposing catastrophic collapse in retrieval/QA dimensions for several models.

Figure 6: Full decay heatmap across all 26 models and seven length-sliced dimensions, blue=robust, red=fragile.

Figure 7: ATLAScore trajectories from 8K to 1M, revealing both gradual and cliff-type degradation.

Reasoning-model inference configurations exhibit lower overall decay than non-reasoning models; 11/15 reasoning models have decay $<$25%, supporting systematic robustness correlates but not causality.

Non-Redundant Layered Signals

Foundational and application aggregates share only 61% of cross-model variance at 128K. Up to 12 rank positions shift between layer ranks per model, supporting the necessity of layered reporting. Profiles expose practical trade-offs: e.g., models may excel at synthetic retrieval but underperform in application tasks such as code understanding.

Figure 8: Per-dimension breakdown of models exhibiting large foundational–application rank discrepancies.

Figure 9: Radar chart for all models at 128K; shape diversity highlights strength spikes and dents.

Figure 10: Radar chart at 1M; dimension contraction on retrieval/QA axes is pronounced, mirroring empirical decay.

Practical and Theoretical Implications

ATLAS validates that length-aware, multi-dimensional benchmarking is essential for robust long-context LLM assessment. Length-specific degradation, capability-level divergence, and layered taxonomy signals reveal distinct failure modes obscured by single-scope or single-dimension benchmarks. These findings are pertinent for both research and deployment: practitioners must report context scope and per-capability profiles, not merely headline scores, to avoid selection pitfalls. For model development, ATLAS provides actionable diagnostics for targeting dimensions most vulnerable to context expansion.

The correlation between reasoning-based inferencing and robustness invites further investigation into training and inference design. Future analyses may exploit ATLAS's principled framework to isolate causal factors and enable more effective long-context architecture optimization.

Conclusion

ATLAS redefines long-context LLM evaluation as a length-dependent, multi-dimensional diagnostic profiling task, replacing single-window and single-task conventions with a rigorous, auditable methodology. Empirical results on 26 models demonstrate substantial ranking instability as context extends, strong capability-specific decay, and critical non-redundancy between taxonomy layers. The benchmark sets a new reporting standard—length-aware, layered, and uncertainty-quantified—for both public model comparison and practical deployment.