100-LongBench: Evaluating Long-Context LLMs

• 100-LongBench is a length-controllable, task-diverse evaluation framework that isolates true long-context scaling from baseline short-context proficiency in LLMs.
• It employs a multi-length, multi-task protocol across eight representative tasks with context lengths ranging from 2K to 256K tokens to determine breakdown lengths.
• The framework introduces the LongScore metric to re-rank models based on genuine long-context gains, enabling clearer comparison and actionable insights in model evaluation.

100-LongBench refers to a length-controllable, task-diverse evaluation framework specifically developed to rigorously quantify the true long-context capabilities of LLMs. The term “100-LongBench” originates from the technical report "100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?" and designates both the framework’s workflow (length sweeping, multi-length, multi-task) and its usage of 100 instances per (task, length) configuration, yielding high statistical robustness (Yang et al., 25 May 2025). The central contribution is a methodology and metric suite that isolates genuine long-context scaling from baseline linguistic and task proficiency—a distinction often conflated in prior work.

1. Motivation: Limitations in Existing Long-Context Benchmarking

Existing real-task-based benchmarks (e.g., LongBench, L-Eval, RULER, ∞-Bench) attempt to measure LLMs’ ability to process and reason over extended contexts—hundreds of thousands of tokens in QA, retrieval, summarization, and counting. Two key deficits are diagnosed:

• Baseline conflation: Previous metrics aggregate scores across context lengths, meaning models with superior short-context (e.g., ≤8K tokens) proficiency but rapid degradation at longer ranges can outrank truly scalable models. This entangles long-context gains with baseline reasoning performance, confounding cross-model comparisons and undermining interpretability.
• Fixed-length obsolescence: Conventional benchmarks fix all evaluation samples near an outdated expected window (e.g., 8K tokens), failing to probe the scaling limits of models that support up to 128K or 256K tokens and precluding measurement of the precise failure point (“breakdown length”).

These shortcomings both obscure comparative evaluation and stymie progress toward genuinely infinite-context LLMs.

2. Benchmark Structure: Task Suite and Length Sweep Protocol

100-LongBench proposes a dataset construction and evaluation protocol that addresses the above limitations:

• Task coverage comprises eight representative tasks spanning retrieval, counting, QA, and summarization, outlined as follows:

| Category | Tasks | |----------------------|---------------------------------------| | Key Retrieval | KV Retrieval, Counting Stars | | Information Retrieval| Passage Retrieval, Passage Count | | Comprehension (QA) | Single-doc QA, Multi-doc QA | | Summarization | Single-doc Sum, Multi-doc Sum |

• Length control: For each task and each target length $L \in \{$2K, 4K, 6K, 8K, 16K, 32K, 64K, 128K, 256K$\}$:
  1. One “real” article (ground truth) is randomly sampled from a task-appropriate corpus.
  2. A pool of distractor articles is drawn from a “noisy” context pool.
  3. Ground truth and distractors are concatenated, shuffled, truncated to $\leq L$ tokens.

• QA Filtering: To eliminate memorization artifacts, single- and multi-doc QA samples are filtered by running the LLM with no context, discarding those where the model exceeds a baseline threshold (signaling training data leakage).

The evaluation protocol sweeps $L$ through all supported lengths for each model $M$ and each task $T$, evaluating $N=100$ randomly sampled instances per ($T,L$) pair and recording the score $S_L(M,T)$. Plotting $S_L$ against $\}$0 reveals the breakdown length $\}$1 at which the model fails.

3. LongScore: Disentangling Baseline from Long-Context Gains

100-LongBench introduces a metric, LongScore ($\}$2), which quantifies true long-context ability relative to each model's intrinsic short-context “base”:

• Base Ability:

$\}$3

• Long-Context Gain:

$\}$4

This normalization isolates the marginal benefit (or degradation) of increasing context length above the model's short-context baseline. It allows comparisons across models with diverging pretraining, architecture, or instruction-tuning paradigms.

Whereas the standard metric (mean accuracy over all lengths) is dominated by base proficiency, LC$\}$5 reorders leaderboards to reflect authentic scaling:

• Example: On RULER’s 128K regime, models with lower base but slower decay can surpass higher-base, quickly-failing models in $\}$6 ranking.

4. Experimental Results and Model Comparison

Comprehensive experiments were conducted on four open-source LLMs across 2K–256K token lengths:

Model	Claimed Context	Base	Avg Score (8–128K)	Avg LC
Qwen2.5-14B-Instruct	32K→128K	59.1	40.7	−31.1
Qwen2.5-7B-Instruct	32K→128K	57.4	39.8	−30.6
Llama3.1-8B-Instruct	128K	44.0	36.3	−17.4
Llama3.2-1B-Instruct	128K	28.7	20.4	−28.8

• All models retain high performance up to their pretrained window (4K–8K).
• Post-hoc window extension (e.g., YaRN interpolation) enables operation up to 128K, but accuracy degrades sharply beyond 32K–64K except in models natively trained on long contexts (e.g., Llama3.1-8B-Instruct).
• When ranked by raw average, high-base models predominate, but when using $\}$7, models like Llama3.1-8B-Instruct are revealed as more robust at extreme lengths.

Breakdown length $\}$8 is model-dependent: - Qwen2.5 models: $\}$9 - Llama3.1-8B-Instruct: $\leq L$0

Key finding: No evaluated LLM achieves nearly flat scaling across the entire evaluated context regime. True “infinite-context” remains unsolved.

5. Recommendations and Design Guidelines

The analysis of 100-LongBench yields several methodological recommendations:

• Adopt length-sweep, not fixed-length: Benchmarks must test all models at multiple lengths, mapping the performance collapse curve directly.
• Employ proper normalization: Metrics like LongScore ($\leq L$1) are essential to decouple short-context proficiency from real scaling capacity.
• QA filtering: To avoid spurious credit due to pretraining memorization, difficult instances should only be retained if the model cannot answer without seeing the context.
• Report breakdown length: The context size $\leq L$2 at which significant degradation begins is an informative parameter for end-users and system integrators.

6. Limitations

Despite its advances, 100-LongBench retains certain limitations:

• The LC$\leq L$3 metric is unstable if the base accuracy is very low ($\leq L$420%), creating high variance or division by near-zero.
• Length control is contingent on corpus diversity and richness; synthetic padding or sampling bias can affect ecological validity.
• LLM-based scoring for open-ended tasks introduces potential noise or bias, especially in QA and summarization evaluation.
• Focused exclusively on text; multimodal or programmatic long-context tasks require adaptation of the methodology.

7. Impact and Influence on the Long-Context Evaluation Ecosystem

100-LongBench fundamentally reorients the field from static, fixed-length benchmarking to a regime-aware, diagnostic approach, resolving key confounds that limited prior work (LongBench, L-Eval, RULER, ∞-Bench, LooGLE) (Yang et al., 25 May 2025). Its methodology and LongScore metric have influenced subsequent benchmark design and reporting standards in long-context LLM research, serving as a template for fair, transparent, and actionable evaluation as models scale toward $\leq L$5–$\leq L$6 token windows and beyond.

Open-source code and data are available at https://github.com/uservan/100-LongBench.git, enabling adoption and extension in both academic and industrial research settings.