LongBench Evaluation Protocol

Updated 1 February 2026

LongBench is a benchmark framework that rigorously evaluates long-context reasoning in large language models across diverse document types.
It employs varied task compositions, precise scoring metrics, and a multi-stage annotation pipeline to ensure reliable and reproducible evaluations.
The protocol supports both zero-shot and chain-of-thought settings, offering practical insights into model accuracy, inference scaling, and human vs. model performance.

LongBench is a benchmark framework designed for rigorous, scalable, and reproducible evaluation of long-context understanding in LLMs. It uses varied task compositions, context lengths, and precise scoring metrics to measure a model’s ability to extract, synthesize, and reason over extended textual inputs, ranging from academic documents and codebases to structured tabular data. The benchmark’s protocol has evolved across multiple releases and variants, incorporating insights from quality auditing, expert evaluation, and computational efficiency studies.

1. Task Taxonomy and Underlying Rationale

LongBench classifies tasks into six main categories, capturing the breadth of long-context reasoning:

Single-document QA: Questions extracted from long-form texts (8 K–2 M words), including academic papers, novels, technical documentation, and detective fiction. Subtypes facilitate both extractive and inferential comprehension; event-ordering and narrative reasoning are included.
Multi-document QA: Synthesis tasks requiring inter-document inference, where answers are constructed by fusing evidence from multiple texts (news, legal reports, scientific literature). Temporal, causal, and logical relationships are explicitly tested.
Long in-context learning: Instances where the context consists of manuals or vocabulary books used as pseudo-training data, with subsequent queries demanding application of learned concepts (e.g., many-shot classification, translation of rare language pairs).
Long-dialogue history understanding: Analyses of extended multi-agent game logs and conversational transcripts, assessing a model’s ability to track entities, commitments, and long-range memory across sequences spanning thousands of turns.
Code repository understanding: Deep program analysis covering entire codebases, with questions targeting data flow, architectural structure, and cross-file configuration logic.
Long structured data understanding: Reasoning over tabular financial data and large-scale knowledge-graph queries, with anonymization mechanisms to mitigate memorization.

This division is designed to test multitasking scenarios that necessitate abstraction and multi-step integration, resisting solution via shallow retrieval.

2. Dataset Construction and Review Pipeline

LongBench v2 includes 503 multiple-choice items with contexts ranging from 8 K to 2 M words (median ≈54 K, mean ≈104 K), mostly under 128 K words. Source data is collected from nearly 100 annotators representing diverse professional backgrounds.

The annotation pipeline proceeds through the following stages:

Step	Description	Review Method
Document Upload	Length/diversity checks, elimination of <8 K words and high-overlap documents	Automated
Question Authoring	Each context receives four-answer multiple-choice questions with explicit evidence	Human annotation
Automated Screening	Three LLMs (GPT-4o-mini, GLM-4-Air, GLM-4-Flash) flag questions if all answer them correctly ("too easy")	LLM (128 K context)
Manual Review	24 expert reviewers test for question quality; any question solvable in <3 min is revised	Expert vetting
Revision Loop	Up to five iterations for compliance and difficulty	Hybrid
Final Integration	Questions may be further Google-proofed (<15 min web search); annotator audits ongoing	Manual + Automated

Annotators are compensated via a base rate (100 CNY), context- and difficulty-based bonuses (up to 50 CNY for length; extra for difficulty if models fail and experts require >10 min).

3. Model Evaluation Settings

Evaluation is conducted on 16 LLMs (10 open-source, 6 proprietary), each handling at least 128 K tokens (Claude-3.5 up to 200 K). Key models include GLM-4-9B-Chat, Llama-3.1, Qwen2.5-72B, Mistral-L, c4ai-command, GLM-4-Plus, GPT-4o-mini/-o, o1-mini/preview, and Claude-3.5-Sonnet.

Contexts exceeding the model’s window are truncated from the middle, preserving both head and tail. Prompts are strictly templated to elicit answers as "The correct answer is (X)."

Two regimes are used:

Zero-shot: temperature=0.1, max_new_tokens=128, answer only
Zero-shot + Chain-of-Thought (CoT): initial call (reasoning): temp=0.1, max_new_tokens=1024; follow-up (answer): temp=0.1, max_new_tokens=128

Accuracy is calculated as:

$A_{\rm model} = \frac{\sum_{i=1}^{N} \mathbb{I}\{\hat y_i = y_i\}}{N}$

where $\hat{y}_i$ is the model prediction, $y_i$ the ground truth. Invalid or unparseable outputs receive partial credit (0.25), compensated as:

$\tilde{A} = \frac{C + 0.25 I}{N}$

with $C$ = correct parses, $I$ = invalid responses.

Temperature is fixed at 0.1; answer extraction relies on regex parsing of parenthesized choice letters.

4. Human Baseline, Difficulty Controls, and Statistical Methods

Twenty-four experts completed all 503 questions within a 15-minute time limit, yielding:

$A_{\rm human} = \frac{270}{503} \approx 53.7\%$

Random guessing (1/4 options) yields 25%. For "I don't know" responses (8%), a random-guess baseline is used (0.25 credit).

Quality controls include:

Automated filtering to reject trivial or factoid-retrievable items (using LLMs).
Manual expert review with a strict 3-minute cutoff for excessive ease.
"Google-proofing" of items: search tests on a sample of 70 indicate 67/70 are not rapidly web-searchable.
Ongoing annotator/reviewer auditing, with revocation for low-quality performance.

Subgroup analyses are performed by difficulty and length bucket (<32 K, 32 K–128 K, >128 K). 95% binomial confidence intervals are approximated as:

$\mathrm{CI} \approx A \pm 1.96 \sqrt{\frac{A (1-A)}{N}}$

5. Empirical Results and Inference-Time Scaling

Notable results (zero-shot+CoT unless noted):

GPT-4o (direct): 50.1%; with CoT: 51.2%
o1-preview (includes reasoning): 57.7% (4% above human experts)
Qwen2.5-72B (SOTA open-source): ≈39–40%

Scaling inference-time compute (CoT reasoning) yields mean gains of 3.4% for open-source models and up to +8.5% for cost-efficient variants (o1-mini vs. GPT-4o-mini). A plausible implication is that enhanced reasoning and increased compute capacity at inference mitigate some limitations of context window scaling.

All experiments use multi-GPU servers (A100/H100), though hardware details are not exhaustively disclosed.

6. Protocol Significance and Recommendations

The LongBench v2 protocol, with its review pipeline, compensation formulae, mixed automated/manual auditing, compensated metrics, and length/difficulty stratifications, yields a reproducible evaluation paradigm for long-context comprehension and reasoning, validated by surpassing human performance (o1-preview).

Recommendations for best practice include strict templating, enforcing reasoning steps, multi-phase review, and difficulty validation to ensure robust, generalizable benchmarking. Comprehensive subgroup analyses by both difficulty and context length are vital for uncovering model limitations and understanding the relative value of scaling inference versus architectural enhancements.

For full reproducibility, macro-level results are summarized as follows:

Model	Zero-shot (%)	CoT (%)	Surpass Human?
GPT-4o	50.1	51.2	No
o1-preview	—	57.7	Yes
Qwen2.5-72B	39–40	—	No

The protocol is accessible at https://longbench2.github.io (Bai et al., 2024). The structure supports ongoing extension to new tasks and emergent LLM capabilities.

LongBench v2 advances the original LongBench design (Bai et al., 2023) by expanding context length into the 2 M word regime and requiring multi-step abstraction and synthesis. Downstream benchmarks such as MiniLongBench optimize for computational efficiency via sample pruning and logistic IRT modeling (Huang et al., 26 May 2025), while new proposals like 100-LongBench introduce length-controllable, baseline-disentangling scoring schemes (Yang et al., 25 May 2025).

Vision-language and code-focused evaluations (MMLongBench, LongBench Pro) adopt similar multi-stage annotation and scoring methodologies, with specific modifications for cross-modal context tokenization and multi-dimensional task taxonomies (Wang et al., 15 May 2025, Chen et al., 6 Jan 2026). All contemporary protocols retain the principle of rigorous, controlled, and reproducible large-context evaluation.

LongBench remains a foundational protocol, setting standards for human/model benchmarking, quality control, compensated scoring, and task diversity in long-context LLM research.