LongReasonArena: Benchmark for Long Reasoning

Updated 9 July 2026

LongReasonArena is a benchmark for evaluating long reasoning in LLMs, emphasizing simulated algorithm execution rather than mere long-context comprehension.
It uses algorithmic tasks derived from LeetCode problems with synthetic inputs to scale required reasoning from approximately 1K to 1M tokens.
The benchmark highlights steep performance declines in state-of-the-art models, pinpointing challenges in retrieval, backtracking, and memory management.

LongReasonArena is a benchmark for evaluating long reasoning in LLMs rather than merely long-context comprehension. It is constructed from LeetCode algorithmic problems and their reference solutions, and it measures difficulty by the estimated length of the reasoning process required to execute the solution on synthetic inputs. By controlling the inputs, the benchmark scales the required reasoning from roughly 1K tokens to 1M tokens, while deliberately targeting long-reasoning primitives such as retrieval, backtracking, memory management, and long-horizon algorithmic consistency. The reported results show that it is challenging even for strong reasoning-oriented models: DeepSeek-R1 achieves only 7.5% accuracy on the hardest level, and accuracy declines approximately linearly with the logarithm of the expected number of reasoning steps (Ding et al., 26 Aug 2025).

1. Scope and conceptual framing

LongReasonArena is motivated by a distinction between long input and long reasoning. In the benchmark’s framing, long-input evaluation concerns the passive processing of long contexts, such as retrieving or summarizing information from a long passage. Long reasoning, by contrast, requires a model to carry out an extended internal process with many intermediate decisions, state updates, and possible corrections. The benchmark was introduced because existing long-context benchmarks such as LongBench, Needle-in-a-Haystack, RULER, and $\infty$ Bench mainly evaluate retrieval and understanding of long inputs, while benchmarks such as LongGenBench or GSM-Infinite emphasize long outputs or structured generation without centering the execution of a long reasoning chain. The paper also presents LR $^2$ Bench as narrower in task variety (Ding et al., 26 Aug 2025).

This distinction is operational, not merely terminological. LongReasonArena is designed so that solving a problem requires the model to effectively simulate or reason through an algorithm rather than simply locate information in context. To keep the benchmark centered on reasoning rather than context-window capacity, samples are restricted to inputs within 32K tokens. A common misconception is therefore that LongReasonArena is simply another “very long input” benchmark. Its design instead makes the reasoning trace length the primary difficulty variable.

2. Construction and task design

The benchmark is built from algorithmic execution tasks derived from LeetCode problems and their reference implementations. For each selected problem, the authors generate synthetic inputs and obtain the ground-truth answer by executing the corresponding solution code. The benchmark therefore uses a controlled programmatic construction in which difficulty can be adjusted by changing the input while holding the underlying algorithm fixed (Ding et al., 26 Aug 2025).

Problem selection is constrained. The benchmark excludes problems involving:

databases
randomized behavior
multithreading
interactive/multi-turn I/O
no return value
custom class types like TreeNode
floating-point output, due to precision issues

Within the retained set, the authors define a subset of core problems intended to stress specific long-reasoning subskills. Problems tagged Depth-First Search or Backtracking are used to probe backtracking, while problems tagged Dynamic Programming or Breadth-First Search are used to probe memory. This suggests a benchmark philosophy in which long reasoning is decomposed into recurrent computational burdens rather than treated as a single undifferentiated capacity.

Input generation is itself automated but filtered. The authors use Qwen2.5-Coder-32B-Instruct to generate an input generator function for each problem. A generator is accepted only if it can produce 10 consecutive valid inputs and achieve over 90% code coverage. Problems without such a generator are removed. The reported outcome is that Qwen2.5-Coder-32B-Instruct produced qualified generators for 86% of the problems. The generator prompt further asks for length-related quantities to be sampled uniformly on a logarithmic scale within the allowed range, so that instances cover both small and large regimes.

The benchmark also contains explicit anti-shortcut filtering. First, the authors prompt Qwen2.5-Coder-32B-Instruct to write a simple guessing program; if that guess matches the true answer, the sample is removed. Second, they generate 5 Level 3 samples for each problem and discard the entire problem if QwQ solves all five. This filtering is intended to ensure that benchmark performance reflects actual long reasoning rather than shallow heuristics or answer-space bias.

3. Difficulty model, scaling regime, and dataset statistics

LongReasonArena estimates required reasoning length using the number of execution lines in the reference solution when run on a given input. The paper treats one execution line as typically corresponding to about 10 reasoning tokens, yielding three benchmark levels (Ding et al., 26 Aug 2025):

$\text{Level 1}: 10^2 \leq \text{lines} < 10^4$

$\text{Level 2}: 10^4 \leq \text{lines} < 10^5$

$\text{Level 3}: 10^5 \leq \text{lines} < 10^6$

These ranges are interpreted as roughly:

Level 1: about 1K tokens of reasoning
Level 2: about 100K tokens
Level 3: about 1M tokens

Because the input controls how many execution lines the reference solution traverses, the required reasoning length is, in principle, arbitrarily scalable.

Level	Execution-line range / reasoning scale	Problems, samples, and summary statistics
Level 1	$10^2$ to $10^4$ lines; about 1K reasoning tokens	262 problems, 514 samples; median execution lines 1,444; 90th percentile 7,210; median input length 64; 90th percentile 570
Level 2	$10^4$ to $10^5$ lines; about 100K reasoning tokens	306 problems, 632 samples; median execution lines 31,592; 90th percentile 80,689; median input length 524; 90th percentile 6,551
Level 3	$10^5$ to $^2$ 0 lines; about 1M reasoning tokens	288 problems, 523 samples; median execution lines 202,412; 90th percentile 557,175; median input length 1,983; 90th percentile 22,877

These statistics are central to the benchmark’s argument. Even at Level 3, the 90th percentile input length is 22,877, well below the 32K-token constraint. The intended implication is that failure should not be attributed solely to inability to ingest long prompts; rather, it should be attributed to inability to sustain a sufficiently long and correct reasoning process.

4. Targeted reasoning skills and diagnostic analyses

LongReasonArena is explicitly designed to test several long-reasoning capabilities: retrieval in the middle of reasoning, backtracking, memory management, algorithmic execution, and long-horizon consistency (Ding et al., 26 Aug 2025). These are not abstract labels; the paper develops task-specific analyses to isolate them.

The retrieval analysis uses Two Sum. Array length is varied to control task difficulty, and the paper reports that accuracy falls with a linear trend against the logarithm of array length, with $^2$ 1. The model fails completely when array length reaches 1,000. The error analysis separates index errors, where the correct values are identified but the indices are wrong, from full errors, where the correct value pair is not found at all. At shorter lengths, index errors dominate; at larger lengths, full errors become more common. The authors further inspect extracted arithmetic equations and report 98% correctness, which is presented as evidence that the bottleneck is mainly retrieval, not arithmetic execution.

The backtracking analysis uses a modified Word Search task. Instead of answering whether a word exists, the model must output the list of positions forming the word, or an empty list otherwise. Word length controls the depth of the depth-first search. The paper reports an approximately linear accuracy decline as word length grows, with $^2$ 2, and notes that the model struggles when search depth reaches about 20. The benchmark formalizes a valid path as a cell sequence

$^2$ 3

of length greater than two, where each adjacent pair is spatially adjacent and

$^2$ 4

A path is a distinct path if it is not a subpath of another valid path. The reported average is only about 4.9 distinct paths, which the paper interprets as evidence of weak exploration and ineffective backtracking.

The broader analysis of reasoning length is equally important. For correct answers, reasoning length tends to increase with task complexity. For incorrect answers, however, reasoning length is often even longer than for correct ones. This suggests that additional generated reasoning does not necessarily correspond to more effective reasoning. A plausible implication is that long reasoning failure often involves drift, repetition, or misdirected search rather than simple premature termination.

5. Evaluation protocol and empirical results

The benchmark evaluates 12 models, including 9 open-source and 3 proprietary systems. Open-source models are evaluated with vLLM on 8 A100 GPUs, while proprietary models and DeepSeek-R1 are evaluated through APIs. For open-source reasoning models, the evaluation uses max new tokens 32K; for non-reasoning models, max new tokens 8K. The default decoding settings are temperature 0.6, top-p 0.95, and top-k 40. For API-based evaluation, DeepSeek-R1 and Claude 3.7 Sonnet use 32K reasoning tokens and 36K output tokens, while GPT-4o and o1 use default API settings. The paper also reports that QwQ is reasonably stable across random seeds, with mean accuracy 50.9 and standard deviation 0.93 on Level 1 (Ding et al., 26 Aug 2025).

Model	Type	Level 1 / Level 2 / Level 3
o1	proprietary reasoning	59.3 / 29.6 / 16.4
QwQ	open-source reasoning	49.4 / 20.4 / 10.7
Claude 3.7 Sonnet	proprietary reasoning	44.2 / 15.5 / 7.8
DeepSeek-R1	reasoning	40.1 / 15.7 / 7.5
DeepSeek-R1-Distill-Qwen-32B	distilled reasoning	38.5 / 13.9 / 7.5
DeepSeek-R1-Distill-Qwen-14B	distilled reasoning	32.7 / 9.8 / 3.3
QwQ-preview	reasoning	29.0 / 8.9 / 3.6
GPT-4o	general-purpose	23.0 / 5.7 / 2.1
Qwen2.5-72B	general-purpose	20.6 / 5.2 / 2.1
DeepSeek-R1-Distill-Qwen-7B	distilled reasoning	16.3 / 3.3 / 1.9
Llama 3.1 70B	general-purpose	12.8 / 3.3 / 1.2
DeepSeek-R1-Distill-Qwen-1.5B	distilled reasoning	1.0 / 0.3 / 0.0

The empirical pattern is sharp. Reasoning-oriented models outperform non-reasoning baselines, but all model families degrade substantially as reasoning length increases. The best overall model reported is o1, yet it reaches only 16.4% on Level 3. The paper highlights DeepSeek-R1 in particular because it obtains only 7.5% accuracy on the hardest level despite being a strong reasoning model.

A central quantitative result is the near-linear relationship between performance and the logarithm of expected reasoning steps. The detailed regressions reported for four reasoning models are:

o1: $^2$ 5, $^2$ 6
QwQ: $^2$ 7, $^2$ 8
Claude 3.7 Sonnet: $^2$ 9, $\text{Level 1}: 10^2 \leq \text{lines} < 10^4$ 0
DeepSeek-R1: $\text{Level 1}: 10^2 \leq \text{lines} < 10^4$ 1, $\text{Level 1}: 10^2 \leq \text{lines} < 10^4$ 2

This regression result underwrites the benchmark’s main claim that its execution-line proxy is a meaningful control variable for long reasoning difficulty.

6. Benchmark position, interpretation, and limitations

LongReasonArena occupies a specific niche within the broader evaluation landscape. It is neither a document-centric long-context benchmark nor an interactive agent benchmark. Relative to methods and evaluations focused on long-context extrapolation—such as Randomized YaRN, which improves out-of-distribution long-context reasoning on BABILong and MRCR by combining YaRN-based positional extrapolation, randomized positional encoding, and a length curriculum (Mehta et al., 22 Jun 2026)—LongReasonArena does not test positional generalization over long documents. Relative to LongR, which improves long-context QA through a dynamic “Think-and-Read” mechanism and a dense contextual utility reward on LongBench v2, RULER, and InfiniteBench (Ping et al., 5 Feb 2026), it does not primarily evaluate evidence consultation in long documents. Relative to OdysseyArena, which re-centers evaluation on long-horizon, active, and inductive interactions (Xu et al., 5 Feb 2026), LongReasonArena remains largely a benchmark of deductive algorithmic execution.

This positioning helps clarify two common misconceptions. First, LongReasonArena is not a generic stress test of context-window size. Its input-length restriction is intended to separate long reasoning from long reading. Second, it is not a general-purpose benchmark of all reasoning modalities. The paper explicitly notes that the benchmark mostly tests deductive reasoning and does not fully evaluate inductive reasoning, analogical reasoning, or multimodal reasoning (Ding et al., 26 Aug 2025).

The paper also states two principal limitations. The first is methodological: the use of execution-line count as a proxy for reasoning steps is only approximate, because a model may in principle solve a problem by a different strategy from the reference solution. The second is scope: by centering algorithm execution, the benchmark covers only a subset of the reasoning phenomena relevant to autonomous systems.

Even with those limitations, LongReasonArena establishes a concrete empirical claim about current reasoning models. Long-horizon reasoning degrades systematically as the expected number of reasoning steps grows, and the degradation is severe even for frontier systems. The benchmark’s controlled construction, logarithmic scaling regime, anti-shortcut filtering, and diagnostic analyses of retrieval and backtracking make it a reference point for studying whether future models can sustain accurate computation over reasoning traces that are far longer than those currently handled reliably (Ding et al., 26 Aug 2025).