---
title: 'LongReasonArena: Benchmark for Long Reasoning'
url: https://www.emergentmind.com/topics/longreasonarena
type: topic
---

# LongReasonArena: Benchmark for Long Reasoning

LongReasonArena is a benchmark for evaluating **long reasoning** in large language models rather than merely **long-context comprehension**. It is constructed from **LeetCode** algorithmic problems and their reference solutions, and it measures difficulty by the estimated length of the reasoning process required to execute the solution on synthetic inputs. By controlling the inputs, the benchmark scales the required reasoning from roughly **1K tokens** to **1M tokens**, while deliberately targeting long-reasoning primitives such as **retrieval**, **backtracking**, **memory management**, and long-horizon algorithmic consistency. The reported results show that it is challenging even for strong reasoning-oriented models: **DeepSeek-R1** achieves only **7.5%** accuracy on the hardest level, and accuracy declines approximately linearly with the logarithm of the expected number of reasoning steps [2508.19363].

## 1. Scope and conceptual framing

LongReasonArena is motivated by a distinction between **long input** and **long reasoning**. In the benchmark’s framing, long-input evaluation concerns the passive processing of long contexts, such as retrieving or summarizing information from a long passage. Long reasoning, by contrast, requires a model to carry out an extended internal process with many intermediate decisions, state updates, and possible corrections. The benchmark was introduced because existing long-context benchmarks such as **LongBench**, **Needle-in-a-Haystack**, **RULER**, and **$\infty$Bench** mainly evaluate retrieval and understanding of long inputs, while benchmarks such as **LongGenBench** or **GSM-Infinite** emphasize long outputs or structured generation without centering the execution of a long reasoning chain. The paper also presents **LR$^2$Bench** as narrower in task variety [2508.19363].

This distinction is operational, not merely terminological. LongReasonArena is designed so that solving a problem requires the model to effectively **simulate or reason through an algorithm** rather than simply locate information in context. To keep the benchmark centered on reasoning rather than context-window capacity, samples are restricted to inputs within **32K tokens**. A common misconception is therefore that LongReasonArena is simply another “very long input” benchmark. Its design instead makes the **reasoning trace length** the primary difficulty variable.

## 2. Construction and task design

The benchmark is built from **algorithmic execution tasks** derived from **LeetCode** problems and their reference implementations. For each selected problem, the authors generate synthetic inputs and obtain the ground-truth answer by executing the corresponding solution code. The benchmark therefore uses a controlled programmatic construction in which difficulty can be adjusted by changing the input while holding the underlying algorithm fixed [2508.19363].

Problem selection is constrained. The benchmark excludes problems involving:

- **databases**
- **randomized behavior**
- **multithreading**
- **interactive/multi-turn I/O**
- **no return value**
- **custom class types like `TreeNode`**
- **floating-point output, due to precision issues**

Within the retained set, the authors define a subset of **core problems** intended to stress specific long-reasoning subskills. Problems tagged **Depth-First Search** or **Backtracking** are used to probe **backtracking**, while problems tagged **Dynamic Programming** or **Breadth-First Search** are used to probe **memory**. This suggests a benchmark philosophy in which long reasoning is decomposed into recurrent computational burdens rather than treated as a single undifferentiated capacity.

Input generation is itself automated but filtered. The authors use **Qwen2.5-Coder-32B-Instruct** to generate an **input generator function** for each problem. A generator is accepted only if it can produce **10 consecutive valid inputs** and achieve **over 90% code coverage**. Problems without such a generator are removed. The reported outcome is that **Qwen2.5-Coder-32B-Instruct** produced qualified generators for **86%** of the problems. The generator prompt further asks for length-related quantities to be sampled **uniformly on a logarithmic scale** within the allowed range, so that instances cover both small and large regimes.

The benchmark also contains explicit anti-shortcut filtering. First, the authors prompt **Qwen2.5-Coder-32B-Instruct** to write a **simple guessing program**; if that guess matches the true answer, the sample is removed. Second, they generate **5 Level 3 samples** for each problem and discard the entire problem if **QwQ** solves all five. This filtering is intended to ensure that benchmark performance reflects actual long reasoning rather than shallow heuristics or answer-space bias.

## 3. Difficulty model, scaling regime, and dataset statistics

LongReasonArena estimates required reasoning length using the number of **execution lines** in the reference solution when run on a given input. The paper treats one execution line as typically corresponding to about **10 reasoning tokens**, yielding three benchmark levels [2508.19363]:

$$
\text{Level 1}: 10^2 \leq \text{lines} < 10^4
$$

$$
\text{Level 2}: 10^4 \leq \text{lines} < 10^5
$$

$$
\text{Level 3}: 10^5 \leq \text{lines} < 10^6
$$

These ranges are interpreted as roughly:

- **Level 1**: about **1K tokens** of reasoning
- **Level 2**: about **100K tokens**
- **Level 3**: about **1M tokens**

Because the input controls how many execution lines the reference solution traverses, the required reasoning length is, in principle, **arbitrarily scalable**.

| Level | Execution-line range / reasoning scale | Problems, samples, and summary statistics |
|---|---|---|
| Level 1 | $10^2$ to $10^4$ lines; about 1K reasoning tokens | **262** problems, **514** samples; median execution lines **1,444**; 90th percentile **7,210**; median input length **64**; 90th percentile **570** |
| Level 2 | $10^4$ to $10^5$ lines; about 100K reasoning tokens | **306** problems, **632** samples; median execution lines **31,592**; 90th percentile **80,689**; median input length **524**; 90th percentile **6,551** |
| Level 3 | $10^5$ to $10^6$ lines; about 1M reasoning tokens | **288** problems, **523** samples; median execution lines **202,412**; 90th percentile **557,175**; median input length **1,983**; 90th percentile **22,877** |

These statistics are central to the benchmark’s argument. Even at **Level 3**, the **90th percentile input length** is **22,877**, well below the **32K-token** constraint. The intended implication is that failure should not be attributed solely to inability to ingest long prompts; rather, it should be attributed to inability to sustain a sufficiently long and correct reasoning process.

## 4. Targeted reasoning skills and diagnostic analyses

LongReasonArena is explicitly designed to test several long-reasoning capabilities: **retrieval in the middle of reasoning**, **backtracking**, **memory management**, **algorithmic execution**, and **long-horizon consistency** [2508.19363]. These are not abstract labels; the paper develops task-specific analyses to isolate them.

The retrieval analysis uses **Two Sum**. Array length is varied to control task difficulty, and the paper reports that accuracy falls with a linear trend against the logarithm of array length, with **$R^2 = 0.975$**. The model fails completely when array length reaches **1,000**. The error analysis separates **index errors**, where the correct values are identified but the indices are wrong, from **full errors**, where the correct value pair is not found at all. At shorter lengths, index errors dominate; at larger lengths, full errors become more common. The authors further inspect extracted arithmetic equations and report **98% correctness**, which is presented as evidence that the bottleneck is mainly **retrieval**, not arithmetic execution.

The backtracking analysis uses a modified **Word Search** task. Instead of answering whether a word exists, the model must output the **list of positions** forming the word, or an empty list otherwise. Word length controls the depth of the depth-first search. The paper reports an approximately linear accuracy decline as word length grows, with **$R^2 = 0.961$**, and notes that the model struggles when search depth reaches about **20**. The benchmark formalizes a valid path as a cell sequence

$$
S = \{(x_i, y_i)\}
$$

of length greater than two, where each adjacent pair is spatially adjacent and

$$
board[x_i, y_i] = word[i].
$$

A path is a **distinct path** if it is not a subpath of another valid path. The reported average is only about **4.9 distinct paths**, which the paper interprets as evidence of weak exploration and ineffective backtracking.

The broader analysis of reasoning length is equally important. For **correct answers**, reasoning length tends to increase with task complexity. For **incorrect answers**, however, reasoning length is often **even longer** than for correct ones. This suggests that additional generated reasoning does not necessarily correspond to more effective reasoning. A plausible implication is that long reasoning failure often involves drift, repetition, or misdirected search rather than simple premature termination.

## 5. Evaluation protocol and empirical results

The benchmark evaluates **12 models**, including **9 open-source** and **3 proprietary** systems. Open-source models are evaluated with **vLLM** on **8 A100 GPUs**, while proprietary models and **DeepSeek-R1** are evaluated through APIs. For open-source reasoning models, the evaluation uses **max new tokens 32K**; for non-reasoning models, **max new tokens 8K**. The default decoding settings are **temperature 0.6**, **top-p 0.95**, and **top-k 40**. For API-based evaluation, **DeepSeek-R1** and **Claude 3.7 Sonnet** use **32K reasoning tokens** and **36K output tokens**, while **GPT-4o** and **o1** use default API settings. The paper also reports that **QwQ** is reasonably stable across random seeds, with **mean accuracy 50.9** and **standard deviation 0.93** on **Level 1** [2508.19363].

| Model | Type | Level 1 / Level 2 / Level 3 |
|---|---|---|
| o1 | proprietary reasoning | **59.3 / 29.6 / 16.4** |
| QwQ | open-source reasoning | **49.4 / 20.4 / 10.7** |
| Claude 3.7 Sonnet | proprietary reasoning | **44.2 / 15.5 / 7.8** |
| DeepSeek-R1 | reasoning | **40.1 / 15.7 / 7.5** |
| DeepSeek-R1-Distill-Qwen-32B | distilled reasoning | **38.5 / 13.9 / 7.5** |
| DeepSeek-R1-Distill-Qwen-14B | distilled reasoning | **32.7 / 9.8 / 3.3** |
| QwQ-preview | reasoning | **29.0 / 8.9 / 3.6** |
| GPT-4o | general-purpose | **23.0 / 5.7 / 2.1** |
| Qwen2.5-72B | general-purpose | **20.6 / 5.2 / 2.1** |
| DeepSeek-R1-Distill-Qwen-7B | distilled reasoning | **16.3 / 3.3 / 1.9** |
| Llama 3.1 70B | general-purpose | **12.8 / 3.3 / 1.2** |
| DeepSeek-R1-Distill-Qwen-1.5B | distilled reasoning | **1.0 / 0.3 / 0.0** |

The empirical pattern is sharp. Reasoning-oriented models outperform non-reasoning baselines, but all model families degrade substantially as reasoning length increases. The best overall model reported is **o1**, yet it reaches only **16.4%** on **Level 3**. The paper highlights **DeepSeek-R1** in particular because it obtains only **7.5%** accuracy on the hardest level despite being a strong reasoning model.

A central quantitative result is the near-linear relationship between performance and the logarithm of expected reasoning steps. The detailed regressions reported for four reasoning models are:

- **o1**: **$R^2 = 0.981$**, **$p < 10^{-4}$**
- **QwQ**: **$R^2 = 0.946$**, **$p < 10^{-4}$**
- **Claude 3.7 Sonnet**: **$R^2 = 0.914$**, **$p < 10^{-4}$**
- **DeepSeek-R1**: **$R^2 = 0.928$**, **$p < 10^{-4}$**

This regression result underwrites the benchmark’s main claim that its execution-line proxy is a meaningful control variable for long reasoning difficulty.

## 6. Benchmark position, interpretation, and limitations

LongReasonArena occupies a specific niche within the broader evaluation landscape. It is neither a document-centric long-context benchmark nor an interactive agent benchmark. Relative to methods and evaluations focused on long-context extrapolation—such as **Randomized YaRN**, which improves out-of-distribution long-context reasoning on **BABILong** and **MRCR** by combining **YaRN-based positional extrapolation**, **randomized positional encoding**, and a **length curriculum** [2606.23687]—LongReasonArena does not test positional generalization over long documents. Relative to **LongR**, which improves long-context QA through a **dynamic “Think-and-Read” mechanism** and a **dense contextual utility reward** on **LongBench v2**, **RULER**, and **InfiniteBench** [2602.05758], it does not primarily evaluate evidence consultation in long documents. Relative to **OdysseyArena**, which re-centers evaluation on **long-horizon, active, and inductive interactions** [2602.05843], LongReasonArena remains largely a benchmark of **deductive** algorithmic execution.

This positioning helps clarify two common misconceptions. First, LongReasonArena is not a generic stress test of context-window size. Its input-length restriction is intended to separate long reasoning from long reading. Second, it is not a general-purpose benchmark of all reasoning modalities. The paper explicitly notes that the benchmark mostly tests **deductive reasoning** and does not fully evaluate **inductive reasoning**, **analogical reasoning**, or **multimodal reasoning** [2508.19363].

The paper also states two principal limitations. The first is methodological: the use of **execution-line count** as a proxy for reasoning steps is only approximate, because a model may in principle solve a problem by a different strategy from the reference solution. The second is scope: by centering algorithm execution, the benchmark covers only a subset of the reasoning phenomena relevant to autonomous systems.

Even with those limitations, LongReasonArena establishes a concrete empirical claim about current reasoning models. Long-horizon reasoning degrades systematically as the expected number of reasoning steps grows, and the degradation is severe even for frontier systems. The benchmark’s controlled construction, logarithmic scaling regime, anti-shortcut filtering, and diagnostic analyses of retrieval and backtracking make it a reference point for studying whether future models can sustain accurate computation over reasoning traces that are far longer than those currently handled reliably [2508.19363].

Source: https://www.emergentmind.com/topics/longreasonarena