ReasonZoo Benchmark

• ReasonZoo is a benchmark for evaluating tool-integrated reasoning (TIR), comparing pure chain-of-thought with tool-assisted approaches across nine diverse reasoning categories.
• Its methodology involves assessing model accuracy and efficiency, revealing that TIR can improve performance metrics like PAC and AUC-PCC in most domains.
• Efficiency metrics such as PAC and AUC-PCC quantify the cost of reasoning, highlighting when external computational tools yield beneficial or counterproductive results.

Searching arXiv for ReasonZoo and closely related tool-integrated reasoning work. ReasonZoo is a benchmark for evaluating tool-integrated reasoning (TIR) in LLMs across a broad range of reasoning domains. It was introduced as the central evaluation suite in "Dissecting Tool-Integrated Reasoning: An Empirical Study and Analysis" (Zhao et al., 21 Aug 2025) to address a gap in prior evaluation practice: existing reasoning benchmarks and tool-use benchmarks were described as fragmented, often concentrated on mathematics or on success/failure accuracy rather than on whether models reason efficiently or merely generate longer traces. In that framing, ReasonZoo functions as a domain-spanning testbed for asking not only whether TIR improves correctness, but also whether it improves the quality and efficiency of the reasoning process itself (Zhao et al., 21 Aug 2025).

1. Definition and scope

ReasonZoo is presented as a comprehensive benchmark encompassing nine diverse reasoning categories for the study of TIR. Its purpose is explicitly broader than evaluating code execution on mathematical tasks. The benchmark is designed so that all tasks are intended to be solvable either by pure verbal reasoning or by using external tools, which makes it suitable for comparing standard chain-of-thought reasoning with TIR under a common experimental protocol (Zhao et al., 21 Aug 2025).

In the paper’s formulation, TIR differs from standard chain-of-thought because it interleaves natural-language reasoning with executable tool use. Standard chain-of-thought remains entirely text-based, whereas TIR allows the model to emit code or structured tool calls, execute them externally, and incorporate the tool’s result back into the reasoning trace. This process is formalized as

$$\begin{aligned} & s_i \sim \pi_\theta(\cdot \vert q, c_{i-1}), \ & c_i \leftarrow c_{i - 1} \oplus s_i \oplus R(s_i), \end{aligned}$$

where $s_i$ is the current reasoning step containing executable code, $R(s_i)$ is the tool result, and $c_i$ is the updated context (Zhao et al., 21 Aug 2025).

A plausible implication is that ReasonZoo is not merely a dataset of difficult questions; it is a controlled substrate for comparing internal deliberation against reasoning augmented by exact external computation. That orientation distinguishes it from benchmarks that treat reasoning quality as equivalent to final-task accuracy.

2. Benchmark composition

ReasonZoo contains nine reasoning categories, each intended to probe a different kind of structured inference. The main benchmark table lists the following categories and sample counts (Zhao et al., 21 Aug 2025):

Category	Sample count
Number Calculation	409
Grade School Math	100
Puzzle	607
Cipher and Code / Communication Code	422
Boolean Logic	401
Daily Logic	505
Operations Research	100
Physics	100
Formal Language	441

The appendix-level description further specifies the character of these tasks. Number Calculation includes geometric transformations, coordinate conversions, rotations in 3D, and base conversion. Grade School Math draws on non-routine Chinese math contest problems from sources such as Hope Cup and Golden Cup. Puzzle covers Sudoku, KenKen, word grids, and blocked-character equations. Cipher and Coding Theory / Communication Code focuses on symbolic transformations, including substitution/transposition ciphers and error detection/correction with block and convolutional codes. Boolean Logic includes Boolean simplification, Karnaugh maps, digital circuits, and set operations. Daily Logic uses procedurally generated constraint puzzles such as Temporal Clue and ZebraLogic. Operations Research adapts integer programming, linear programming, max-flow/min-cut, and supply-chain planning. Physics contains high-school and undergraduate mechanics problems, including collisions, springs, friction, and projectile motion. Formal Language asks models to complete derivations in context-free grammars by traversing parse trees and inferring missing terminals (Zhao et al., 21 Aug 2025).

This category design is central to the benchmark’s role. The paper explicitly uses ReasonZoo to support the claim that TIR should be evaluated outside mathematics. The inclusion of Boolean Logic, Daily Logic, Operations Research, Physics, and Formal Language is therefore not incidental; it is the mechanism by which the benchmark tests whether tool use yields domain-general benefits rather than merely helping with arithmetic.

3. Experimental protocol and model families

The empirical study on ReasonZoo evaluates two broad model families. The first consists of general large reasoning models: Qwen3-8B, Qwen3-32B, Qwen3-235B-A22B, and DeepSeek-R1-0528. The second consists of models specifically optimized for tool use or code-integrated reasoning via reinforcement learning, including ToRL and CIR, tested on Qwen2.5-Math-7B in the reported table (Zhao et al., 21 Aug 2025).

Several TIR paradigms are compared: Program of Thoughts (PoT), Multi-Turn TIR (MT-TIR), and Tool-Integrated Thinking (TIT). Accuracy is used as the main task metric across the nine ReasonZoo categories, while PAC and AUC-PCC are introduced to quantify reasoning efficiency (Zhao et al., 21 Aug 2025). The PAC table uses a maximum cost $C_{\text{max} = 32{,}768$ and performance thresholds $[0.1, 0.2, \ldots, 1.0]$. For AUC-PCC, the paper focuses mainly on MT-TIR because evaluating multiple cost budgets is computationally expensive (Zhao et al., 21 Aug 2025).

The rationale for this protocol is reinforced by adjacent research on adaptive reasoning selection. "Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection" (2505.19435) also treats reasoning quality and computational efficiency as a joint optimization problem over model–strategy pairs, rather than assuming that the strongest model or longest reasoning trace is always preferable. This suggests a broader methodological context in which ReasonZoo can be read as an evaluation environment for testing when additional reasoning structure, including tool use, is beneficial and when it becomes wasteful.

4. TIR performance on ReasonZoo

The paper’s principal empirical result is that TIR consistently improves performance across both mathematical and non-mathematical tasks, although not uniformly (Zhao et al., 21 Aug 2025). On Qwen3-8B, average accuracy rises from 37.6% without TIR to 43.0% with PoT and 41.0% with MT-TIR. On Qwen3-32B, the average rises from 42.4% to 49.9% with PoT and 47.3% with MT-TIR. On Qwen3-235B-A22B, average accuracy increases from 45.8% to 52.8% with PoT and to 61.0% with MT-TIR. DeepSeek-R1-0528 improves from 41.9% to 55.3% with MT-TIR (Zhao et al., 21 Aug 2025).

The paper emphasizes that these gains are not confined to mathematics. It explicitly highlights improvements on Boolean Logic, Daily Logic, Puzzle, and Operations Research. For Qwen3-235B-A22B with MT-TIR, reported category scores include 71.6% on Boolean Logic, 77.6% on Daily Logic, 70.4% on Puzzle, and 58.0% on Operations Research (Zhao et al., 21 Aug 2025). This is the core evidentiary basis for the claim that TIR’s gains are domain-general rather than merely a mathematical effect.

The gains are nonetheless heterogeneous. Some categories and models show drops when tool use is introduced. The reported table shows that DeepSeek-R1-0528 with PoT improves on Number Calculation and Puzzle but drops on Grade School Math, Boolean Logic, Physics, and Formal Language (Zhao et al., 21 Aug 2025). The qualitative case studies make the caveat sharper: on abstract physics and some out-of-distribution problems, TIR can hurt because the model tries to force an awkward tool-based formulation onto a conceptually abstract task (Zhao et al., 21 Aug 2025).

This pattern aligns with a broader distinction between useful deliberation and counterproductive overthinking. The paper’s interpretation is that tool use helps when exact external computation matches task structure, but not when the task is primarily conceptual or when tool invocation is poorly aligned with the reasoning required.

5. Efficiency metrics: PAC and AUC-PCC

A central contribution of ReasonZoo is that it is paired with two metrics for reasoning efficiency: Performance-Aware Cost (PAC) and Area Under the Performance-Cost-Budget Curve (AUC-PCC) (Zhao et al., 21 Aug 2025). These metrics are intended to make the cost of reasoning visible rather than treating all correct answers as equally good.

PAC asks how much token cost is needed to reach a target performance level. The paper describes it as the minimum average normalized cost over all subsets that satisfy a given threshold. The equations are reported as

$\operatorname{PAC}_\tau = \min_{s \in \mathcal{S}_{\tau} \left\{ 1 - \frac{1}{\left\vert s \right\vert \cdot C_{max} \sum_{i \in s} C_i \right\},$

and

$$\operatorname{PAC} = \max_{\tau} \left\{ \operatorname{PAC}_\tau \vert \operatorname{PAC}_\tau > 0 \right\}.$$

The provided text notes visible formatting corruption in these expressions, but the surrounding explanation is explicit: higher PAC indicates that a model reaches good performance with less cost, which the authors interpret as more efficient reasoning and less overthinking (Zhao et al., 21 Aug 2025).

AUC-PCC summarizes performance across different token budgets. The paper defines normalized cost and performance as

$C_i' = \frac{C_i}{C_{max}, P_i' = \frac{P_i}{P_{max}, \end{equation}$

and computes

$$AUC\text{-}PCC = \sum_{i=1}^N \left[ \frac{P_i' + P_{i-1}'}{2} \times (C_i' - C_{i-1}') \right].$$

Again, the excerpt contains formatting corruption, but the intended interpretation is clear: a model is better if it achieves high performance early and sustains it as more budget is added (Zhao et al., 21 Aug 2025).

Empirically, PAC often improves under TIR. For Qwen3-8B, “All Tokens” PAC rises from 25.4 to 33.6 with PoT and to 32.8 with MT-TIR. For Qwen3-32B, it rises from 32.2 to 42.0 with PoT and 40.7 with MT-TIR. For Qwen3-235B-A22B, it rises from 34.6 to 42.6 with PoT and 49.2 with MT-TIR. DeepSeek-R1-0528 improves from 36.5 to 45.7 with MT-TIR. ToRL and CIR on Qwen2.5-Math-7B also improve, with ToRL moving PAC from 6.6 to 14.3 and CIR to 11.0 (Zhao et al., 21 Aug 2025).

AUC-PCC shows a similar pattern. Under MT-TIR, it improves from 32.92 to 33.30 for Qwen3-8B, from 38.31 to 39.62 for Qwen3-32B, from 37.10 to 46.18 for Qwen3-235B-A22B, and from 37.27 to 45.36 for DeepSeek-R1-0528. The largest reported gain is +9.08 on Qwen3-235B-A22B (Zhao et al., 21 Aug 2025).

A plausible implication is that ReasonZoo’s distinctive contribution is not only to broaden task coverage, but also to shift evaluation from a pure outcome view toward a performance–cost view. That perspective is closely related to work on adaptive routing under budget constraints, where reasoning systems are optimized for quality–efficiency trade-offs rather than for maximum deliberation alone (2505.19435).

6. Interpretation, attribution, and limitations

ReasonZoo is used in the paper not only to show that TIR improves correctness, but also to argue that it alters reasoning behavior. The authors state that TIR reduces overthinking by shifting work from verbose internal deliberation to precise external computation (Zhao et al., 21 Aug 2025). This interpretation is supported by the outcome-efficiency analysis in Table 4: for Qwen3-8B, the fraction of reasoning tokens used up to the first correct answer, denoted $s_i$0, improves from 0.116 to 0.214 with TIR; for Qwen3-32B, from 0.147 to 0.271; and for Qwen3-235B-A22B, from 0.161 to 0.343 (Zhao et al., 21 Aug 2025). The paper interprets this as evidence that more of the reasoning trace is productively spent before the model reaches the correct answer.

The attribution analysis further argues that tool use is genuinely responsible for a nontrivial part of the gains. Using an auxiliary model to classify whether improvements are tool-related, the authors report that on Qwen3-32B, 15.93% of previously wrong predictions become correct with tool assistance under PoT and 10.43% under MT-TIR. On Qwen3-235B-A22B, the corresponding tool-related correction rates are 18.98% and 20.15% (Zhao et al., 21 Aug 2025). At the same time, some initially correct answers from the base model become incorrect after tool use, indicating that TIR is not yet reliably faithful or always beneficial (Zhao et al., 21 Aug 2025).

The limitations are explicit. First, TIR does not universally help; on abstract or out-of-distribution tasks such as some Physics and Formal Language problems, tool use can interfere with otherwise correct verbal reasoning. Second, benefits depend on model scale and tool-calling competence: smaller models may incur enough overhead that TIR helps little or even hurts, as reflected in the appendix result where Qwen2.5-7B-Instruct drops in AUC-PCC from 3.57 to 2.96 under TIR, while larger Qwen2.5 models improve (Zhao et al., 21 Aug 2025). Third, PAC and AUC-PCC are token-based proxies rather than full measures of compute or monetary cost (Zhao et al., 21 Aug 2025).

These caveats are consistent with other contemporary work on reasoning control. "GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes" (Wang et al., 27 Nov 2025) reports both “reasoning collapse” on very hard samples and mild overthinking on easy cases, while "Route to Reason" (2505.19435) explicitly frames strategy selection as a budgeted routing problem because fixed “best model + best strategy” choices can induce redundant reasoning. This suggests that ReasonZoo’s broader significance lies in providing a benchmark where such claims about helpful versus wasteful reasoning can be tested systematically.

7. Position within the broader research landscape

ReasonZoo sits at the intersection of reasoning benchmarks, tool-use evaluation, and efficiency-aware inference. Its immediate contribution is to serve as the vehicle for the paper’s claim that TIR is not just a math helper but a domain-general method whose benefits and failure modes become visible only when evaluation spans heterogeneous task types and explicitly accounts for reasoning cost (Zhao et al., 21 Aug 2025).

The supplied materials also show that the term “ReasonZoo” can be used loosely or analogically in other contexts. REASONER is described as a real-user, multi-aspect, multi-modal explainable recommendation dataset plus benchmarking library, and one summary states that it is “likely what you mean by ‘ReasonZoo’” in the sense of a “zoo” of explanation data, tasks, and models for recommendation explanations (Chen et al., 2023). Likewise, one summary characterizes sourceR as “ReasonZoo” in the sense of a source-attribution framework for zoonotic infectious agents (Miller et al., 2017). These are not the formal benchmark introduced under the name ReasonZoo, but they indicate that the label can be used informally for diverse benchmark-like ecosystems.

In the precise arXiv sense, however, ReasonZoo refers to the benchmark introduced in "Dissecting Tool-Integrated Reasoning: An Empirical Study and Analysis" (Zhao et al., 21 Aug 2025). Its defining features are the nine-category reasoning suite, the explicit comparison between chain-of-thought and tool-integrated reasoning, and the paired efficiency metrics PAC and AUC-PCC. The benchmark’s central conclusion is that, at sufficient model scale and with appropriate TIR design, external tools can improve both accuracy and efficiency across a broad range of reasoning tasks, while simultaneously revealing the circumstances under which tool use becomes counterproductive (Zhao et al., 21 Aug 2025).