ToolMATH-Hard: A Curated Math Benchmark

• ToolMATH-Hard is a curated hard subset of the ToolMATH benchmark, designed to evaluate tool-augmented reasoning in complex, long-horizon multi-tool environments.
• It features 329 challenging questions and 626 tools, where human-authored gold tools replace brittle, automatically extracted ones to ensure robust testing conditions.
• The benchmark emphasizes the need for coherent planning and control architectures, highlighting performance degradation in high-hop scenarios and error propagation in multi-step execution.

Searching arXiv for the benchmark and related tool-augmented math reasoning work to ground the article in current papers. ToolMATH-Hard is a curated hard subset of ToolMATH, a benchmark for evaluating tool-augmented LLMs in realistic multi-tool environments where the model must select among schema-specified tools, sustain multi-step execution, and return a correctness-checkable mathematical answer. It is derived from the broader ToolMATH framework, which converts MATH problems into reusable Python tools, but it targets a narrower regime: intrinsically difficult, long-horizon, brittle tool-use problems for which automatic extraction of usable gold tools failed even after repair. ToolMATH-Hard contains 329 questions and 626 tools, and functions as the stress-test portion of the benchmark for studying multi-step execution under both tool redundancy and tool insufficiency (Choi et al., 24 Feb 2026).

1. Origin within the ToolMATH benchmark

ToolMATH is organized as a collection of instances

$\{(p,\mathcal{S}(p))\},$

where $p$ is a problem from the MATH dataset and $\mathcal{S}(p)$ is the tool environment exposed to the model. Its core motivation is that deployed agents rarely operate with a small, clean, perfectly matched tool set; instead, they must reason over large catalogs with overlapping interfaces, missing capabilities, and long dependency chains. After validation, the main ToolMATH benchmark contains 7,699 questions and 12,369 tools (Choi et al., 24 Feb 2026).

ToolMATH-Hard was introduced to isolate the most brittle portion of this setting. Its defining construction criterion is not merely high surface difficulty, but failure of automatic tool extraction in a strong sense: for the selected questions, none of the extracted gold tools passed question-wise validation, even after a targeted human audit and repair stage. In the benchmark’s protocol, a tool is question-wise validated only if some validator model both solves the original problem correctly and successfully calls that tool in the trace. ToolMATH-Hard is formed by uniformly sampling 329 such questions and replacing brittle extracted tools with human-authored gold tools (Choi et al., 24 Feb 2026).

This construction has several consequences already emphasized by the benchmark design. The hard subset consists of problems that are difficult for direct reasoning without tools, difficult for automatic tool extraction, and typically more long-horizon and more sensitive to intermediate errors. The authors use it specifically to contrast tool availability versus tool insufficiency, and to compare control frameworks in settings where long-range execution coherence matters most.

2. Formal task structure, tool schemas, and environments

Each ToolMATH-Hard instance inherits the ToolMATH formalization: the model receives a problem $p$ together with a tool environment $\mathcal{S}(p)$. For each problem there is a gold tool set $\mathcal{G}(p)$, while distractors are drawn from a global pool $\mathcal{U}$ of validated tools. For distractor similarity level $\ell \in \{1,\dots,5\}$ and budget $k \in \{5,10,20,50\}$, the benchmark defines

$$\mathcal{S}_{\ell,k}(p)=\mathcal{G}(p)\cup \mathcal{D}_{\ell,k}(p),$$

and the distractors-only environment

$p$0

The five similarity levels range from different-category random sampling to keyword-overlap plus embedding similarity, and distractor sets are precomputed in ordered lists so that larger budgets strictly extend smaller ones (Choi et al., 24 Feb 2026).

A ToolMATH tool is specified by a globally unique Python function name, a natural-language description, a typed input schema, a hidden deterministic Python implementation, and provenance metadata linking it to source problems and solution steps. Models do not see the implementation. They interact only through a strict JSON action schema of the form $p$9 and receive environment feedback as $\mathcal{S}(p)$0 The environment checks JSON validity and schema conformity before execution, and enforces a duplicate-call cache so that identical $p$1 calls reuse cached results (Choi et al., 24 Feb 2026).

ToolMATH-Hard uses the same global distractor pool as the main benchmark, but its gold tools are human-authored rather than automatically extracted. In comparative evaluations between ToolMATH and ToolMATH-Hard, distractors are fixed to Level 2 random distractors with $p$2. The subset therefore preserves the benchmark’s realistic catalog setting: a problem may expose one or a few gold tools together with ten unrelated yet valid tools sampled from all other problems. Appendix traces show that in distractors-only mode, models can occasionally repurpose such tools as substitutes, for example by treating complex numbers as two-dimensional vectors and using a generic vector-addition tool, or by mapping a denominator polynomial into the arguments of a rational-equation solver (Choi et al., 24 Feb 2026).

3. Logical hops, execution protocol, and measurement

A central notion in ToolMATH-Hard is the logical hop count $p$3, which approximates the depth of dependent tool use required to solve a problem. Hop count is derived from solution steps, but it is not a raw count of tool invocations; parallelizable operations are grouped into the same hop layer. For evaluation, hops greater than or equal to eight are bucketed as “hop8+”. This makes it possible to separate intrinsic long-horizon difficulty from the effects of tool-list overlap and distractor density (Choi et al., 24 Feb 2026).

Episodes are run in a multi-step agent loop. At each step, the model emits a reasoning segment (“Thought: ...”), may emit a tool call through the strict action schema, receives an observation with the tool output, and continues until it produces a terminal answer of the form ANSWER: <numeric answer>. Correctness is defined by exact-match answer accuracy after normalization against the MATH ground truth. No separate aggregate metric is assigned to tool-call correctness; traces are logged and later inspected for error analysis (Choi et al., 24 Feb 2026).

The benchmark supports several evaluation regimes. In gold-only mode, the model receives only $p$4; in gold-plus-distractors mode it receives $p$5; in distractors-only mode the intended capability is absent. Comparative analyses between ToolMATH and ToolMATH-Hard fix distractors at Level 2 with $p$6 and report results by hop group under three conditions: no tools, gold-only, and distractors-only. Framework comparisons instead use gold-only environments to isolate the contribution of the control architecture itself (Choi et al., 24 Feb 2026).

The released benchmark artifacts follow this organization. Tool catalogs are stored as JSON objects containing name, description, input schema, implementation pointer, and provenance; problem instances include the problem text, gold answer, gold tool set $p$7, and hop annotation $p$8; and distractor pools are precomputed as ordered lists of 100 distractors for each problem and similarity level.

4. Empirical difficulty and characteristic failure modes

The benchmark’s central empirical result is that accuracy decreases almost monotonically with hop count, and that this degradation is materially steeper on ToolMATH-Hard than on the main ToolMATH benchmark. In ToolMATH-Hard, low-hop questions are rare, hop 1 does not appear in framework comparison plots, and high-hop buckets dominate. Even in gold-only mode, where the correct tools are present, accuracy collapses at high hops, indicating that access to the intended capability does not by itself solve the core problem of long-range execution control (Choi et al., 24 Feb 2026).

Human error analysis reinforces that interpretation. The authors manually label 100 failed episodes per model in gold-present settings using a taxonomy that includes Plan Error, Tool Selection Error, Tool Hallucination, Wrong Parameter Value, Formatting Error, Thought Error, Observation Omission, Repeated Call, and Incomplete Execution. Thought Error dominates more than 90% of failed cases. This is significant because it rules out a narrow “API formatting” explanation of failure; the dominant bottleneck is reasoning that misuses, ignores, or drifts away from tool outputs, rather than mere inability to emit syntactically valid calls (Choi et al., 24 Feb 2026).

Tool-list redundancy also behaves in a more structured way than simple additive noise. The paper explicitly argues that redundancy does not simply add noise, but amplifies small early deviations into irreversible execution drift. On ToolMATH-Hard, the separation between gold-only and distractors-only grows larger than on ToolMATH as hop count increases, which indicates that missing the intended capability is especially damaging once multi-step dependencies accumulate. Distractors can occasionally serve as partial substitutes, but they more often redirect the model into ungrounded tool trajectories (Choi et al., 24 Feb 2026).

The model-level comparisons illustrate this asymmetry. Qwen2.5-7B sometimes outperforms the no-tools baseline even in distractors-only conditions on the hard set, suggesting that it can repurpose generic tools when the intended tool is absent. Llama 3-8B, by contrast, often fails to exploit distractors and can underperform the no-tools baseline in distractors-only mode. This does not alter the overall trend: all models have lower absolute accuracy on ToolMATH-Hard than on ToolMATH at the same hop buckets.

5. Planning protocols and control architectures

ToolMATH-Hard is also a benchmark for control architectures. The evaluated frameworks are no tools, ReAct, DFSDT, and Plan+ReAct. ReAct interleaves reasoning and tool calls without a distinct planning stage. Plan+ReAct first produces a short numbered plan and then executes it in ReAct style, marking deviations with a <PLAN_CHANGED> token. DFSDT, in the style of ToolLLM, uses an external controller to manage branching, restart actions such as give_up_and_restart, and diversity prompts intended to force alternative actions when previous ones fail (Choi et al., 24 Feb 2026).

The framework comparison shows that low-hop problems do not strongly differentiate these protocols: at shallow depths, ReAct, DFSDT, and Plan+ReAct have similar performance. The separation becomes sharp as hop count increases, especially on ToolMATH-Hard. Plan+ReAct dominates at higher hops; DFSDT generally improves over plain ReAct at intermediate hops but remains below Plan+ReAct in the most difficult regime. The no-tools baseline degrades fastest with hop, underscoring that these tasks require tool-mediated computation rather than internal arithmetic alone (Choi et al., 24 Feb 2026).

The paper’s interpretation is that long-range plan coherence, rather than local action selection, is the decisive control property in the hard set. Simple local interleaving is insufficient when small early errors constrain later choices, and search-style branching alone does not compensate for the absence of a globally coherent plan. The findings therefore shift emphasis from tool syntax and per-step scoring toward controller design, observation discipline, and explicit maintenance of cross-step data flow.

Model-specific sensitivities align with this view. Llama 3-8B shows the steepest degradation with hop on ToolMATH-Hard and benefits most from the planning framework. GPT-4o-mini degrades more smoothly, but its gains from planning are still consistent; it also exhibits high rates of Repeated Call failures, indicating weak self-correction when trapped in loops. Qwen2.5-7B attains the best absolute performance overall, yet follows the same qualitative pattern: Plan+ReAct becomes increasingly decisive as hops grow (Choi et al., 24 Feb 2026).

6. Design rationale, interpretive lessons, and common misconceptions

ToolMATH-Hard was constructed to disentangle several confounded factors in tool-augmented evaluation. First, the “no question-wise validated tools” criterion concentrates the benchmark on intrinsically hard reasoning problems rather than merely noisy extraction artifacts. Second, replacing brittle extracted tools with human-authored gold tools minimizes semantic noise in the tool layer, so that observed failures are more plausibly attributable to planning, reasoning, and execution control. Third, the benchmark preserves both redundancy and insufficiency, making it possible to evaluate not only idealized tool use but also fail-safe behavior when the intended capability is absent (Choi et al., 24 Feb 2026).

Several misconceptions are therefore directly contradicted by the benchmark’s evidence. ToolMATH-Hard is not simply a schema-following test, because Thought Error rather than Formatting Error dominates failure analysis. It is not merely a distractor benchmark, because accuracy remains poor at high hops even in gold-only mode. It is also not best understood as a benchmark of faulty tools, because the hard subset uses human-authored and manually audited gold tools precisely to reduce that confound (Choi et al., 24 Feb 2026).

The paper’s design insights further suggest a concrete diagnosis of current tool-using LLMs. Stable long-horizon reasoning is weak; disciplined use of observations is inconsistent; data-flow handling remains fragile, as seen in Wrong Parameter Value and Observation Omission; and fail-safe behavior under insufficiency is underdeveloped, since many models continue to pursue unsupported or ungrounded trajectories rather than abstaining or reverting to direct reasoning. The paper therefore emphasizes stronger base reasoning, better planning and reflection mechanisms, more robust controllers, and tool-aware learning as the most plausible improvement directions.

The benchmark also has explicit limits. It is math-grounded and correctness-checkable, which makes automatic scoring possible, but this same design means it does not capture the ambiguity and open-endedness of many real-world tasks. The authors also note that validation reduces but cannot entirely eliminate extraction noise, and that results depend on specific protocols and prompting.

7. Relation to adjacent benchmarks and tool-augmented math agents

ToolMATH-Hard belongs to a broader post-2025 literature on hard, verifiable, tool-aware mathematical reasoning, but it occupies a distinct niche. MathConstraint is an adaptive generator-verifier benchmark for combinatorial reasoning that uses parameterized CSP families, solver-backed verification, and an explicit sandboxed Python environment with SAT/SMT tools. It contains 329 hard instances, frontier no-tools accuracy ranges from 18.5% to 66.9%, tool access roughly doubles frontier accuracy on the hard set with a mean gain of 28 percentage points, and halving the tool budget from 8 to 4 rounds can erase up to 37 points. Relative to ToolMATH-Hard, MathConstraint emphasizes solver-backed witness construction and explicit budget sensitivity, whereas ToolMATH-Hard emphasizes large overlapping tool catalogs, missing capabilities, and execution drift under multi-hop dependency (Pati et al., 8 May 2026).

A second adjacent line is represented by AgentMath, which is not a benchmark but a tool-augmented agent framework for competition-level mathematics. AgentMath interleaves >, <code>, and <interpreter> segments, trains on approximately 316k tool-augmented trajectories, and in RL training reports average trajectories of about 24k tokens with about 27 tool calls, with maxima of 96k tokens and 96 tool calls. Its results on AIME24, AIME25, and HMMT25 show that explicit tool-augmented training and interleaved execution can substantially improve hard-math performance (Luo et al., 23 Dec 2025).

Taken together, these works delineate complementary research directions. ToolMATH-Hard serves as a diagnostic stress test for realistic multi-tool reasoning under redundancy and insufficiency; MathConstraint provides an adaptive generator-verifier framework for combinatorial tool use under explicit round budgets; and AgentMath studies how training procedures can produce agents able to exploit such environments effectively. A plausible implication is that future progress on ToolMATH-Hard will depend less on incremental improvements in local tool selection and more on integrated advances in benchmark design, controller architecture, and training for long-horizon observation-grounded planning.