THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning (2509.13761v1)

Abstract: LLMs have made remarkable progress in mathematical reasoning, but still continue to struggle with high-precision tasks like numerical computation and formal symbolic manipulation. Integrating external tools has emerged as a promising approach to bridge this gap. Despite recent advances, existing methods struggle with three key challenges: constructing tool-integrated reasoning data, performing fine-grained optimization, and enhancing inference. To overcome these limitations, we propose THOR (Tool-Integrated Hierarchical Optimization via RL). First, we introduce TIRGen, a multi-agent actor-critic-based pipeline for constructing high-quality datasets of tool-integrated reasoning paths, aligning with the policy and generalizing well across diverse models. Second, to perform fine-grained hierarchical optimization, we introduce an RL strategy that jointly optimizes for both trajectory-level problem solving and step-level code generation. This is motivated by our key insight that the success of an intermediate tool call is a strong predictor of the final answer's correctness. Finally, THOR incorporates a self-correction mechanism that leverages immediate tool feedback to dynamically revise erroneous reasoning paths during inference. Our approach demonstrates strong generalization across diverse models, performing effectively in both reasoning and non-reasoning models. It further achieves state-of-the-art performance for models of a similar scale on multiple mathematical benchmarks, while also delivering consistent improvements on code benchmarks. Our code will be publicly available at https://github.com/JingMog/THOR.

• The paper presents THOR, which integrates external tools and hierarchical RL to improve accuracy in complex mathematical reasoning tasks.
• It introduces the TIRGen pipeline, enabling actor-critic data construction for generating policy-aligned reasoning steps and precise code execution.
• Results show state-of-the-art performance across diverse benchmarks, validating the effectiveness of its self-correction and fine-grained reward mechanisms.

Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning

Introduction and Motivation

The paper introduces THOR, a framework for enhancing mathematical reasoning in LLMs by integrating external tools and hierarchical reinforcement learning (RL). The motivation stems from the persistent limitations of LLMs in high-precision tasks such as numerical computation and symbolic manipulation, where probabilistic next-token prediction is insufficient. While tool-integrated reasoning (TIR) has shown promise, existing approaches are hampered by three core challenges: (1) constructing high-quality, policy-aligned TIR data, (2) performing fine-grained optimization beyond trajectory-level RL, and (3) leveraging tool feedback for robust inference.

Figure 1: Overview of the THOR method, including motivation, the TIRGen data construction pipeline, the hierarchical RL framework, and experimental results.

TIRGen: Actor-Critic Data Construction Pipeline

A central contribution is TIRGen, a multi-agent actor-critic pipeline for generating TIR data. The Actor agent produces natural language reasoning steps, while the Critic agent identifies steps amenable to tool execution, converts them into executable code, and interacts with an external interpreter to obtain precise results. This iterative process yields a dataset that is both policy-aligned and broadly applicable across models and tools.

Figure 2: The TIRGen pipeline, where the Actor generates reasoning steps and the Critic transforms tool-executable steps into code-augmented reasoning, followed by multi-stage filtering.

Key advantages of TIRGen include:

• Reduced reliance on large models: The Actor handles reasoning, while the Critic only requires basic instruction-following and code generation.
• Policy alignment: The Critic operates on isolated steps, ensuring the generated data remains in-distribution for the Actor's policy.

The multi-stage filtering process enforces format consistency, code quality (requiring non-trivial code such as library calls or control flow), and problem difficulty, resulting in a high-quality cold start dataset.

Hierarchical Reinforcement Learning Strategy

THOR introduces a hierarchical RL framework that jointly optimizes at the trajectory and step levels.

Figure 3: The hierarchical optimization framework: (a) trajectory-level RL for problem solving, (b) step-level RL for code generation, and (c) a self-correction mechanism for inference.

• Trajectory-level RL: Utilizes GRPO, optimizing for final answer correctness. The reward is sparse, but directly targets the end-to-end reasoning objective.
• Step-level RL: Targets failed tool invocations by backtracking to the relevant reasoning step, partitioning it into prefix and suffix, and regenerating the suffix and action. The reward is based on execution correctness, providing dense, fine-grained feedback for code generation.

The final objective is the sum of trajectory-level and step-level losses, with masking of tool observations to stabilize training.

Self-Correction During Inference

THOR incorporates a self-correction mechanism during inference. When a tool call fails, the model backtracks to the relevant reasoning step, regenerates the suffix and action, and retries up to a fixed number of attempts. This approach leverages immediate tool feedback to dynamically revise reasoning paths, significantly improving robustness and final accuracy with minimal computational overhead.

Empirical Results and Ablation

THOR is evaluated on a suite of mathematical and code generation benchmarks, including MATH500, AIME 2024/2025, AMC, Minerva Math, OlympiadBench, HumanEval, MBPP, and LiveCodeBench. The framework is tested on both non-reasoning and reasoning models, with consistent improvements across all settings.

• State-of-the-art results: THOR achieves the highest average scores among models of comparable size, outperforming both TIR and CoT-based baselines.
• Generalization: The approach generalizes across model scales (1.5B to 8B) and architectures, and improves code generation in zero-shot settings.
• Ablation: Removing any component (cold start, trajectory-level RL, step-level RL, or self-correction) leads to a measurable drop in performance, confirming the necessity of each module.

  Figure 4: Ablation on cold-start efficiency, showing TIRGen's superiority over other TIR datasets in code ratio and pass@16 metrics.

  

  Figure 5: Distribution of code call rounds in the cold start dataset, demonstrating diversity in tool usage.

Qualitative Analysis

Case studies illustrate THOR's ability to solve complex mathematical problems by interleaving natural language reasoning with precise tool calls. The model efficiently invokes external tools for numerical computation, equation solving, and self-validation of reasoning steps.

Figure 6: Example of THOR-7B using external tools for precise computation on an AIME 2024 problem.

Figure 7: Example of THOR-7B solving and verifying complex equations with tool support on an AIME 2025 problem.

Figure 8: Example of THOR-Thinking-8B self-validating reasoning steps using tool integration on an AIME 2024 problem.

Implications and Future Directions

THOR demonstrates that hierarchical RL with tool integration and step-level feedback is highly effective for mathematical reasoning. The strong empirical results, especially the substantial gains in both reasoning and code generation tasks, suggest that dense, execution-based feedback is critical for overcoming the sparse reward problem in long reasoning chains. The policy-aligned data construction of TIRGen further mitigates out-of-distribution issues, a common failure mode in prior TIR approaches.

Theoretically, the finding that intermediate tool call success is a strong predictor of final correctness provides a foundation for future reward shaping and curriculum learning strategies. Practically, the framework's generality and low inference cost make it suitable for deployment in educational, scientific, and engineering applications requiring reliable automated reasoning.

Future work should explore:

• Extension to multi-tool and multi-modal integration (e.g., symbolic systems, search engines).
• Scaling to larger models and more diverse domains.
• Joint optimization of tool selection and invocation strategies.
• Further analysis of the interplay between tool feedback and model introspection.

Conclusion

THOR presents a comprehensive solution to the core challenges of tool-integrated reasoning in LLMs. By combining policy-aligned data construction, hierarchical RL, and self-correction, it achieves state-of-the-art performance on mathematical and code generation benchmarks. The approach is robust, generalizes across model types and sizes, and provides a blueprint for future research in tool-augmented LLM reasoning.