Reinforcement Learning with Tool-Use Rewards (RLTR)

• Reinforcement Learning with Tool-use Rewards (RLTR) is a framework that assigns rewards to tool-use trajectories, decoupling planning from summarization for improved agent performance.
• It leverages a structured reward design—including tool-use completeness, repetition penalties, and fine-grained reward decomposition—to optimize multi-turn tool interactions in LLM agents.
• Empirical results show significant boosts in planning completeness and response quality, while addressing challenges like sparse rewards and over-verbosity in tool utilization.

Searching arXiv for papers on reinforcement learning with tool-use rewards and adjacent tool-learning RL frameworks. Searching arXiv for "reinforcement learning tool use rewards LLM" and specific RLTR-related frameworks. Reinforcement Learning with Tool-use Rewards (RLTR) names a planner-focused framework introduced for LLM agents and, more broadly, a family of reinforcement-learning schemes in which reward is assigned to tool-use trajectories rather than only to final textual answers. In its canonical form, RLTR decouples action planning from answer summarization, trains the planning module with a reward based on tool-use completeness, and reports an 8%–12% improvement in planning performance together with a 5%–6% increase in final response quality (Li et al., 27 Aug 2025). Across later work, the same general idea appears in fine-grained tool-call rewards, step-grained credit assignment, trajectory-level reward models, verification-based pseudo-rewards, and live-environment programmatic scoring for tool-augmented agents (Qian et al., 16 Apr 2025).

1. Conceptual scope and antecedents

RLTR extends standard RL to settings in which the policy must interleave reasoning with external tool calls over multi-turn trajectories. In representative formulations, the state contains the query or dialogue history together with prior tool outputs, the action space includes either next-token generation or explicit tool/API invocation, and the reward is decomposed into sub-metrics tied to tool use rather than treated as a single opaque scalar (Wu et al., 23 Mar 2026). ToolRL makes this explicit for Tool-Integrated Reasoning (TIR): after $k$ steps, the state is

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$

and the policy chooses the next reasoning output and tool calls so as to maximize expected cumulative reward over the interaction (Qian et al., 16 Apr 2025).

A precursor theme appears in robotics, where tool-use problems were cast as discounted MDPs with a shaped reward

$R(s,a)=\sum_i w_i\,r_i(s,a),$

combining “tool-grasping” and “task-specific” terms. In that setting, reward shaping, favorable initialization, and demonstration guidance were used to make high-dimensional tool behaviors learnable and generalizable to unseen tools (Mosbach et al., 2023). This suggests that RLTR is best understood not as a single reward formula, but as a design principle: tool execution itself becomes the object of optimization.

2. Canonical RLTR: decoupled planner optimization

The defining formulation of RLTR in LLM agents separates the agent into two modules: a Planner $\pi_p$, which produces a sequence of tool calls and a final ANSWER token, and a Summarizer $\pi_s$, which consumes the full trajectory $\tau$ and generates the natural-language response $y$. The training objective is

$$\pi_p^*=\arg\max_{\pi_p}\mathbb{E}_{\tau\sim\pi_p}[R(\tau)],\qquad y=\pi_s(\tau)\quad\text{where }\tau\sim\pi_p^*.$$

This decoupling is presented as an alternative to end-to-end agents that tie planning and summarization to a single reward on $y$, thereby inducing competing gradients and credit-assignment problems (Li et al., 27 Aug 2025).

The key RLTR signal is a tool-use completeness reward. A completeness checker $\gamma:\mathcal S\to\{0,1\}$, implemented by a small “Comp. Checker” LLM prompt, returns 1 if the trajectory covers all required tools/actions and 0 otherwise. Sampling the check $$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$0 times yields

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$1

This is augmented by a repetition penalty

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$2

and an invalid-tool penalty

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$3

The total reward is

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$4

with $$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$5. Planner optimization is then performed with a KL-regularized on-policy objective,

$$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$6

The point of this construction is methodological as much as algorithmic. RLTR “never touches $$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$7 during Planner training”; its reward depends only on the action sequence $$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$8 and a generic completeness check $$s_k=((r_1,\mathcal T_1,o_1),\dots,(r_k,\mathcal T_k,o_k)),$$9, which is presented as eliminating the need for ground-truth answers and avoiding reward hacking associated with answer scoring. The motivating examples are deliberately process-centric: an end-to-end system may be rewarded for a numerically correct answer produced by an incomplete tool sequence, whereas RLTR rewards the complete sequence Search("Beijing weather") → Search("Shanghai weather") → [Code](https://www.emergentmind.com/topics/karpathy-agent-code)(compute difference) because the plan itself is complete (Li et al., 27 Aug 2025).

3. Reward decomposition and signal design

A central axis of RLTR research is reward decomposition. ToolRL organizes the design space along four dimensions—reward type, scale, granularity, and temporal assignment—and advocates a two-component core reward

$R(s,a)=\sum_i w_i\,r_i(s,a),$0

where format compliance is separated from semantic accuracy. Its correctness term is itself decomposed into tool-name, parameter-name, and parameter-value matching, precisely because coarse exact-match rewards are too sparse for structured tool use (Qian et al., 16 Apr 2025).

Later work pushed this decomposition further. ToolSample defines four sub-rewards—$R(s,a)=\sum_i w_i\,r_i(s,a),$1—and uses them both for scalar scoring and for Dynamic Sampling with Curriculum Learning (DSCL), where reward statistics guide which examples are emphasized during training (Feng et al., 18 Sep 2025). ToolRLA introduces a domain-specific reward

$R(s,a)=\sum_i w_i\,r_i(s,a),$2

with multiplicative correctness

$R(s,a)=\sum_i w_i\,r_i(s,a),$3

and a compliance term with a large negative penalty $R(s,a)=\sum_i w_i\,r_i(s,a),$4. The multiplicative form is meant to enforce prerequisite structure: if the tool name is wrong, correctness is zero regardless of downstream parameter scores (Liu, 2 Mar 2026).

Programmatic reward design in live environments follows the same logic. PROVE evaluates trajectories with five components: graduated validity scoring, dependency-aware coverage, an adaptive efficiency penalty with a complexity-scaled call budget, a tool-name signal, and an argument-value matching bonus. The combined reward is

$R(s,a)=\sum_i w_i\,r_i(s,a),$5

and is computed without an external judge model (Abdelaziz et al., 2 Jun 2026). In multimodal settings, ToolsRL defines tool-specific rewards directly on tool semantics: modified $R(s,a)=\sum_i w_i\,r_i(s,a),$6 for zoom-in, binary orientation match for rotate/flip, and continuous coordinate rewards for line and point drawing (Dong et al., 21 Apr 2026). The cumulative picture is that RLTR has become increasingly explicit about what aspect of tool use is being rewarded: structural validity, semantic correctness, efficiency, compliance, or task-specific operational competence.

4. Temporal credit assignment, optimization, and curricula

RLTR differs not only in what it rewards, but in when and how credit is assigned. StepTool defines a step-grained reward at each tool interaction. For intermediate steps,

$R(s,a)=\sum_i w_i\,r_i(s,a),$7

and at the terminal step,

$R(s,a)=\sum_i w_i\,r_i(s,a),$8

The policy is optimized with PPO and GAE, so multi-step tool decisions receive local credit rather than being judged only by the final answer (Yu et al., 2024).

PORTool generalizes this idea by organizing rollouts into a tree. Shared early steps across multiple sampled trajectories receive the same step-wise reward, while diverging branches are compared by fork-relative advantages blended with trajectory-relative advantages. The resulting surrogate objective remains PPO-like, but the advantage is no longer tied to a single linear rollout; it reflects both branch quality and full-trajectory outcome (Wu et al., 29 Oct 2025). AutoTraj uses a different temporal mechanism: it trains a trajectory-level reward model $R(s,a)=\sum_i w_i\,r_i(s,a),$9 on repaired-versus-original trajectory pairs and combines this learned score with outcome and format rewards during GRPO optimization (Gong et al., 30 Jan 2026).

Optimization backbones are overwhelmingly PPO-family methods, but with important tool-specific modifications. TAPO adapts DAPO to tool invocation by masking tool-generated tokens during backpropagation, dropping the KL penalty, and shaping each sequence reward with format/accuracy plus a length penalty so that excessive tool calls and overly long reasoning traces are discouraged (Wu et al., 8 Oct 2025). ReTool interleaves natural-language reasoning with real-time code execution, uses a sparse end-of-episode reward $\pi_p$0, and also masks <interpreter> tokens from the loss; synthetic cold-start traces are used to bootstrap code-integrated reasoning before RL (Feng et al., 15 Apr 2025). ToolBrain exposes the same pattern as a framework: custom reward callables over execution traces, or batch ranking by an LLM judge, can be consumed by GRPO or DPO without changing the agent-facing API (Le et al., 24 Sep 2025).

Curriculum design has become equally central. ToolSample reweights sub-rewards across three stages—format focus, name/key focus, and value focus—while using reward-based dynamic sampling to suppress fully mastered examples (Feng et al., 18 Sep 2025). ToolsRL separates tool mastery from answer optimization: stage 1 is optimized solely by tool-specific rewards, and stage 2 is trained with answer-targeted rewards while allowing tool calls, explicitly to avoid optimization conflict among heterogeneous tasks (Dong et al., 21 Apr 2026). Tool-R0 replaces static data entirely with self-play: a Generator proposes tasks near a Solver’s competence frontier, and both are trained with verifiable rewards under GRPO (Acikgoz et al., 24 Feb 2026).

5. Empirical record across domains

The empirical record of RLTR is broad and domain-diverse. In the planner-centered formulation, RLTR reports an 8%–12% absolute boost in completeness and a 5%–6% increase in final-response accuracy when improved plans are fed to a fixed summarizer (Li et al., 27 Aug 2025). ToolRL, framed as the first comprehensive study of reward design for tool selection and application tasks, reports a 17% improvement over base models and a 15% gain over SFT models across BFCL V3, API-Bank, and Bamboogle (Qian et al., 16 Apr 2025). StepTool reports average pass-rate gains of 3–7 points over SFT and 5–10 points over RLHF-PPO on StableToolBench, together with a 5–8% improvement in tool invocation success rate (Yu et al., 2024).

On math, retrieval, and mixed fact-computation workloads, sparse or semi-sparse RLTR variants also show strong gains. ReTool reports that its 32B model achieves 67% accuracy with 400 training steps on AIME, compared with 40% accuracy and 1080 steps for a text-based RL baseline; in extended settings, ReTool-32B reaches 72.5% on AIME2024 (Feng et al., 15 Apr 2025). TAPO-trained Qwen2.5-7B reaches 61.1% on TAPO-easy-60K and 60.6% on TAPO-hard-18K, while also using fewer tool calls than baseline methods (Wu et al., 8 Oct 2025). AutoTraj raises average accuracy on its benchmark suite from 31.98 for Tool-Star to 34.38 and reduces average trajectory length from 1,078 tokens to 251 (Gong et al., 30 Jan 2026).

On function-calling benchmarks and production-like environments, the evidence is similarly consistent. ToolSample reports a 3.29% improvement on BFCLv3, with Multi-Turn performance increasing from 13.25% for ToolRL to 18.50% for DSCL (Feng et al., 18 Sep 2025). ToolRLA, deployed on a financial advisory copilot serving 80+ advisors, 1,200+ daily queries, and 15+ heterogeneous APIs, reports end-to-end task completion rising from 62% to 91%, tool invocation error falling from 38% to 14%, and regulatory violation dropping from 12% to 0.8% (Liu, 2 Mar 2026). PROVE, trained on live stateful MCP servers, reports gains of up to +10.2 on BFCL Multi-Turn, +6.8 on $\pi_p$1-bench, and +6.5 on T-Eval (Abdelaziz et al., 2 Jun 2026).

RLTR has also expanded beyond text-only agents. ToolsRL reports state-of-the-art results across document, spatial, and chart reasoning with simple visual tools such as zoom-in, rotate, flip, and point/line drawing (Dong et al., 21 Apr 2026). In long-horizon planning, the STAR study on TravelPlanner reports that a 7B RLTR-trained model reaches 62.8% success with GRPO plus dense Sum reward, while smaller models benefit more from curriculum rewards and exploration-heavy algorithms (Wu et al., 23 Mar 2026). In test-time adaptation, T$\pi_p$2RL shows that tool verification improves over standard test-time RL on MATH-500, AMC, and AIME 2024, with larger relative gains on harder problems (Liao et al., 2 Mar 2026).

6. Misconceptions, tensions, and open problems

One persistent misconception is that tool-using agents must be trained against verified final answers. The original RLTR formulation explicitly disputes this by using tool-use completeness rather than answer correctness to optimize planning (Li et al., 27 Aug 2025). A second misconception is that more reasoning or more tool calls are inherently desirable. ToolRL states that “Longer reasoning is not inherently better,” and several systems impose direct pressure against verbosity or over-calling: TAPO uses a length penalty, PROVE uses an adaptive efficiency penalty with a complexity-scaled call budget, and PORTool discounts long trajectories through a decay factor in its step-wise reward (Qian et al., 16 Apr 2025).

A separate tension concerns whether outcome rewards alone are sufficient. ReTool succeeds with a sparse $\pi_p$3 outcome reward in a code-interpreter environment, but AWPO argues that explicit reasoning rewards can preserve variance when outcome rewards saturate, particularly on hard or compositional tasks. AWPO therefore combines outcome and reasoning rewards through variance-aware gating, difficulty-aware weighting, and dynamic clipping rather than naïvely summing them (Lin et al., 22 Dec 2025). Test-time RL exposes a related failure mode: T$\pi_p$4RL shows that majority-vote pseudo-labels can reinforce a spurious yet high-frequency unverified consensus, causing incorrect mode collapse; verification-aware voting with tool checks is proposed as the remedy (Liao et al., 2 Mar 2026).

The main limitations in the literature are also recurrent. Judge-based systems can introduce bias or inconsistency: ToolBrain notes that LLM judge prompts may be biased, and AWPO explicitly proposes online judge calibration as future work (Le et al., 24 Sep 2025). Some optimization schemes show side effects, such as the response-entropy drop reported by TAPO (Wu et al., 8 Oct 2025). Environmental realism matters: STAR emphasizes that environmental stability is critical to prevent policy degradation, whereas PROVE notes the engineering cost of implementing live stateful servers for each domain (Wu et al., 23 Mar 2026). Outside LLM agents, robotic tool-use reward shaping remains largely simulation-bound, with real-world sensor noise and tracking left unresolved (Mosbach et al., 2023).

Open directions follow directly from these tensions. The literature names online judge calibration, hierarchical tools, multi-agent tool hierarchies, human-in-the-loop feedback, continuous verification scores, and extensions beyond tool use to code generation, retrieval-augmented generation, and grounding to external knowledge as natural next steps (Lin et al., 22 Dec 2025). Tool-R0 further suggests that zero-data self-play and competence-frontier curricula can become a general mechanism for open-ended tool-learning, although at substantial computational cost (Acikgoz et al., 24 Feb 2026). Taken together, these developments indicate that RLTR is evolving from a narrow solution to final-answer scarcity into a general methodology for allocating credit over structured tool-use processes.