Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

Abstract: Humans know when to reach for help e.g. $347 \times 28$ warrants a calculator while $2+2$ does not. LLMs do not. Prompt-based approaches can instruct a model when to invoke tools, but this scaffolding does not teach it to recognize the boundary of its own knowledge. RL approaches that assign a single outcome reward to the whole trajectory fare no better: trajectory-level credit cannot isolate which tool call in a successful episode actually helped, nor penalize unnecessary calls. We propose \textbf{CARL} (\textbf{C}ompetence-\textbf{A}ware \textbf{R}einforcement \textbf{L}earning), which trains a critic on the model's own rollouts to learn where parametric knowledge suffices and where it needs external help. By decomposing each rollout at natural tool-use boundaries (e.g., code fence delimiters and context block transitions), CARL assigns independent credit to each segment from a single binary outcome, without external judges or step-level annotations. As a result, erroneous tool calls, incorrect extractions, and unnecessary calls each receive appropriately signed advantages. The trained critic captures the model's domain competence: it separates parametrically solvable from tool-dependent questions with AUC 0.93 at 7B. On five benchmarks spanning arithmetic, multi-hop factual QA, and numerical reasoning over financial tables, CARL improves exact-match accuracy by 6.7 points at 7B and 9.7 points at 3B over the best RL baseline, with the largest gain (+8.3 EM at 7B, +9.0 EM at 3B) on Musique. The model issues 53\% fewer tool calls on parametrically answerable questions while remaining ${\sim}10$ EM points more accurate on them. Gains are largest at small scale: the 3B improvement is $1.4\times$ the 7B improvement, suggesting that knowing when to ask disproportionately benefits models with smaller parametric memory.

• The paper presents CARL, which decomposes tool-use trajectories into invoke, assimilate, and commit segments for precise credit assignment.
• It employs a value head critic to compute per-segment advantages, leading to significant improvements in exact-match accuracy and reduced unnecessary tool calls.
• CARL achieves notable efficiency gains, including up to +13.8 EM improvement and a 23–35% token reduction, especially benefiting smaller models in multi-hop tasks.

Segment-Level Credit Assignment for LLM Tool Use: A Formal Analysis of CARL

Problem Formulation and Limitations of Prior Approaches

The paper “Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use” (2605.27788) addresses the critical challenge of credit assignment in the training of LLMs that can invoke external tools (e.g., code execution, search engines). The foundational observation is that while humans possess intrinsic metacognitive competence to selectively invoke tools only when their own parametric knowledge is insufficient, LLMs—trained with either purely prompt-based or trajectory-level reinforcement learning (RL) protocols—lack such selectivity. Prompted models treat the tool interface as a scaffold but do not obtain gradient feedback on the when-to-use decision, and trajectory-level RL methods, which assign a uniform outcome-based reward to the entire tool-use trajectory, fail to localize which tool calls improved or harmed outcomes. As a result, unnecessary or misapplied tool invocations are reinforced identically to genuinely beneficial ones.

The core limitations in prior RL-based tool use—including variants such as Search-R1 PPO, GRPO, and GRPO with step-level manipulation—are (i) the inability to perform fine-grained credit assignment to different tool-use decisions within a trajectory, and (ii) no explicit modeling of the model’s own competence boundary, i.e., distinguishing between cases where tool use is actually required versus redundant.

CARL: Competence-Aware Reinforcement Learning

The proposed method, CARL (Competence-Aware Reinforcement Learning), directly addresses both deficiencies via a semi-Markov decision process (SMDP)-grounded formulation for segment-level credit assignment. Tool-use trajectories are decomposed into structurally-observable segments, specifically: invoke (formulating a tool call, e.g., searching or code execution), assimilate (integrating tool outputs into context), and commit (final answer generation). Each segment boundary corresponds to a substantive change in the internal state of the episode (such as the transition from a reasoning context to a post-tool-output context), making credit assignment tractable and semantically meaningful.

Figure 1: The rollout pipeline in CARL, with explicit segmentation at invoke, assimilate, and commit boundaries, and the critic $V_\phi$ evaluated at each boundary to produce per-segment advantages.

A value head critic $V_\phi$ is trained to predict the trajectory-level binary reward at these boundaries. The fundamental insight is that by observing the change in $V_\phi$ between boundaries (for intermediate segments) and the mismatch with the terminal outcome (for the commit segment), it is possible to derive segment-specific advantage estimates solely from the terminal reward, without requiring external (e.g., GPT-4) step judgements or dense step-level annotation. The per-segment advantage for each segment $k$ is

$$A(\mathrm{seg}_k) = \begin{cases} V_\phi(s_{k+1}) - V_\phi(s_k), & k < N\ R - V_\phi(s_N), & k = N \end{cases}$$

This reallocation of credit enables the assignment of positive or negative reinforcement specific to the quality of the tool use (or its absence), and crucially, unnecessary tool calls receive near-zero advantage when $V_\phi(s_k)$ is properly calibrated.

Empirical Results and Analysis

Experiments are performed using Qwen2.5-3B/7B-Instruct as the policy backbone on a suite of five benchmarks (GSM8K, HotpotQA, 2WikiMQA, FinQA, Musique) that cover diverse tool types (search, Python code execution), multi-hop reasoning, and both in-distribution and OOD settings. CARL is compared against strong baselines: prompt-based tool-use, SFT on correct episodes, and two RL baselines (Search-R1 PPO and GRPO). Training and evaluation rigor includes matched warm-up for critic initialization, scale and ablation sweeps, and multiple seeds.

CARL achieves substantial improvements in both exact-match accuracy and tool-use efficiency as compared to the best RL baseline:

• On search-based tasks (HotpotQA, 2WikiMQA, Musique), 7B models see +6.7 EM (exact-match) improvement (+13.8 EM at 3B) over Search-R1 PPO; gains concentrate on harder multi-hop and out-of-distribution tasks (+8.3 EM on Musique) and are more pronounced at smaller scale.
• On questions solvable by parametric knowledge (“Tier 2”), CARL invokes tools 53% less often, but remains $\sim$10 EM points more accurate than baselines that make unnecessary tool calls.
• The segment-level critic $V_\phi(s_0)$ provides a highly calibrated boundary of parametric competence (AUC 0.93 at 7B), enabling robust selectivity; positive advantages align with meaningful tool calls, and negative/zero advantages emerge for unnecessary or erroneous interactions.
• Gains scale inversely with backbone size: the 3B model's improvement is 1.4× larger than the 7B model's, consistent with the hypothesis that segment-level credit assignment is especially beneficial when parametric “memory” is more limited.

Additional analyses show that segment-level credit assignment allows single trajectories to carry within-trajectory sign divergence (e.g., a positive credit for an informative search but a negative credit for an erroneous extraction in the same rollout), which is not achievable under trajectory-level RL. Tool-abuse and hallucination rates also decrease substantially, with models learning to avoid hallucinated answers and revert to parametric knowledge on Tier 2 queries.

Theoretical and Practical Implications

The methodological contribution of CARL lies in making RL-based tool use for LLMs both selective and aligned with the policy’s actual knowledge limitations. By exploiting the structural boundaries present in tool-augmented LLM rollouts, the segment-level SMDP value function encodes nuanced epistemic awareness, not merely repeated tool invocation strategies. This presents a robust solution to the RL credit assignment bottleneck for LLMs, avoiding the pitfalls of step-level reward shaping and external judge divergence.

The selective tool-use regime realized by CARL has significant implications:

• Model Efficiency: Through reducing unnecessary tool invocations and extraneous generation, CARL models achieve 23–35% reduction in episode token count and 17–18% fewer tool calls, directly translating into reduced latency and operational cost in deployment, especially important in latency-and-cost-sensitive environments.
• Scalability for Edge Models: The finding that improvement is more pronounced at smaller scale (3B) suggests segment-level credit assignment is a practical alternative to simply increasing parametric model size for applications where model footprint matters, such as on-device assistants or local coding agents.
• Tool Modularity and Routing: The structural framework of CARL readily extends to richer tool ecosystems, and the calibrated $V_\phi(s_0)$ could function as a dynamic routing signal for downstream tool-selection meta-agents.

Limitations and Future Directions

Despite significant advances, the approach has inherent trade-offs:

• Warm-up dependency: CARL requires a well-calibrated critic prior to PPO fine-tuning; cost and quality of warm-up directly affect sample efficiency and peak performance.
• Segment granularity: Intra-segment token-level credit variation is not modeled; if fine-grained token-level miscrediting within a segment is important, further decomposition would be required.
• Compute and engineering: Segment inference with mid-generation boundary detection and context replacement creates engineering friction with existing LLM serving stacks.
• Upstream limitations: In benchmarks with imperfect retrieval (e.g., BM25 on Musique), the main error bottleneck shifts to retrieval rather than credit assignment, as shown in ablations using high-quality search APIs.

Conclusion

The segment-level SMDP-based credit assignment introduced in CARL defines a new standard for RL-driven tool-use policies for LLMs, yielding simultaneously more accurate and more selective tool invocation without reliance on external supervisors. It aligns reinforcement signals precisely with the epistemic transition points in tool-augmented reasoning, resulting in models with emergent competence awareness and efficient, human-like selectivity. As tool-augmented LLM workflows become increasingly complex and multi-modal, the structural insights of CARL will likely inform subsequent advances in agentic reasoning architectures, meta-cognitive routing, and RLHF at scale.