Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement

Abstract: Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.

• The paper introduces the AKBE framework, enabling dynamic per-instance evaluation of knowledge boundaries to choose optimal tool-use trajectories.
• It employs dual-path rollouts over seven QA benchmarks to maintain high answer accuracy while cutting redundant tool calls by 18%.
• AKBE integrates auxiliary cross-entropy losses with standard RL objectives, achieving up to 15% faster training and reducing hallucination risks.

Efficient Agentic Reinforcement Learning via On-Policy Knowledge Boundary Enhancement

Motivation and Problem Analysis

LLM-based agents operating in tool-augmented QA settings exhibit a persistent failure mode under standard agentic RL training: increasingly redundant tool calls, where the policy invokes external retrieval tools even when parametric knowledge suffices or invokes more tools than minimally necessary. This results in computational inefficiency and susceptibility to hallucination from noisy external information, as shown through trajectory degradation and tool-call growth during training (Figure 1).

Figure 1: Redundant tool-call growth in agentic RL (GRPO); correct samples transition to trajectories with more tool calls, some degrading to hallucinated/incorrect responses due to noisy retrieval.

Reward shaping methods that penalize tool calls at the outcome/reward level (e.g., OTC-PO, $\beta$-GRPO, HiPRAG) make coarse-grained optimization targets. These approaches incentivize indiscriminate tool call suppression, subject to reward hacking, and fail to model per-instance knowledge boundaries—i.e., whether a question is within the agent’s parametric knowledge or genuinely requires tool use. Such static or reward-coupled strategies are unable to adapt as the agent’s knowledge boundary evolves during RL training.

AKBE: Methodological Overview

AKBE (Agentic Knowledge Boundary Enhancement) proposes a fundamentally different on-policy instance-level signal construction paradigm. For each question during training, AKBE parallelizes dual-path rollouts: one with tool access, another without. Correctness is evaluated for both, enabling fine-grained categorization of the knowledge boundary per instance (Figure 2).

Figure 2: Framework of AKBE: dual-path rollouts (with-tool, no-tool) yield per-question signals guiding efficient tool-use in RL.

Four categories are identified:

• Tool-dependent: correct only with tool calls; select the minimum tool-call correct trajectory as the supervision target.
• Efficiency: correct in both paths; select correct no-tool trajectory, teaching avoidance of redundant tool calls.
• Hallucination: only correct in no-tool path; select no-tool trajectory, steering away from harmful tool reliance.
• Both-wrong: neither path correct; rely solely on RL signal, no additional supervision.

These knowledge boundary-guided signals are integrated as auxiliary cross-entropy losses, with a scaling coefficient $\lambda$, alongside the main RL objective (e.g., GRPO, DAPO, GSPO).

Empirical Results and Signal Mechanism

AKBE is evaluated across seven QA benchmarks (HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle, NQ, TriviaQA, PopQA), using Qwen3-4B and Qwen2.5-7B backbones. On both model scales, AKBE achieves the highest average EM while reducing tool calls by 18% and increasing tool productivity by 25%, with no trade-off between answer accuracy and efficiency.

AKBE’s on-policy signal construction dynamically tracks the evolving knowledge boundary, as evidenced by distributional shifts during training (Figure 3).

Figure 3: Early vs. late trajectory category distributions on Qwen2.5-7B Multi-Hop; Efficiency category increases, Both-wrong decreases, confirming knowledge internalization.

Reward-shaping baselines (OTC-PO, $\beta$-GRPO) reduce tool calls but collapse accuracy. Offline AKBE (static signal from frozen checkpoint) incurs premature tool suppression and significant accuracy degradation, underscoring the necessity for dynamic signal construction aligned to the current policy’s knowledge boundary.

AKBE’s plug-and-play compatibility is verified with GRPO, DAPO, GSPO, AEPO: in all cases, AKBE yields improvements in EM and tool efficiency regardless of the base RL algorithm. Ablation studies further demonstrate categorical signal contributions—Tool-dependent signals prevent over-suppression (accuracy collapse when removed), Efficiency signals are the main driver for redundancy elimination, Hallucination corrects over-retrieval-induced error.

Signal integration via cross-entropy is empirically superior to DPO-based preference optimization, which introduces instability and over-suppression in later training (Figure 4).

Figure 4: Cross-entropy signal integration maintains stable accuracy; DPO-based integration collapses after mid-training due to overly aggressive rejection of similar patterns.

Computational Efficiency and Dynamic Effects

Despite additional no-tool rollouts, AKBE actually accelerates training time, achieving 15% faster per-step duration than GRPO due to substantially lighter rollouts and “self-compounding” shorter trajectories as tool calls decrease (Figure 5). Tool calls and mean response lengths drop consistently throughout training (Figure 6), confirming compounding computational efficiency.

Figure 5: AKBE achieves faster average training time, as reduced redundant tool calls and lightweight no-tool rollouts outweigh additional rollout costs.

Figure 6: AKBE yields lower tool-call counts and shorter response lengths per step, reflecting efficiency improvements.

Analysis reveals that the majority of questions classified as "within knowledge boundary" (NT=1) during AKBE signal construction have multiple correct no-tool rollouts, indicating reliable boundary estimation rather than stochastic guessing (Figure 7).

Figure 7: Most NT=1 instances have high proportions of correct no-tool rollouts, evidencing reliable boundary estimation.

Signal Strength Calibration

Optimal $\lambda$ selection is shown to lie in $[0.05, 0.2]$, matching the theoretical ratio of AKBE to RL signal gradient contributions, empirically balancing accuracy gains and tool call suppression (Figure 8).

Figure 8: AKBE outperforms GRPO for moderate $\lambda$; accuracy drops if boundary-guided loss is overweighted.

Practical and Theoretical Implications

AKBE’s fine-grained, dynamically adaptive boundary modeling provides an effective mechanism for balancing accuracy and efficiency in RL-trained agentic LLMs. It avoids reward hacking, catastrophic over-suppression, and static misalignment intrinsic to earlier reward shaping and offline approaches. The plug-and-play nature suggests broad applicability across RL paradigms in agentic training, with theoretical calibration of signal strength and empirically validated signal construction reliability. AKBE advances the agent’s ability to internalize knowledge, suppress unnecessary cognitive offloading, and mitigate hallucination—all critical for scalable, efficient, tool-augmented LLM agents.

Practically, AKBE is expected to be a generalizable module for RL-based agentic systems, reducing deployment costs and inference latency in production. Theoretically, AKBE exposes a new axis for RL signal design: dynamic, on-policy, per-instance knowledge boundary probing, paving the way for meta-adaptive signal generation, curriculum-aware signal scaling, and more granular step-level supervision.

Conclusion

AKBE establishes a rigorous, on-policy boundary-guided agentic RL paradigm that achieves simultaneous improvements in accuracy and tool use efficiency without trade-off. It reveals nuanced signal design and integration mechanisms, exposes dynamic knowledge boundary evolution, and demonstrates robust empirical and computational advantages. Its implications support future development of adaptive RL training for agentic LLM systems, enabling efficient reasoning and judicious tool-use under dynamic knowledge boundary modeling (2605.26952).