Effects of policy entropy on agentic RL training

Characterize how policy entropy affects training effectiveness, stability, and final performance in GRPO-based agentic reinforcement learning where language model agents interleave tool calls with internal reasoning.

Background

Conflicting prescriptions exist in the literature: some advocate entropy minimization for determinism, while others emphasize high-entropy tokens to avoid early collapse. Tool-call steps may induce useful uncertainty in agentic RL.

The authors stress that the general role of entropy in agentic RL remains unclear, motivating experiments on clipping and entropy regimes to identify balanced exploration that avoids instability without premature convergence.