Optimal RL recipe for agentic reasoning

Develop the optimal reinforcement learning training recipe for agentic reasoning in large language model agents that integrate external tools, specifying the algorithmic components and settings that yield the best performance and stability.

Background

The paper highlights that GRPO-based variants for agentic reasoning differ widely in optimization granularity (token-, sequence-, or trajectory-level), clipping, KL regularization, and entropy management, yet a principled best practice is not established. This uncertainty motivates the authors’ systematic comparison of GRPO recipes and techniques.

Clarifying the optimal recipe is crucial for scaling agentic RL reliably, avoiding entropy collapse, and ensuring efficient exploration when LLM agents interleave internal reasoning with external tool calls.

References

Despite rapid progress in GRPO-based variants, the optimal RL recipe for agentic reasoning remains unclear.

— Demystifying Reinforcement Learning in Agentic Reasoning  (2510.11701 - Yu et al., 13 Oct 2025) in Introduction, Algorithm-wise paragraph (#1{2})

Effective reward design, credit assignment, and optimization stability remain open challenges, particularly under joint multimodal interaction and agentic execution.

— Omni Interaction Agent Technical Report  (2609.08977 - Orantqing et al., 8 Sep 2026) in Section 6, “Conclusion and Future Work,” bullet “Stable Post Training”

Finally, while GRPO shows advantages over PPO in our setting, improving the stability and effectiveness of RL methods for structured generation remains an open challenge.

— When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use  (2608.11715 - Chauhan et al., 12 Aug 2026) in Section 8.4, Limitations and Future Directions

How to model and assemble these calls into training samples remains an open question.

— Agent Lightning v1.0: Towards Harnessed Agentic RL  (2608.17528 - He et al., 18 Aug 2026) in Abstract; discussed in Sections 1 and 2

It remains unclear whether the same routing paradigm generalizes to substantially different settings such as agent planning or open-ended instruction following.

— From Rollouts to Recipes: Self-Contained Post-Training for LLMs  (2609.01422 - Li et al., 1 Sep 2026) in Limitations section, p. 7