Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

Abstract: Group-based reinforcement learning (RL) has become an effective paradigm for improving LLM agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

• The paper introduces ProGPO, a framework that leverages context-consistent step grouping and rollout-derived transition credit to improve agentic RL in sparse-reward environments.
• It demonstrates significant performance gains on ALFWorld and WebShop, achieving up to 90.1% success compared to previous group-based methods.
• The study confirms that semantic expansion combined with multi-resolution fusion enhances state potential reliability and overall policy optimization.

Progress- and Reliability-Oriented Group Policy Optimization for Agentic RL

Motivation and Background

Group-based RL paradigms, such as GRPO and DAPO, have been effective in optimizing LLM agents for single-turn tasks due to their avoidance of learned critics and reliance on group-relative advantage estimation. However, agentic RL in long-horizon environments (e.g., WebShop, ALFWorld), where LLM agents must plan, act, and recover from mistakes over multiple sequential steps, introduces severe credit assignment challenges under sparse rewards. Existing trajectory-level optimization methods fail to deliver sufficiently granular updates, whereas step-level group-based approaches provide improved credit assignment but suffer from comparability and signal sparsity issues. Step-level grouping either sacrifices fairness by mixing histories (if grouped only by state) or yields fragmented groups and missing peer-comparison signal (if enforcing historical prefix consistency).

Figure 1: Motivation for ProGPO; state-only grouping mixes inconsistent histories, while context-consistent grouping creates small, often singleton peer groups.

Sparse Signal Diagnostics

Through empirical analysis in ProGPO training traces, the authors demonstrate the extent of sparse peer-comparison signal in context-consistent grouping. On WebShop, 33.7% of steps, and on ALFWorld, 44.5% of steps have zero prefix-consistent peer-comparison advantage prior to the addition of transition credit. This diagnostic reveals that context-consistent step grouping severely restricts directly usable signal, underutilizing many actionable steps.

Figure 2: Sparse-signal diagnostic; substantial proportions of steps lack peer-comparison signal or fall within small value groups, motivating transition credit.

ProGPO Framework

ProGPO introduces a learned-critic-free, context-consistent credit assignment mechanism tailored to agentic RL with LLM agents. The framework comprises two key components:

1. Prefix-Consistent Step Comparison: Steps are grouped by exact observation-history prefixes, ensuring action comparisons are drawn only from contextually identical histories.
2. Transition Credit from State Potentials: For steps lacking peer-comparison signal (singletons, low-contrast groups), transition credit is calculated using rollout-derived state potentials. These potentials are estimated through batch statistics, with semantic expansion for stabilizing sparse groups and inverse-variance multi-resolution fusion across prefix depths for reliability.

State potentials are estimated from rollout outcomes; combinatory fusion across eligible history depths weights estimates by sample count and variance precision. Semantic similarity expansion incorporates nearest-neighbor states by policy embedding proximity when exact groups are undersampled.

Figure 3: Overview of ProGPO; state potentials are estimated, expanded, and fused prior to transition credit calculation.

Empirical Results and Benchmark Comparisons

ProGPO was evaluated on ALFWorld and WebShop using Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct. In matched local reruns, ProGPO improved ALFWorld overall success from 75.8% (GRPO), 85.5% (GiGPO), and 87.8% (HGPO) to 90.1%. For WebShop, success improved to 71.5%, outperforming GiGPO (67.6%) and HGPO (65.6%) under comparable computational overhead. Breakdown by task diagnostics indicates robust improvements across Pick, Clean, Cool, Look, and Heat sub-tasks, with only minor deficits in two-object settings compared to HGPO.

The ablation studies exhibit the pivotal role of semantic expansion and reliability-aware fusion: transition credit alone is markedly weaker, achieving only 87.5% training success; combined stabilization components raise it to 96.9%. Larger-scale tests on Qwen2.5-3B-Instruct (ALFWorld) confirm scalability; ProGPO consistently achieves higher training and validation scores late in training.

Figure 4: Training curves highlight ProGPO’s superior convergence rates and final success metrics compared to GRPO, GiGPO, and HGPO.

Mechanism Diagnostics

The mechanism diagnostics indicate transition credit and peer comparison activate complementary subsets of steps. Transition-only signal is present in 11.4% of steps, but transition credit provides signal in a broader union (43.0% of WebShop and 63.3% of ALFWorld steps). The fusion of multi-resolution state potential estimates is prominent, with higher-depth estimates active in 82.5% of WebShop and 73.3% of ALFWorld steps.

Figure 5: Mechanism diagnostics presenting the evolution of transition-active, transition-only, and peer-comparison signal shares across training.

Implications and Limitations

The ProGPO framework addresses inherent weakness in credit sparsity for context-consistent group RL by introducing progress-sensitive transition credit, obviating the need for additional parametric value networks or learned critics. The method’s reliance on rollout-batch statistics and semantic expansion introduces practical efficiency without computational burden, demonstrating tangible improvements in agent success across diverse long-horizon tasks.

However, the approach presupposes sufficient recurrence within rollout batches; residual steps remain unscored when both the current and subsequent state lack reliable batch evidence. The quality and consistency of semantic state representation impact the reliability of state potential estimation, posing potential noisiness when unique or superficially similar states are encountered. Future directions include fortified state abstraction, improved uncertainty calibration, and adaptation to more complex open-ended environments and multimodal agent domains.

Conclusion

ProGPO advances group-based agentic RL by preserving the fairness of context-consistent step comparison while restoring denser credit assignment through robust, rollout-derived transition signals. Empirical results demonstrate pronounced improvements on ALFWorld and WebShop benchmarks compared to GRPO, GiGPO, and HGPO, with additional scalability shown in larger policy models. The methodological innovations—multi-resolution fusion and semantic expansion of state potentials—substantially enhance credit reliability, supporting more effective policy optimization in sparse-reward, long-horizon agentic settings (2607.04242).