GRPO: Group-Based Relative Policy Optimization

• GRPO is a reinforcement learning framework that computes relative advantages through intra-group comparisons, removing the need for a critic.
• Its integration in GRPOformer employs Transformer networks to sequentially propose and evaluate hyperparameter configurations, enhancing sample efficiency.
• Policy Churn Regularization (PCR) stabilizes training by dampening drastic policy updates, ensuring robust performance in sparse-reward and complex settings.

Group-Based Relative Policy Optimization (GRPO) is a family of reinforcement learning algorithms that introduce groupwise, relative reward normalization for efficient and stable policy optimization, with broad applications in both machine learning system tuning and domain-specific control problems. GRPO eliminates the need for a learned value function (critic) by computing advantages through intra-group comparisons, and is especially valuable in complex or sparse-reward settings where critic estimation becomes unstable or uninformative. This approach has been instrumental in recent advances in large-scale LLM alignment, hyperparameter optimization, robotics, and multi-objective learning.

1. Theoretical Foundations and Mathematical Formulation

GRPO builds upon the Proximal Policy Optimization (PPO) framework but replaces value-based advantage estimation with group-based comparisons. In its canonical form, for a group of $K$ candidate actions $\{a_1,\ldots,a_K\}$ drawn from the policy, with corresponding rewards $r(a_k)$, the relative advantage for action $a_k$ is: $A(a_k) = r(a_k) - \frac{1}{K}\sum_{j=1}^K r(a_j)$ where $r(a_k)$ is the immediate reward, and the baseline is the mean reward within the group.

The GRPO objective adopts PPO's clipped surrogate loss, substituting standard advantages with relative advantages: $$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E} \Big[ \min \left( \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A(a),\, \text{clip}\left(\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}, 1-\epsilon, 1+\epsilon\right)A(a) \right) \Big]$$ where $\pi_\theta$ is the current policy, $\pi_{\theta_{\text{old}}}$ is the previous policy, and $\epsilon$ is the clipping parameter to prevent destabilizing policy updates. This critic-free design leads to stable optimization dynamics, especially in environments where learning a reliable value function is impractical or inefficient.

2. GRPOformer: Transformer-Integrated Hyperparameter Optimization

GRPOformer demonstrates a state-of-the-art application of GRPO in hyperparameter optimization (HPO) via integration with Transformer architectures. The framework uses a Transformer as a policy network to sequentially propose new hyperparameter configurations, conditioning on:

• Current task identity ($\tau$),
• Hyperparameter search space ($\mathcal{H}$),
• Historical optimization trajectory ($\mathcal{T} = \{(h_1, r_1), \ldots, (h_t, r_t)\}$).

The Transformer models

$$p(h_{t+1} | \mathcal{T}, \tau, \mathcal{H}) = \mathcal{F}_\theta(\mathcal{T}, \tau, \mathcal{H})$$

where $\mathcal{F}_\theta$ is parameterized by $\theta$.

At each iteration:

1. Groups of new hyperparameter candidates are sampled from the Transformer;
2. Each candidate is evaluated on the underlying ML task to yield rewards;
3. Relative advantages are computed within the group;
4. The Transformer’s weights are updated according to the GRPO loss above, using these groupwise advantages;
5. The optimization trajectory is augmented with newly evaluated candidates.

This integration achieves parallel evaluation, rapid exploitation of observed structure, efficient use of limited experimental budgets, and improved sample efficiency in HPO tasks (Guo et al., 21 Sep 2025).

3. Policy Churn Regularization (PCR) for Stable Training

To address the practical issue of policy “churn”—unintended large shifts between policy iterations that can manifest in PPO-style RL, PCR is introduced as an auxiliary KL-regularization term. PCR penalizes the KL divergence between the current and previous policies: $$L_{\text{PC}} = \mathbb{E}_{s \sim D} \left[ D_{\text{KL}} \left(\pi_{\text{old}}(\cdot|s) \| \pi_{\theta}(\cdot|s)\right) \right]$$ where $D$ is a set of reference states (early trajectory positions), and $\lambda_{\text{PC}}$ controls the regularization strength. The total GRPO loss becomes

$$L_{\text{policy}} = L_{\text{GRPO}} + \lambda_{\text{PC}}L_{\text{PC}}$$

PCR dampens policy oscillations, smooths learning curves, and enables aggressive exploration without instability—a property shown to be critical for sample-limited HPO and optimization tasks (Guo et al., 21 Sep 2025).

4. Empirical Performance and Ablation Studies

GRPOformer was empirically tested on 36 HPO tasks across 6 ML models and 6 datasets within the OpenML platform. Key baselines included BOformer, LLM-HPO, OPTformer, and SBOA. Across all evaluation metrics—BtR (Beat random), Median normalized performance, Mean normalized performance, and Mean Rank—GRPOformer outperformed all comparators:

Model	BtR (%)	MP	MnP	MnR
GRPOformer	94.44	0.9545	0.9187	1.81
BOformer	88.89	0.6437	0.5767	3.14
LLM-HPO	83.33	0.3224	0.0863	3.83
OPTformer	77.78	0.5882	0.4360	3.75
SBOA	91.67	0.8983	0.8678	1.92

Ablation demonstrates that removal of PCR reduces BtR and normalized performance (e.g., BtR drops from 94.44% to 91.67%), while ablation of RL (i.e., removing GRPO group updates) leads to a pronounced drop (BtR 83.33%), establishing the complementary necessity of both GRPO-based RL updates and regularization for state-of-the-art results.

5. Trade-Offs, Scaling, and Implementation Considerations

The GRPOformer framework’s critic-free structure improves scaling and robustness under limited data, sidesteps slow or unstable value function training, and is naturally parallelizable due to its groupwise update mechanism. The Transformer policy network is updated using off-the-shelf RL infrastructure, with PCR and GRPO both regularizing learning. The design supports efficient trajectory construction without dependence on subseasonal or historical optimization data—critical for HPO benchmarks or new domains.

Key considerations for deployment include:

• Choice of group size (balances gradient variance with batch efficiency),
• Selection of PCR coefficient to trade off exploration vs. stability,
• Resource allocation between candidate evaluation and policy updates,
• Integration of task metadata for policy conditionality.

6. Implications, Generalization, and Broader Impact

The introduction of GRPO to HPO, particularly when combined with Transformer sequence models and policy churn regularization, offers a paradigm shift in automated ML pipeline optimization. The approach:

• Reduces reliance on large archives of historical trajectories by enabling rapid optimization from scratch,
• Achieves state-of-the-art performance with less supervision,
• Offers sample-efficient, adaptive optimization over complex spaces,
• Can be extended to new tasks, models, or hyperparameter spaces with minimal modifications.

This suggests broad applicability of RL-based policy learning, not just for neural architecture or parameter search but potentially to automated experiment design, empirical configuration tuning, and adaptive online learning scenarios. The empirical dominance of GRPOformer in the OpenML setting highlights its practical superiority, while the conceptual framework—critic-free policy optimization, PCR regularization, and Transformer integration—sets a foundation for future research and applications in scalable, robust HPO (Guo et al., 21 Sep 2025).

---

Summary Table: GRPO and GRPOformer Formulation

Component	Mechanism/Formulation	Comments
Advantage	$A(a_k) = r(a_k) - \text{mean}_{j=1}^K r(a_j)$	Groupwise, no critic
Objective	PPO-style clipped loss with $A(a_k)$	Stability and robust policy learning
Generator	Transformer $p(h_{t+1}|\mathcal{T}, \tau, \mathcal{H})$	Captures sequence structure
RL Regularization	PCR: $L_{\text{PC}}$ (KL to previous policy)	Dampens policy churn
Empirical SOTA	GRPOformer: BtR 94.44%, normalized performance >0.95	Across HPO tasks/datasets

---

Group-Based Relative Policy Optimization, as realized in GRPOformer, establishes a scalable, stable, and sample-efficient framework for reinforcement learning-driven hyperparameter optimization, integrating deep sequence modeling with robust, critic-free policy updates and regularization for superior real-world performance (Guo et al., 21 Sep 2025).