--- title: 'iGRPO: Iterative Group Relative Policy Optimization' url: https://www.emergentmind.com/topics/iterative-group-relative-policy-optimization-igrpo type: topic --- # iGRPO: Iterative Group Relative Policy Optimization Iterative Group Relative Policy Optimization (iGRPO) is a reinforcement learning framework developed for post-training alignment of large language models (LLMs) using verifiable, group-normalized rewards. iGRPO generalizes and addresses key biases in Group Relative Policy Optimization (GRPO), providing both theoretical guarantees and enhanced empirical performance across mathematical reasoning and other structured tasks. The method combines group-based advantage estimation with iterative updates or self-feedback-driven conditioning, supporting both on- and off-policy training regimes and yielding robust and scalable improvements without reliance on a learned value function. ## 1. Mathematical Foundations of GRPO and iGRPO At its foundation, GRPO operates on sampled groups of completions per prompt, using their rewards to derive centered or standardized advantages for each sample. Let $q$ denote the prompt and $\{o_i\}_{i=1}^G$ the group of $G$ candidate responses. The group-centered (mean-removed) advantage is: \[ A_i = r_i - \frac{1}{G}\sum_{j=1}^G r_j \] or, in standardized form with groupwise mean $\mu$ and std $\sigma$, \[ \hat A_i = \frac{r_i - \mu}{\max(\sigma, \delta)} \] where $\delta > 0$ prevents division by zero [2602.09000]. The general surrogate objective for GRPO-style methods is formulated as: \[ \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q, \{o_i\}} \biggl[ \sum_{i=1}^G \sum_{t=1}^{|o_i|} \alpha_{i,t} \min( s_{i,t}(\theta)A_i,\ \mathrm{clip}(s_{i,t}(\theta), 1-\varepsilon_\text{low}, 1+\varepsilon_\text{up})A_i ) - \beta R(\theta) \biggr] \] with $\alpha_{i,t}$ denoting per-token weights, $s_{i,t}(\theta)$ the importance ratio between new and old policy, and $R(\theta)$ typically a KL-regularization penalty [2601.05002]. iGRPO builds on this by (a) replacing the value function with a group-based advantage as above, (b) supporting iterative application, and (c) extending to two-stage or off-policy scenarios [2505.22257, 2503.06639, 2602.09000]. ## 2. Theoretical Properties and Bias Corrections GRPO exhibits several structural objective mismatches: - Non-uniform group weighting, via non-constant $\alpha_{i,t}$, introduces systematic gradient bias on prefixes shared among group sequences. For length-normalized weights ($\omega_{i,t} \propto 1/|o_i|$), shorter sequences can disproportionately influence shared prefix updates, causing a form of structural length bias independent of reward structure [2601.05002]. - The surrogate objective’s token-level gradient in the unclipped region is: \[ \nabla_\theta \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \sum_{i=1}^G A_i \sum_{t=1}^{|o_i|} \alpha_{i,t} s_{i,t}(\theta) \nabla_\theta \log \pi_\theta(y_{i,t} | x, y_{i, 0$) [2601.05002]. ## 3. Iterative and Two-Stage iGRPO Algorithms ### Standard Iterative iGRPO The canonical iGRPO loop is as follows [2505.22257, 2503.06639]: 1. Sample a batch of prompts $q$. 2. For each $q$, sample $G$ outputs $\{o_j\}$ from the current or lagged policy. 3. Compute group-based mean and std of the rewards; normalize each $o_j$’s advantage. 4. Construct the clipped, KL-regularized surrogate loss using the group-based advantage. 5. Update the policy via one or more gradient steps; iterate as needed, with options for on-policy ($v=1$) or off-policy ($v>1$) sample reuse. 6. Repeat steps 1-5, plugging the new policy back as the "old" in the next round. ### Two-Stage (Self-Feedback) iGRPO A major variant is the self-feedback-driven, two-stage iGRPO [2602.09000]: - **Stage 1 (Exploration):** For each prompt $q$, sample $N$ drafts from the frozen policy; pick the highest-reward draft $\hat d$. - **Stage 2 (Refinement):** Augment $q$ with $\hat d$ as an in-context example ($q' = \mathrm{Concat}(q, \hat d)$), then sample $G$ completions from $q'$, normalize as above, and apply a GRPO-style update on these samples. - Only Stage 2 gradients are used for learning; Stage 1 influences learning indirectly via the structure and challenge of $q'$. This architecture can be summarized by the following pseudo-algorithm: ```python for training_step in range(num_steps): for prompt q in batch: # Stage 1: explore drafts = [sample(pi_theta_old, q) for _ in range(N)] d_hat = argmax_{d in drafts} R_phi(d) # Stage 2: refine q_prime = concat(q, d_hat) outputs = [sample(pi_theta_old, q_prime) for _ in range(G)] advantages = group_normalize([R_phi(o) for o in outputs]) # update policy on outputs w.r.t. q_prime update_theta(grad of GRPO objective) ``` This dynamic introduces a bootstrapped, policy-coupled feedback that empirically delays entropy collapse and improves exploration [2602.09000]. ## 4. Policy-Improvement Guarantees and Convergence For both on- and off-policy iGRPO, theoretical lower bounds on expected reward improvement can be established [2505.22257]: \[ J(\pi|x) - J(\pi_k|x) \ge L_\alpha(\pi|x) - 2 \frac{1 - \sigma_{\alpha,r,\varepsilon}(x)}{\sigma_{\alpha,r,\varepsilon}(x)} TV(\pi(\cdot|x), \alpha(\cdot|x)) - 2 TV(\pi_k(\cdot|x), \alpha(\cdot|x)) \] where $L_\alpha(\pi|x)$ denotes the value of the clipped, whitened surrogate, and $TV$ is total variation. For binary verifiable rewards, the iGRPO recursion admits a closed-form in terms of the distribution’s empirical success probability $p_n$, weights $\omega^\pm_\varepsilon(p_{n-1})$, and KL-regularization $\beta$: \[ \pi_n(o|q) \propto \pi_{\rm ref}(o|q) \exp\left\{ \frac{1}{\beta} [\omega^+_\varepsilon(p_{n-1}) \mathbf{1}_{r=1} - \omega^-_\varepsilon(p_{n-1}) \mathbf{1}_{r=0}] \right\} \] This induces a scalar map $p_n = h_{\varepsilon,p_{\mathrm{ref}}}(p_{n-1})$ whose unique fixed point $p^* > p_{\mathrm{ref}}$ amplifies the probability of success [2503.06639]. Local contraction and monotonic convergence to $p^*$ are guaranteed for appropriate $\beta$. ## 5. Empirical Performance and Practical Configurations Empirical studies on math and reasoning tasks demonstrate that iGRPO, in both vanilla and two-stage forms, reliably matches or outperforms single-step GRPO and self-verification baselines under matched rollout budgets [2505.22257, 2602.09000]. Key findings include: - On GSM8K, math challenge benchmarks, and DeepScaleR-Preview, iGRPO is consistently as stable or more stable than on-policy GRPO (e.g., Pass@1: on-policy, 45% ± 3%; iGRPO, 50% ± 1%) [2505.22257]. - In large-scale settings (Nemotron-8B, DeepSeek-7B, OpenMath-7B/14B), iGRPO delivers a +1–4 point gain over GRPO with the same or reduced serve-side inference cost, as the staged setup induces better learning signals under group-based normalization [2602.09000]. - Ablations confirm that the two-stage wrapper is optimizer-agnostic and that the use of a generative judge (GPT-5) brings further improvements. Entropy collapse occurs more slowly, preserving exploration [2602.09000]. A summary of key hyperparameters and their rationale: | Parameter | Default/Range | Significance | |---------------------|----------------------|--------------------------------------------------------------| | $N$, $G$ | $N+G=G_{\rm base}$ | Budget-split between drafts and refinements. | | Learning rate | $1\times 10^{-6}$ | Empirically validated. | | KL penalty ($\beta$)| $0$ or small $>0$ | No regularization if value-insensitive; otherwise tune. | | Decoding temperature/p | $T \sim 0.6$, $p \sim 0.95$ | Supports moderate exploration. | ## 6. Algorithmic Recommendations and Practical Caveats To address the documented hidden biases and inefficiencies in GRPO, effective iGRPO design incorporates the following recommendations [2601.05002]: - **Unbiased group weighting:** Enforce $\sum_{i\in G}\alpha_{i,t}=0$ when weighted by the centered advantage, especially for shared prefixes. - **Reward scaling and optimizer configuration:** If using AdamW without KL regularization, reward scaling becomes irrelevant for parameter updates due to moment cancellation. When regularization is required ($\beta>0$), tune $\beta$ jointly with the reward scale as both impact update magnitude. - **Momentum overshoot control:** Use single-step updates (no inner-loop SGD), or apply moment resets/repositioning for AdamW if more steps are needed. - **Metrics:** Monitor held-out reward distributions, not the surrogate objective, for true progress. A practice-supported implication is that, for large language models using group-based objectives, subtle choices in weighting, batching, and normalization are critical for unbiased, efficient policy improvement; iGRPO methods explicitly address these angles. ## 7. Broader Impact and Generality iGRPO frameworks have been adopted for LLM post-training in math reasoning, code synthesis, and other domains with verifiable or scalar rewards [2602.09000, 2505.22257]. The group-relative, critic-free approach provides a modular, scalable alternative to value-based RL, with theoretical guarantees and extensibility to both on-policy and off-policy regimes. The two-stage iGRPO wrapper generalizes beyond GRPO surrogates and can be fruitfully combined with diverse RL and reward modeling strategies. Key empirical and theoretical results demonstrate that iGRPO - Offers consistent multi-point improvements on competitive reasoning datasets. - Incurs negligible additional computation or inference budget compared to GRPO. - Enables self-feedback-driven learning dynamics, supporting delayed mode-collapse and improved sampling efficiency. - Amplifies success probability with guaranteed convergence under mild regularization settings, for both binary and scalar rewards. By correctly applying group normalization, reward shaping, optimizer control, and iterative bootstrapped learning, iGRPO sets a new standard for scalable, verifiable reward-driven LLM alignment [2601.05002, 2602.09000, 2505.22257, 2503.06639]. Source: https://www.emergentmind.com/topics/iterative-group-relative-policy-optimization-igrpo