Progressive Group Relative Policy Optimization (MAPGRPO)

• MAPGRPO is a reinforcement learning framework for fine-tuning generative models via staged, multi-scale optimization.
• It incrementally optimizes coarse, mid-level, and fine-scale features to enhance stability, sample diversity, and aesthetic alignment.
• The approach mitigates reward hacking and improves convergence speed by leveraging group-relative advantages and progressive parameter freezing.

Progressive Group Relative Policy Optimization (MAPGRPO) is a reinforcement learning (RL) framework designed for fine-tuning generative models, particularly multi-scale visual autoregressive (VAR) models, by aligning generation outputs with complex reward signals such as aesthetic predictors or CLIP-based metrics. MAPGRPO extends the Group Relative Policy Optimization (GRPO) technique to explicitly exploit the multi-scale, coarse-to-fine generation structure of contemporary generative architectures. Through a staged optimization process, MAPGRPO incrementally aligns the global, mid-level, and fine-scale content of generated samples, yielding enhanced stability, computational efficiency, and sample diversity relative to single-stage or non-progressive alternatives (Gallici et al., 29 May 2025).

1. Background: Group Relative Policy Optimization and Its Limitations

Group Relative Policy Optimization (GRPO) utilizes group-based sampling to stabilize policy gradients and eliminate the need for a learned value function. Given a set of $G$ sampled outputs per context (e.g., prompt or class label), GRPO computes within-group, standardized advantages to mitigate prompt-level reward bias, then updates the policy via a clipped surrogate loss with KL-penalty against a reference model. Empirical studies have demonstrated that, for both language and vision tasks, GRPO improves generalization, sample diversity, and robustness compared to vanilla PPO or supervised fine-tuning (Simoni et al., 3 Jul 2025). However, the efficacy of GRPO in high-dimensional, multi-scale environments is often bottlenecked by poor credit assignment and computational costs associated with large group sizes or all-scale updates, especially in deep autoregressive architectures.

2. MAPGRPO Formalism and Progressive Multi-Scale Scheduling

MAPGRPO (Multi-scale Adaptive Progressive GRPO) extends GRPO by leveraging the inherent multi-layered latent representation of next-scale VAR models. Rather than performing RL updates across all latent scales jointly, MAPGRPO proceeds in $K$ progressive stages, each corresponding to a generative subpolicy for a particular latent image scale:

• Stage $k$: Parameters for scales $k+1$ to $K$ are frozen. Only the parameters for scales $1$ to $k$ are updated via GRPO.
• Transition: After achieving convergence or completing a predetermined update budget at stage $k$, scale $k+1$ is unfrozen and subject to training in the subsequent stage.

This staged freezing/unfreezing procedure ensures that the model first optimizes global structural features (coarse scales), then incrementally incorporates detail, thereby stabilizing learning and mitigating issues such as mode collapse at the finest scales.

The formal objective optimized at stage $k$ is:

$$J_k(\theta) = \sum_{\ell=1}^k \mathbb{E}_{g,i}\left[\min(\rho_{g,i}^\ell A_\ell^g,\; \text{clip}(\rho_{g,i}^\ell,1-\epsilon,1+\epsilon)\,A_\ell^g)\right] - \beta \sum_{\ell=1}^k \mathbb{E}_{g,i}[D_{\mathrm{KL}}(\pi_\theta(\cdot | s_{g,i}^\ell) \| \pi_{\mathrm{ref}}(\cdot | s_{g,i}^\ell))]$$

where $A_\ell^g$ are standardized, group-relative advantages computed per scale and group, and $\rho_{g,i}^\ell$ are the policy ratios for PPO-style clipping.

3. Algorithmic Structure and Implementation

The MAPGRPO algorithm is structured around progressive, scale-wise repeated GRPO updates. For each stage:

1. Freeze parameters above current scale.
2. Sample B class labels; for each, sample $G$ sets of multi-scale tokens up to scale $k$ with temperature-controlled multinomial sampling.
3. Decode final images and compute scalar rewards using the desired predictor (e.g., aesthetic or CLIP score).
4. Compute group means and standard deviations to derive group-relative advantages for all active scales.
5. Optimize the clipped surrogate objective using only tokens up to scale $k$ and apply KL regularization.
6. Repeat for a fixed number of stage updates; advance to the next stage by unfreezing one additional scale.

This structure is compatible with the discrete multi-scale latent spaces of modern VAR models (e.g., VAR-d16, VAR-d30), which employ VQ-VAE decoders and class-conditional inputs. The staged approach also leverages sub-linear complexity in token generation costs, as each scale only incurs sampling and computational overhead once it is unfrozen.

4. Experimental Evaluation and Empirical Findings

MAPGRPO was evaluated on fine-tuning pre-trained VAR models for 256×256 image generation using three latent scales (coarse=64, mid=128, fine=256). Core empirical findings include:

• Aesthetic Fine-Tuning: For VAR-d30 (2B params), MAPGRPO increased the aesthetic predictor (AES) from 4.80 (pre-trained) to ≈5.80, with a modest decrease in ImageNet-top-5 accuracy (99.8%→90%).
• CLIP-Prompt Alignment: Sustained increase in CLIP-score through RL. Qualitative analysis showed that lighting/composition improved at coarse scales, while color/texture refinement occurred at fine scales.
• Ablation Results: Progressive scheduling yielded 15–20% faster convergence and fewer reward-hacks (e.g., vignetting as CLIP exploit) compared to single-stage GRPO. Optimal KL penalty $\beta$ was found to be 0.2. Performance improved monotonically up to group size $G=16$, after which computational costs offset further gains.

A summary table of hyperparameters and settings is provided:

Parameter	Value(s) / Setting	Description
Model	VAR-d16 (310M), VAR-d30 (2B)	Underlying visual autoregressive
Scales	$K=3$	Coarse (64), Mid (128), Fine (256)
Group size ($G$)	16	Per-label group in sampling
PPO clip ($\varepsilon$)	0.2	Clipping threshold
KL penalty ($\beta$)	0.2	Per-token regularization
Temperature ($\tau$)	0.7	Sampling temperature
Stage updates	10k, 10k, 20k (by scale)	Updates per progressive stage

5. Theoretical Justification and Credit Assignment

MAPGRPO’s technique of sharing the same standardized, group-relative advantage across all tokens of a group at each scale addresses the delayed reward problem inherent in multi-scale generative models, where a single scalar reward is computed only after final decoding. This design precludes the need for a learned value function or rollout-based credit assignment, reducing potential instability.

Progressive scheduling is theoretically motivated by the observation that aligning global semantics first constrains subsequent local refinements, thus preventing “reward hacks” that would otherwise be incentivized if all scales were trained jointly from the outset. A plausible implication is that this decoupling addresses scale interference, although excessively short or imbalanced stage durations can still allow finer-scale gradients to override coarse alignment.

6. Limitations and Future Prospects

MAPGRPO’s current design applies uniform advantages within each group and stage, an approximation that may be suboptimal for architectures with significant intra-group or intra-scale heterogeneity. More granular attention-based credit assignment is suggested as a future improvement. Additionally, scale interference and optimal stage-length scheduling remain open research questions, as does adaptation to prompt-conditioned (rather than fixed class label) VAR and the use of human feedback-based reward models.

Extending MAPGRPO to other multi-scale, non-autoregressive, or non-visual domains would require substantial modification to its credit assignment and scheduling heuristics. Further evaluation in RLHF settings or with dynamic, learned scale-weighting could yield models with improved alignment to human preferences without sacrificing structural consistency or sample diversity.

7. Summary and Impact

MAPGRPO introduces and rigorously implements a progressive, scale-wise extension of GRPO optimized for multi-scale visual autoregressive generation (Gallici et al., 29 May 2025). By first aligning high-level global structure and then progressively refining detail through staged training, MAPGRPO achieves stronger semantic and aesthetic alignment, improved convergence rates, and greater robustness to reward hacking artifacts compared to non-progressive or single-stage approaches. This approach critically leverages both the architectural and computational properties of VAR and similar architectures, setting a precedent for multi-stage RL optimization in generative model alignment.