Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition

Abstract: Post-training via Group Relative Policy Optimization (GRPO) has emerged as a powerful paradigm for aligning flow-based generative models with human preferences. However, the iterative denoising nature of flow models incurs substantial costs when generating group rollouts for policy-gradient updates, compelling existing methods to train with extremely few denoising steps. This temporal sparsity severely restricts preference optimization: reward feedback can only reach a handful of stages per trajectory, leaving the vast majority of intermediate denoising steps without direct supervision and thus compromising alignment granularity. To address this, we propose Pave-GRPO, which reformulates the GRPO objective through Principled average velocity decomposition. Rather than generating expensive high-step rollouts, we maintain efficient few-step group sampling but decompose each coarse transition into an equivalent ensemble of finer sub-trajectories spanning multiple intermediate timesteps. This propagates reward feedback to a denser set of temporal stages for more comprehensive preference alignment without additional generation cost. This design offers two benefits: (i) zero-cost horizon expansion: through the direct reuse of piece-wise group samples and their associated rewards, Pave-GRPO significantly broadens the effective optimization scope under fixed sampling budgets; and (ii) comprehensive temporal supervision: by equivalently decomposing an instantaneous velocity target into a multi-timestep ensemble, it distributes reward signals across more intermediate stages of the denoising process, enabling finer-grained and more thorough preference optimization. Extensive experiments validate that Pave-GRPO effectively advances preference alignment across different reward settings, offering comprehensive performance enhancement.

• The paper introduces Pave-GRPO, which decomposes coarse denoising transitions into finer hybrid ODE–SDE sub-trajectories so few-step rollouts provide dense supervision without additional sampling.
• Pave-GRPO improves FLUX.1-dev alignment results, reaching a UniGenBench score of 63.96 and outperforming Flow-GRPO by 4.31 points on text rendering.
• The method adds likelihood-evaluation overhead rather than sampling overhead, with decomposition factors {2,3} offering the strongest cost–quality trade-off while broader sets show diminishing returns.

Motivation and problem statement

Group Relative Policy Optimization (GRPO) has become the dominant post-training paradigm for aligning flow-based text-to-image models with human preferences. Methods such as Flow-GRPO and DanceGRPO convert deterministic ODE sampling into stochastic SDE trajectories to supply the exploration noise that policy-gradient updates require. A structural inefficiency persists, however: because GRPO needs large rollout groups for reliable intra-group advantage estimation, and because flow models denoise iteratively, existing methods train with far fewer denoising steps than they use at inference (16 training steps versus 50 inference steps for FLUX.1-dev). The consequence is temporal sparsity — reward feedback reaches only a handful of transitions per trajectory, leaving most intermediate denoising stages without direct supervision. Pave-GRPO addresses this by decoupling the generation step budget from optimization granularity (2606.01636).

Method

The method rests on two observations. First, iterative denoising is a Markov process: any intermediate state $x_{t_{i+1}}$ is a sufficient statistic for all subsequent states, so a single reward evaluation at $x_0$ can score multiple alternative trajectories that arrive at the same anchor state. Second, both pure SDE sampling and hybrid ODE–SDE sampling yield tractable Gaussian probability paths. For marginal-preserving SDEs, pure and hybrid paths share the same Fokker–Planck evolution and hence identical marginals; for SNR-preserving SDEs, the signal-to-noise ratio depends only on diffusion coefficients at the target time, independent of path. This means the likelihood of an observed few-step sample can be evaluated exactly under finer-grained hybrid distributions without generating new samples.

Concretely, Pave-GRPO proceeds in three stages: (i) efficient group rollouts via few-step pure-SDE sampling; (ii) intra-group normalized advantage estimation from terminal rewards; and (iii) principled velocity decomposition, in which each coarse transition $x_{t_i} \to x_{t_{i+1}}$ is treated as a fixed boundary condition and decomposed into equivalent hybrid ODE–SDE sub-trajectories with granularity factors $K \in \mathcal{S}$ (e.g., $\{2,3\}$). The training objective augments the standard clipped surrogate with importance ratios $\eta_{t,k}^i(\theta)$ computed under each hybrid path, alongside the coarse-path ratio $\rho_t^i(\theta)$, both evaluated on the same target state. The single coarse velocity target is thereby split into $K$ piecewise sub-velocities, propagating reward signal to otherwise unsupervised timesteps at zero additional sampling cost.

Experimental results

All experiments use FLUX.1-dev (12B parameters), trained full-parameter on HPD prompts with 16 denoising steps and evaluated at 50 steps, against Flow-GRPO, DanceGRPO, and Flow-CPS under four reward configurations (HPS-v2, HPS-v3, and each combined with CLIP). Pave-GRPO consistently outperforms baselines on in-domain metrics while also improving out-of-domain rewards — e.g., under HPS-v2 it raises ImageReward to 1.3331 versus 1.2561 for Flow-CPS, and under HPS-v2+CLIP it attains UnifiedReward-v1 of 3.7020 versus 3.6482 for Flow-CPS. The authors interpret the out-of-domain gains as evidence against reward overfitting.

On UniGenBench (600 prompts, 10 semantic dimensions), Pave-GRPO with marginal-preserving SDE achieves the highest overall score of 63.96, exceeding Flow-GRPO by +1.31 overall, +2.03 on attribute binding, and notably +4.31 on text rendering (35.06 versus 30.75). The gains span knowledge, action, and compositional dimensions rather than purely aesthetic ones.

Ablations over the decomposition-factor set show consistent improvement over the baseline across all settings ($\mathcal{S}=\{2\}$ through $\{2,3,4,5\}$), but two caveats emerge directly from the data: per-iteration time grows roughly linearly with $x_0$0 (71s for Flow-GRPO up to 79s), and performance does not scale monotonically — $x_0$1 is selected as the best cost–quality trade-off. So while the decomposition itself adds no sampling cost, the extra hybrid likelihood evaluations do add nontrivial compute per training step.

Limitations and open questions

The paper concedes several points. The method is validated only on FLUX.1-dev for text-to-image; extension to video or other flow backbones is not demonstrated. The equivalence argument relies on exact Gaussian probability paths for the specific SDE parameterizations used; behavior under other discretizations or higher-order solvers is unexamined. The diminishing returns beyond small decomposition sets suggest an unresolved question about what limits effective supervision density — whether the bottleneck lies in the decomposition scheme itself or in the fidelity of endpoint-only reward signals propagated through shared anchors. Finally, since advantages are recycled from cheap rollouts rather than measured on high-step trajectories, any distributional mismatch between 16-step training dynamics and 50-step inference remains an implicit assumption rather than something the paper quantifies.

Conclusion

Pave-GRPO reformulates GRPO for flow models so that optimization granularity is no longer tied to the number of sampled denoising steps. By exploiting the Markov property of denoising and the shared Gaussian structure of ODE–SDE probability paths, it converts cheap few-step rollouts into dense multi-timestep supervision at negligible overhead, yielding consistent improvements over Flow-GRPO, DanceGRPO, and Flow-CPS across reward settings and achieving state-of-the-art UniGenBench performance on FLUX.1-dev.