MixGRPO: Efficient Mixed ODE-SDE Training

• MixGRPO is a training framework that partitions the denoising trajectory into a stochastic window using SDE sampling and deterministic ODE sampling outside it, balancing exploration and efficiency.
• It reduces computational cost and accelerates convergence by confining GRPO updates to a sliding window, ensuring focused exploration on high-variance denoising steps.
• MixGRPO-Flash extends the approach by deploying higher-order ODE solvers outside the window, achieving up to 71% training-time reduction with negligible performance loss.

MixGRPO is a training framework that integrates mixed stochastic-deterministic sampling within Group Relative Policy Optimization (GRPO), specifically designed to overcome optimization inefficiencies in flow matching and diffusion models for human preference alignment and reinforcement learning from human feedback (RLHF) in generative AI. In contrast to prior full-step SDE- or random-timestep sampling schemes, MixGRPO uses a novel sliding-window approach, confining stochastic SDE sampling and GRPO optimization to a short segment of the denoising trajectory while employing ODE-based deterministic sampling elsewhere. This targeted stochasticity facilitates efficient policy optimization, reduces training cost, preserves exploration, and accelerates convergence. Further, MixGRPO-Flash extends the paradigm by applying higher-order ODE solvers outside the optimization window, yielding dramatic training-time reductions with negligible alignment-quality loss (Li et al., 29 Jul 2025, Sheng et al., 12 Oct 2025).

1. Motivation and Background

Flow-based preference alignment algorithms such as FlowGRPO and DanceGRPO model the denoising (diffusion or rectified flow) process as a Markov Decision Process (MDP) spanning all denoising timesteps. These methods perform SDE-based sampling at every step, requiring computation of policy likelihood ratios for both the current and prior policy at all timesteps, resulting in high NFE (Number of Function Evaluations), slow training, and substantial optimization overhead (Li et al., 29 Jul 2025). DanceGRPO attempted to alleviate this by subsampling optimization steps, but experimental results demonstrated significant performance degradation when this subsampling was too aggressive. Therefore, there existed a critical need for a fine-grained mechanism that preserved both computational efficiency and policy-gradient learning signal.

2. MixGRPO Framework and Mathematical Formulation

MixGRPO replaces monolithic SDE or ODE sampling with a mixed-strategy sampling design, partitioning the MDP denoising trajectory into a windowed region and its complement. The window is a contiguous block of timesteps, $S = [t_1, t_2)$, typically at the earliest and noisiest portions of the trajectory. Within $S$, SDE sampling steps are executed and GRPO surrogate losses are computed. Outside of $S$, deterministic ODE sampling is used, and these steps are not subject to policy ratio evaluation or gradient updates.

Formally, the update objective over a trajectory segment $S$ is: $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$ where $s_t = x_t$ (state), $a_t$ parameterizes the velocity field, and the reward $R(x_T,c)$ is terminal, with GRPO advantage normalization performed within each group (Li et al., 29 Jul 2025, Sheng et al., 12 Oct 2025).

MixGRPO discretizes this procedure: $$x_{t+\Delta t} = \begin{cases} x_t + [f(x_t, t) - g^2(t)\nabla\log q_t(x_t)]\Delta t + g(t)\sqrt{\Delta t}\,\epsilon, & t\in S \ x_t + [f(x_t, t) - \tfrac12 g^2(t)\nabla\log q_t(x_t)]\Delta t, & t\notin S \end{cases}$$ where $f$ and $S$0 are drift and diffusion coefficients, and $S$1. In practice, $S$2 is replaced by the learned velocity $S$3.

Group-relative advantages are computed as $S$4 over group samples, and the surrogate PPO-style objective at each time step $S$5 is

$S$6

with importance ratio $S$7 (Li et al., 29 Jul 2025).

The window $S$8 slides forward as training progresses, concentrating exploration and optimization initially on high-variance, high-difficulty denoising steps and gradually shifting toward cleaner portions of the trajectory—remotely analogous to discounting schedules in RL (Li et al., 29 Jul 2025, Sheng et al., 12 Oct 2025).

3. Algorithms and Sliding-Window Procedure

MixGRPO implements a deterministic schedule for sliding the SDE window, ensuring that all portions of the denoising chain are eventually subject to exploration and policy update. The algorithm iterates as follows:

1. Initialize $S$9, window $S$0 of length $S$1, stride $S$2, shift interval $S$3.
2. At each iteration:
   • Set $S$4.
   • For each prompt $S$5 and sample $S$6:
     • Sample $S$7.
     • For $S$8:
       • If $S$9, run SDE step; else run ODE step under $S$0.
     • Compute $S$1 and $S$2.
   • For $S$3, compute policy gradient and update $S$4.
   • Every $S$5 iterations, shift window $S$6.

MixGRPO-Flash extends this by switching to higher-order ODE solvers (e.g., 2nd-order DPM-Solver++) for $S$7, and allows for a "frozen" window variant (Flash*) by keeping $S$8 throughout. This achieves significant reductions in NFE for $S$9, with training-time reductions up to 71% (Li et al., 29 Jul 2025).

4. Empirical Results and Ablation Analyses

Experiments were performed on the HPDv2 image preference dataset using the FLUX.1 model, with evaluations on HPS-v2.1, Pick Score, ImageReward, and Unified Reward. Main findings include:

• MixGRPO achieved HPS-v2.1 = 0.367 and ImageReward = 1.629, outperforming DanceGRPO (0.334 and 1.335, respectively) while halving iteration time (151 s vs. 292 s).
• MixGRPO-Flash reached HPS ≈ 0.358 and ImageReward = 1.624 at 112 s and 83 s per iteration, representing 62–71% speedup with negligible performance loss (Li et al., 29 Jul 2025).
• Optimal window length was $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$0 for $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$1 timesteps; shift interval $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$2 and stride $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$3 were robust. Too small $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$4 reduced exploration and final alignment, while large windows increased computational cost.
• 2nd-order midpoint or DPM-Solver++ gave the best sample quality and reward alignment outside the window; 3rd-order solvers offered marginal gains at greater expense.
• Visualization via t-SNE confirmed that early windowed SDE sampling enhances trajectory diversity, supporting the window design rationale.

5. Theoretical Guarantees and Convergence

While MixGRPO does not introduce a unique convergence proof, (Sheng et al., 12 Oct 2025) establishes tight bounds on the reward gap between SDE (used for exploration in training) and ODE (used for efficient inference) sampling for classically dissipative diffusion processes. Specifically, for C-Lipschitz rewards, the mean reward difference between SDE and ODE sampling is bounded by a function of the window's stochasticity parameter, dissipativity, and the noise coefficient—yielding $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$5 or $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$6 decay in standard Gaussian cases. This provides formal justification for the observed empirical result that reward gaps between training (mixed) and inference (ODE) sampling diminish as models converge (Sheng et al., 12 Oct 2025).

6. Design Insights, Trade-offs, and Practical Guidelines

MixGRPO's core insight is that mixing ODE and SDE steps—rather than using one globally—specifically targets the trade-offs between controlled exploration and efficient convergence. Key design principles:

• Restricting randomness to a small sliding window ensures efficient gradient estimation, as most policy ratios are evaluated only within $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$7.
• Deterministic ODE regions can leverage advanced solvers, accelerating $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$8 trajectory generation.
• Window scheduling can be tuned for task hardness, RL analogy, and early-stage exploration.

Recommended settings are $$\max_\theta \; \mathbb{E}_{\Gamma_S\sim\pi_\theta}\bigg[\sum_{t\in S}\Big(R(x_T, c) - \beta\, D_{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{ref}}(\cdot|s_t))\Big)\bigg],$$9, $s_t = x_t$0, stride $s_t = x_t$1, shift interval $s_t = x_t$2. For maximal acceleration with minor performance loss, use MixGRPO-Flash with a frozen early window and compression ratio $s_t = x_t$3 for the ODE region (Li et al., 29 Jul 2025). If the window is too small or ODE compression too aggressive, under-exploration can degrade the final policy; too large or pure-SDE approaches negate MixGRPO's efficiency benefits.

7. Relation to Broader GRPO Variants: MixGRPO as Normalization Interpolator

MixGRPO can also refer, outside the diffusion context, to a more general normalization scheme for group-relative policy optimization in RL for reasoning and symbolic tasks (Bay et al., 30 Jun 2026). Here, it denotes a parameterized interpolation among GRPO (standard deviation normalization), Dr.GRPO (no normalization), and DAPO (skipping zero-variance groups), controlled by a mixing function $s_t = x_t$4 based on the empirical standard deviation $s_t = x_t$5 of group rewards: $s_t = x_t$6 where $s_t = x_t$7 if $s_t = x_t$8 (skip group), and $s_t = x_t$9 transitions linearly from 0 (Dr.GRPO) to 1 (GRPO) between thresholds $a_t$0. This adaptive rule avoids both overamplifying noise in near-unanimous groups and underweighting hard cases, ensuring silent (zero-variance) prompts do not waste updates and high-variance prompts receive appropriately scaled gradients (Bay et al., 30 Jun 2026).

Table: MixGRPO Sliding-Window Hyperparameter Range (Image Diffusion RLHF context)

Parameter	Typical Value	Role
Window size $a_t$1	4	Number of SDE steps in window
Total steps $a_t$2	25	Denoising trajectory length
Shift interval $a_t$3	25	Iterations between window moves
Stride $a_t$4	1	Steps window advances per shift
ODE solver	2nd-order DPM++	Deterministic sampling outside $a_t$5

These settings are empirically tuned for the HPDv2 human preference alignment task. For more diverse tasks or architectures, adaptation may be warranted; however, the principle of confining stochasticity and gradient computation to a small, strategically-placed region persists across use cases.

• MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE (Li et al., 29 Jul 2025)
• Understanding Sampler Stochasticity in Training Diffusion Models for RLHF (Sheng et al., 12 Oct 2025)
• GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity (Bay et al., 30 Jun 2026)