---
title: 'DanceGRPO: Unified Visual RL Framework'
url: https://www.emergentmind.com/topics/dancegrpo
type: topic
---

# DanceGRPO: Unified Visual RL Framework

DanceGRPO is a unified reinforcement learning framework that adapts Group Relative Policy Optimization (GRPO) to modern visual generative paradigms, including diffusion models and rectified flows. The approach generalizes across three core task types—text-to-image, text-to-video, and image-to-video—by framing the multi-step denoising process as a Markov Decision Process (MDP) suitable for RL from human feedback (RLHF). DanceGRPO introduces algorithmic and implementation-level unification over diverse generative models, tasks, and reward models, establishing a robust and scalable methodology for aligning generative outputs with complex, multi-criteria user preferences [2505.07818].

## 1. Mathematical Foundations and GRPO Objective

The denoising process underpinning both diffusion and flow-based models is formalized as an MDP, where states \(s_t=(c, t, z_t)\) encode conditioning information, temporal context, and sample latents; actions \(a_t=z_{t-1}\) correspond to generative steps; and the policy \(\pi_\theta(a_t|s_t)\) is realized as a reverse SDE/ODE sampler. DanceGRPO restricts reward assignment exclusively to terminal time, with \(R(s_0,a_0)=r(z_0,c)\), and propagates a single advantage signal across all timesteps within a trajectory.

The GRPO surrogate objective extends PPO through group-relative normalization, stabilizing large-scale RLHF for visual generation:

\[
\mathcal{J}(\theta) = \mathbb{E}_{\{\tau_i\}_{i=1}^G \sim \pi_{\theta_{\rm old}}} 
\Bigg[
\frac{1}{G}\sum_{i=1}^G \frac{1}{T}\sum_{t=1}^T
\min\Bigl(
\rho_{t,i}A_i,\;\mathrm{clip}(\rho_{t,i},1-\epsilon,1+\epsilon)A_i
\Bigr)
\Bigg]
\]

with key components as follows:

- \(\rho_{t,i}=\frac{\pi_\theta(a_{t,i}|s_{t,i})}{\pi_{\theta_{\rm old}}(a_{t,i}|s_{t,i})}\)
- \(A_i=\frac{r_i-\mu_r}{\sigma_r}\), where \(\mu_r=\tfrac{1}{G}\sum_j r_j\), \(\sigma_r=\mathrm{Std}(\{r_j\})\)
- Each \(r_i = r(z_0^{(i)},c)\) may itself be a sum of \(K\) individual reward components

First-order expansion reveals a group-normalized policy gradient, regularized by clipping to control KL divergence.


## 2. Unified Algorithmic Workflow

DanceGRPO's implementation is modular, adapting to both diffusion and rectified-flow sampling through a single pseudocode template. For each prompt, the workflow proceeds as follows:

1. Sample a batch of prompts.
2. For each prompt:
   - Sample \(G\) denoising trajectories sharing an initial noise seed.
   - For diffusion: reverse SDE sampling (\(\mathrm{d}z_t=(f_tz_t-\tfrac{1+\eta_t^2}{2}g_t^2\nabla\log p_t(z_t))\,\mathrm{d}t + \eta_tg_t\,\mathrm{d}w\)).
   - For rectified flow: inject noise into ODE (\(\mathrm{d}z_t=(u_t - \tfrac{1}{2}\varepsilon_t^2\nabla\log p_t(z_t))\,\mathrm{d}t + \varepsilon_t\,\mathrm{d}w\)).
   - Compute \(K\) reward signals per trajectory.
   - Normalize and sum advantages across each reward dimension:
      \[
      A_i = \sum_{k=1}^K \frac{r_i^k - \mu^k}{\sigma^k}
      \]
   - Randomly subsample timesteps (\(\tau T\)).
   - Accumulate gradients and update parameters.

This process is repeated for all iterations, yielding robust policy updates even in the presence of high-dimensional, multi-reward feedback and large group sizes.


## 3. Task Adaptation and Foundation Models

DanceGRPO supports three main generative tasks, uniformly handling conditioning and reward fusion:

- **Text-to-Image:** Utilizes Stable Diffusion v1.4, FLUX (flow-based), and HunyuanVideo-T2I.
- **Text-to-Video:** Employs HunyuanVideo.
- **Image-to-Video:** Relies on SkyReels-I2V with image-based conditioning.

Distinct adaptations per task include: employing VideoAlign multi-dimensional reward for videos; restricting motion-quality optimization for image-to-video (where fidelity is determined by the input image); and alternating conditional/unconditional samples for classifier-free guidance models. All tasks inject prompt conditioning \(c\) directly into the sampler state.


## 4. Reward Model Integration and Normalization

DanceGRPO leverages up to five reward functions simultaneously, synthesizing them by summing group-normalized advantages rather than raw scores:

- **Image Aesthetics:** HPS-v2.1
- **Text–Image Alignment:** CLIP score
- **Video Aesthetics and Motion Quality:** VideoAlign metrics
- **Binary Threshold Rewards:** e.g., \(r = 1\) if HPS > 0.28, else 0

Group-wise normalization avoids reward scale mismatches:

\[
A_i = \sum_{k=1}^K \frac{r_i^k - \mu^k}{\sigma^k}
\]

This approach stabilizes training and supports efficient handling of sparse binary feedback, maintaining informative gradients by ensuring nonzero variance in group statistics.


## 5. Experimental Protocol and Benchmark Results

Extensive empirical validation demonstrates DanceGRPO's scaling and robustness:

- **Hardware:** 8–64 NVIDIA H800 GPUs
- **Datasets:** >10k prompts for training, 1k for each evaluation split, 200–240 for human evaluation
- **Benchmarks:** HPS-v2.1, CLIP score, Pick-a-Pic, GenEval, VideoAlign (multi-dim)

Representative results:

| Task                         | Baseline → DanceGRPO | % Improvement | Metric         |
|------------------------------|:--------------------:|:-------------:|:--------------:|
| T2I (Stable Diffusion)       | HPS: 0.239 → 0.365   | +53%          | HPS-v2.1       |
| T2I (Stable Diffusion)       | CLIP: 0.363 → 0.395  | +9%           | CLIP Score     |
| T2I (FLUX)                   | HPS: 0.304 → 0.372   | +22%          | HPS-v2.1       |
| T2I (FLUX)                   | CLIP: 0.405 → 0.427  | +5%           | CLIP Score     |
| T2V (HunyuanVideo)           | Motion: 1.37 → 3.85  | +181%         | VideoAlign     |
| I2V (SkyReels-I2V)           | Motion Q: baseline→+91%| +91%        | --             |

Human evaluation preferred RLHF-tuned outputs ~70% of the time across major task/model combinations.

Stability plots confirm that DanceGRPO maintains smooth reward curves even for video models and sparse binary signals, with competing methods (DDPO) diverging on rectified flows. Best-of-N inference scaling with top/bottom-k sample selection accelerates convergence.


## 6. Practical Insights and Implementation Considerations

Key observations:

- **Stability in RLHF:** Sharing initial noise seeds within prompt-groups deters reward hacking; group-normalized advantages mitigate outlier gradients; subsampling timesteps (\(\tau \approx 0.6\)) reduces compute and variance.
- **Best-of-N Scaling:** Focusing updates on informative extremes of the sample pool speeds training but requires additional sampling. This procedure operates orthogonally to the core algorithm.
- **Sparse Binary Feedback:** The normalization mechanism retains gradient informativeness even with strict thresholding, supporting ascent on sparse reward landscapes.
- **ODE-based Sampler Compatibility:** Injecting a calibrated SDE noise term into ODE-based generative samplers restores Markovian stochasticity for RL; this permits direct adaptation of GRPO without restructuring samplers into discrete-step MDPs.

DanceGRPO achieves a single policy-gradient framework for visual generation, harmonizing RLHF objectives with advanced denoising architectures. The methodology is notable for its empirical performance, mathematical rigor, and practical extensibility across contemporary generative pipelines [2505.07818].

Source: https://www.emergentmind.com/topics/dancegrpo