RealDPO: Real-data Driven Video Alignment

• RealDPO is an alignment paradigm that employs real-world video data to guide text-to-video models in synthesizing natural human motions.
• It contrasts real (‘win’) samples with model-generated (‘lose’) outputs to drive self-correction and ensure smoother, contextually coherent motion.
• Empirical evaluations demonstrate significant improvements in motion realism, temporal consistency, and text-video alignment compared to traditional fine-tuning methods.

RealDPO is an alignment paradigm for video generative modeling that directly leverages real-world video data as the preferred reference in a direct preference optimization framework. It is designed to improve the synthesis of complex, naturalistic human motions in text-to-video systems, especially where traditional supervised fine-tuning or reward-model-based approaches have failed to yield satisfactory motion consistency, smoothness, or context alignment. RealDPO uses an explicit comparison between real data and erroneous model generations, enabling iterative self-correction and stable post-training of large diffusion-transformer video models (Cheng et al., 16 Oct 2025).

1. Motivation and Problem Setting

Video diffusion and transformer-based generative models have shown substantial improvements in synthesis quality, but remain fundamentally limited in producing realistic, contextually coherent human motions. Pretrained large models such as CogVideoX-5B often suffer from artifacts (e.g., implausible limb trajectories, timing inconsistencies, context mismatches) even after supervised fine-tuning (SFT) on limited human-motion video corpora. The SFT paradigm, which provides only positive real-world samples, lacks explicit corrective feedback about model-generated errors and thus tends toward overfitting and limited motion improvement (Cheng et al., 16 Oct 2025).

Reward-model-based reinforcement learning strategies (e.g., PPO-like RLHF or synthetic DPO using paired model outputs) increase computational cost and introduce reward hacking and multi-objective imbalance, yet still do not anchor motion realism to ground-truth distributions.

RealDPO addresses this by using real data for the "win" (preferred) samples and model generations for "lose" (dispreferred) samples, casting motion alignment as a preference optimization problem grounded in veridical motion data.

2. Methodological Foundations

2.1 Direct Preference Optimization (DPO) in RealDPO

DPO is a preference-alignment technique that operates over (input, win, lose) triples—here, (prompt, real video, model sample)—without an explicit reward model. RealDPO maximizes the log-likelihood ratio of the preferred versus dispreferred outputs under a KL-regularized fine-tuning regime that anchors the model close to its pretrained reference (Cheng et al., 16 Oct 2025).

For diffusion-based transformers, let

• $x_0^{\text{w}}$ : clean latent of a real (“win”) video
• $x_0^{\ell}$ : clean latent of a model-generated (“lose”) video
• $\hat{x}_0^{\text{w}}, \hat{x}_0^{\ell}$ : denoised reconstructions by the fine-tuned model
• $\bar{x}_0^{\text{w}}, \bar{x}_0^{\ell}$ : reconstructions by the frozen reference model
• $\beta$ : temperature parameter
• $T$ : total diffusion steps
• $\omega(\lambda_t)$ : (optional) timestep weighting

The RealDPO loss is:

$$L_{\text{RealDPO}}(\theta) = -\mathbb{E}_{x_0^{\text{w}}, x_0^{\ell}, t}\left[\log \sigma\left(-\beta T \omega(\lambda_t) \left(\lVert x_0^{\text{w}} - \hat{x}_0^{\text{w}} \rVert_2^2 - \lVert x_0^{\text{w}} - \bar{x}_0^{\text{w}}\rVert_2^2 - \left[\lVert x_0^{\ell} - \hat{x}_0^{\ell}\rVert_2^2 - \lVert x_0^{\ell} - \bar{x}_0^{\ell}\rVert_2^2\right]\right)\right)\right]$$

This contrastive term penalizes the fine-tuned model’s reconstruction error on the real sample (pulled closer than the reference) and pushes its error on the negative (model-generated) sample further away, all in a single update step.

2.2 Positive and Negative Sample Construction

• Positives ("win"): Video clips from the RealAction-5K dataset (curated real videos, see Section 3).
• Negatives ("lose"): Clips generated by the pretrained model using the same prompt but different random seeds (3 candidate negatives per prompt).
• Training: Both samples undergo noise injection via $t$ diffusion steps; both the fine-tuned model and the frozen reference attempt to denoise and reconstruct, with losses computed as above (Cheng et al., 16 Oct 2025).

2.3 Iterative Self-Correction

Throughout post-training, the reference model parameters $\theta_{\text{ref}}$ are updated by exponential moving average ($x_0^{\ell}$0) to track the improving fine-tuned model, enabling the reference to gradually reflect progress without introducing instability. Periodically, negative samples are regenerated by sampling the improving model to refresh error modes and drive further correction.

3. The RealAction-5K Dataset

RealAction-5K is a curated dataset supporting the RealDPO paradigm, comprising 5,000 video clips (3–5 seconds, 24–30 fps, 256×256 to 512×512) across ≈10 categories of daily human activities (e.g., walking, eating, sports). Clips are collected via search/filtering on public video platforms (e.g., Pexels), automatically screened by a video-LLM (Qwen2-VL), and human-verified for action correctness and clarity.

Each clip receives a caption generated by LLaVA-Video, yielding fine-grained text descriptions of actions, objects, and settings, which supply text-prompt/video pairs for preference optimization (Cheng et al., 16 Oct 2025).

4. Training Protocol and Implementation

4.1 Training Workflow

1. Precompute and store latents of RealAction-5K and paired negatives.
2. Initialize reference model $x_0^{\ell}$1 as the pretrained checkpoint.
3. Interleave epochs of preference learning via the RealDPO loss (minibatch of real/negative pairs, random diffusion timesteps, AdamW optimizer, lr=$x_0^{\ell}$2, batch size 8), with periodic EMA update of $x_0^{\ell}$3.
4. Repeat for 10 epochs; save $x_0^{\ell}$4 for inference.

4.2 Key Hyperparameters

• Batch size: 8 videos/GPU on 8×NVIDIA H100
• EMA decay: $x_0^{\ell}$5
• DPO temperature: $x_0^{\ell}$6–$x_0^{\ell}$7
• Dataset: 10 passes over RealAction-5K
• Architecture: Diffusion-transformer backbone (e.g., CogVideoX-5B or similar), VAE-encoded video latents, denoised at 4× spatial downsampling

5. Empirical Results

5.1 Evaluation Methodology

• User Study: Human raters scored outputs on Overall Quality, Visual Alignment, Text Alignment, Motion Quality, Human Quality (converted to win-rates).
• Automated LLM-scoring: Qwen2-VL evaluated the same axes.
• Objective Metrics: VBench-I2V suite (I2V Subject, Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality).

5.2 Quantitative Performance

Model	Overall	Visual	Text	Motion	Human
CogVideoX-5B	65–67	–	–	–	–
RealDPO	73.3	77.4	77.0	71.0	72.9

VBench-I2V scores indicate that RealDPO improves motion smoothness and dynamic degree over all state-of-the-art baselines. The Qwen2-VL LLM’s assessments corroborate these human ratings (Cheng et al., 16 Oct 2025).

5.3 Qualitative Analysis

RealDPO-synthesized videos display smoother and more anatomically plausible limb movements, stable backgrounds, and correct fine-grained action phases (e.g., realistic transitions in drinking, walking sequences). Compared to SFT (which may produce occasional limb collapse) and LiFT (which misses motion detail), RealDPO outputs are consistently more text-aligned and motion-realistic.

6. Ablations, Analysis, and Comparative Insights

Experiments replacing real "win" samples with synthetic outputs as in VideoAlign result in a 6–8% drop in motion-quality win-rate, confirming the importance of using real data to anchor physical plausibility. Ablations on the number of negative samples per positive reveal diminishing returns beyond three negatives. Hyperparameter sensitivity shows that overly sharp $x_0^{\ell}$8 penalizes negatives excessively, while too small values yield sluggish updates ($x_0^{\ell}$9–$\hat{x}_0^{\text{w}}, \hat{x}_0^{\ell}$0 recommended) (Cheng et al., 16 Oct 2025).

7. Context and Relation to Preference Optimization Variants

RealDPO generalizes DPO from the RLHF setting to video diffusion, aligning closely with the UDRRA theoretical framework for preference optimization (Su et al., 5 Feb 2025). The main innovation lies in anchoring the preference signal to real-world observations rather than synthetic or reward-model-labeled outputs, thus bypassing reward hacking and providing a stable, data-driven learning signal. Related contemporary work—such as RDPO (Qian et al., 23 Jun 2025) for physics consistency—shows similar benefits in deploying automated or data-grounded preference signals but targets physical law fidelity rather than human motion realism.

A plausible implication is that "real-data-as-preference" approaches may generalize to other modalities where model errors are more evident relative to genuine world exemplars, potentially informing fine-tuning paradigms outside generative video.

---

Key Contribution: RealDPO establishes and empirically validates the "real-data-as-preference" paradigm for aligning generative video models, achieving marked improvements in motion realism and text-video consistency over both supervised fine-tuning and reward-model-based methods, with a compact and high-quality real video dataset as the backbone (Cheng et al., 16 Oct 2025).