ReDPO: Preference Optimization for Video Diffusion

• ReDPO is a targeted preference-based loss that guides pruned text-to-video diffusion models to recover critical generative properties, avoiding oversmoothing and blurred outputs.
• It combines a diffusion-form DPO objective with an SFT regularizer to replace exact teacher imitation with selective property recovery, enhancing model stability.
• Embedded in the iterative V.I.P. framework, ReDPO enables significant model compression—achieving up to 67.5% parameter reduction—while maintaining or improving video fidelity.

Searching arXiv for the specified paper and closely related work to ground the article. ReDPO, short for Regularized Diffusion Preference Optimization, is a preference-based distillation loss proposed for pruning and distilling text-to-video diffusion models under capacity mismatch. It is introduced within the broader V.I.P. framework, whose title expansion is Iterative Online Preference Distillation and whose method section also describes it as Video diffusion distillation via Iterative Preference Optimization. The method is designed for the setting in which aggressively pruned students cannot exactly reproduce the denoising behavior of a full teacher, making purely imitation-based supervised fine-tuning (SFT) a poor fit. ReDPO replaces exact global imitation with a preference-guided objective that directs the student toward the properties most damaged by pruning, while retaining an SFT term to stabilize optimization and preserve absolute sample quality (Kim et al., 5 Aug 2025).

1. Problem setting and motivation

ReDPO is motivated by a specific failure mode of SFT-only distillation for text-to-video diffusion. In the setting considered by the paper, prior distillation methods largely rely on SFT, typically an $L_2$ loss between teacher and student noise predictions or features. The paper argues that this is ill-suited to low-capacity students because a pruned student cannot match the teacher everywhere. Minimizing SFT under this capacity gap induces distributional averaging, in which the student allocates scarce capacity to reducing average error globally rather than preserving the most important generative modes. The stated consequences are oversmoothing, blurry outputs, weaker motion dynamics, samples outside the teacher’s distribution, and overall quality degradation (Kim et al., 5 Aug 2025).

A second part of the motivation is selective degradation. The paper explicitly argues that pruning does not uniformly damage all properties: some attributes degrade, while others are preserved or can even improve. Under that condition, exact imitation is wasteful because SFT continues to “blindly mimic teacher,” including dimensions in which the student is already adequate or better. ReDPO is therefore framed not as a generic replacement for SFT, but as a mechanism for concentrating limited student capacity on the properties that actually need recovery.

The paper also contrasts this design with adversarial alternatives. It states that adversarial sharpening can be unstable and prone to distortion and mode collapse. ReDPO is positioned as a more targeted and more stable preference-based alternative for compressed diffusion students.

2. Objective formulation

Technically, ReDPO combines a diffusion-form DPO objective with an SFT regularizer. The trainable policy is the pruned student $\pi_\theta$, and the fixed reference model is the full teacher $\pi_{\mathrm{ref}}$. The DPO component does not require exact output matching on every sample. Instead, it encourages the student to assign relatively higher likelihood to a preferred sample than to a non-preferred sample, measured through denoising error relative to the fixed reference model. The paper states that this lets the student focus on “recovering only the targeted properties” rather than matching the teacher indiscriminately (Kim et al., 5 Aug 2025).

The preferred/dispreferred pair is written $(x_0^w, x_0^l)\sim \mathcal D$, where $w$ denotes the winning or chosen sample and $l$ the losing or rejected sample. Diffusion timesteps are sampled as $t\sim \mathcal U(0,T)$, and noisy latents are drawn from the forward process $q(x_t\mid x_0)$. The true noises are $\epsilon^w$ and $\epsilon^l$, the student prediction is $\pi_\theta$0, the teacher prediction is $\pi_\theta$1, $\pi_\theta$2 is the logistic sigmoid, $\pi_\theta$3 is the DPO scale, $\pi_\theta$4 is the diffusion horizon, $\pi_\theta$5 is a timestep-dependent weighting function inherited from diffusion DPO, and $\pi_\theta$6 weights the regularizer.

The SFT regularizer is defined on the preferred sample only: $\pi_\theta$7

The final ReDPO loss is: $\pi_\theta$8

The paper writes this as $\pi_\theta$9, where $\pi_{\mathrm{ref}}$0 refers to ReDPO. No explicit preference margin beyond the logistic relative score is introduced, and no extra KL regularizer is written explicitly in the final loss, although the text states that DPO’s KL-style constraint alone is insufficient to prevent over-optimization.

The paper’s intuition is precise. The DPO term compares how much worse the student is than the reference on a chosen sample versus a rejected sample. If the student is especially poor on the chosen sample relative to the teacher, and less poor on the rejected one, the loss pushes the student to correct that imbalance. This is why the paper characterizes ReDPO as targeted preference recovery rather than teacher imitation. Because pure DPO can over-optimize and damage absolute quality of the preferred sample, the SFT term on the winning sample anchors the student to the teacher.

3. Preference construction and the V.I.P. framework

ReDPO is operationalized inside V.I.P., which supplies the curriculum and the preference data rather than treating optimization as a single offline pass. The framework is iterative. It begins from the full model $\pi_{\mathrm{ref}}$1, prunes a small number of blocks, diagnoses the most degraded property, curates targeted teacher–student preference pairs for that property, trains the pruned student with ReDPO, and then repeats at the next stage (Kim et al., 5 Aug 2025).

Preference construction is fully automatic. Prompts are first filtered so that they are suitable for video generation and relevant to the property currently targeted for recovery. The full model and the current pruned model then generate videos for the same prompts. These videos are scored by VideoScore, which the paper uses as a human-preference-aligned reward model. Preferred/dispreferred pairs are formed when the full model outperforms the pruned model on the target property, under the condition

$\pi_{\mathrm{ref}}$2

where $\pi_{\mathrm{ref}}$3 is the reward score for video $\pi_{\mathrm{ref}}$4, and $\pi_{\mathrm{ref}}$5 is a minimum threshold for targeted property $\pi_{\mathrm{ref}}$6.

The supplement adds two further curation constraints. First, the rejected sample must remain above

$\pi_{\mathrm{ref}}$7

with $\pi_{\mathrm{ref}}$8, so that the losing sample is still meaningful. Second, the gap in the targeted property must exceed the gap in any other property, to reduce confounding and make the pair signal specific to the intended weakness.

Prompt filtering is also staged. Prompts are restricted to 5–25 words with articles removed. Then Gemini 2.0 Flash scores them for relevance to the target property. For dynamic degree, prompts with score 3 are selected; for visual quality, prompts with score 3 are selected; for text alignment, nonzero-score prompts from the dynamic and visual-quality pools are selected, with additional linguistic constraints. Preferences are defined over full generated videos, scored by video-level reward metrics, while the ReDPO loss supervises the corresponding diffusion denoising trajectories.

The paper describes this as an online or on-policy framework because the rejected samples are continually regenerated from the current student rather than taken from a static dataset. The full teacher remains fixed as $\pi_{\mathrm{ref}}$9; the student changes from stage to stage. This continual regeneration is presented as a central reason that the preference data remain calibrated to the student’s actual failure modes.

4. Pruning curriculum, architectures, and training procedure

V.I.P. uses a stepwise pruning procedure. For the current model $(x_0^w, x_0^l)\sim \mathcal D$0 with $(x_0^w, x_0^l)\sim \mathcal D$1 modules, the supplement formalizes Algorithm 1: Step-by-Step Pruning Algorithm as follows. For each module $(x_0^w, x_0^l)\sim \mathcal D$2, compute the performance drop

$(x_0^w, x_0^l)\sim \mathcal D$3

Then select the $(x_0^w, x_0^l)\sim \mathcal D$4 modules with the smallest $(x_0^w, x_0^l)\sim \mathcal D$5, prune them to obtain $(x_0^w, x_0^l)\sim \mathcal D$6, perform dataset curation and training on $(x_0^w, x_0^l)\sim \mathcal D$7, update $(x_0^w, x_0^l)\sim \mathcal D$8, and repeat until the desired pruning level is reached (Kim et al., 5 Aug 2025).

The supplement states that each pruning stage removes four blocks: four motion-module blocks for AnimateDiff and four U-Net blocks for VideoCrafter2. Candidate pruned models are sorted using both total VideoScore and consistency, because a misleadingly high dynamic score can mask poor motion quality; final block choices are taken from the intersection of top-ranked models under total score and consistency.

The evaluation setting spans two U-Net-based text-to-video systems. For AnimateDiff, pruning is applied only to the motion module, because the Stable Diffusion 1.5 U-Net backbone is frozen and only the motion module is trained. For VideoCrafter2, pruning is applied to entire U-Net blocks. The paper states that ReDPO is not limited to distillation and can be applied robustly for general diffusion preference alignment purposes, but the experiments reported here are confined to video diffusion U-Nets.

The implementation details given explicitly are the following. Prompts are filtered from WebVid-10M, and the pair dataset uses a 2k prompt subset. Training uses batch size 2, 2 epochs per stage, and 4 A100 GPUs. The learning rate is $(x_0^w, x_0^l)\sim \mathcal D$9. The DPO scale is $w$0, following VideoDPO. The SFT weight is $w$1 for AnimateDiff and $w$2 for VideoCrafter2. The teacher/reference model remains the full model $w$3 throughout training.

5. Empirical results, ablations, and limitations

The paper reports that under V.I.P., ReDPO achieves 36.2% parameter reduction on VideoCrafter2 and 67.5% parameter reduction on AnimateDiff while maintaining or exceeding full-model performance (Kim et al., 5 Aug 2025).

Model	Parameters	VideoScore average
VideoCrafter2 full vs final ReDPO student	1.413B vs 0.902B	2.613 vs 2.623
AnimateDiff full vs final ReDPO student	0.453B vs 0.147B	2.563 vs 2.568

The reported TFLOPs also decrease: VideoCrafter2 drops from 9.4 to 7.4 TFLOPs and AnimateDiff from 4.9 to 3.3 TFLOPs. Thus the compression is not purely parametric; it also reduces compute.

Against SFT, ReDPO improves consistently on the direct VideoScore comparison. For VideoCrafter2, SFT yields visual 2.628, temporal 2.613, dynamic 2.724, and text alignment 2.505, whereas ReDPO yields visual 2.629, temporal 2.617, dynamic 2.728, and text alignment 2.518. For AnimateDiff, SFT yields visual 2.564, temporal 2.515, dynamic 2.679, and text alignment 2.477, whereas ReDPO yields visual 2.569, temporal 2.513, dynamic 2.695, and text alignment 2.496. On VBench, ReDPO also exceeds SFT on the reported quality and semantics-oriented metrics; for example, semantic score rises from 72.2 to 73.9 on VideoCrafter2 and from 74.7 to 76.8 on AnimateDiff.

The ablations isolate both components of the method. In Table 4, removing the SFT term (w/o SFT) hurts most metrics markedly, especially temporal consistency and text alignment, supporting the claim that pure DPO over-optimizes and requires regularization. For AnimateDiff, w/o SFT produces temporal consistency 2.437 versus 2.513 for the full V.I.P.+ReDPO model, despite a superficially high dynamic degree 2.744, which the authors interpret as poor-quality motion rather than better video. Removing the online/iterative setting (w/o online) also degrades most metrics, supporting the value of stagewise pruning and pair regeneration.

The SFT-weight ablation on VideoCrafter2 shows that $w$4 is a substantive control parameter. The paper tests $w$5 through $w$6. Too small a weight causes abnormally high dynamics and degraded quality; too large a weight collapses the objective back toward teacher imitation and broadly degrades performance. The best operating point in the main setup is around $w$7 for VideoCrafter2.

The limitations stated by the paper are also concrete. V.I.P. introduces additional pipeline complexity because it requires repeated evaluation, prompt filtering, sample generation from teacher and student, reward scoring, and multistage retraining. The paper does not quantify wall-clock or data-generation overhead beyond the training setup. It is dependent on the reward model, specifically VideoScore. Experiments are limited to U-Net-based video diffusion models rather than DiT-based architectures. The paper also notes that heavy pruning can cause severe degradation; for VideoCrafter2 stage 3, quality drops drastically under extreme pruning, though ReDPO still recovers a large portion of the loss. A human user study with 30 participants, 36 prompt sets across both model families, and 1080 responses indicates that the V.I.P.+ReDPO model is preferred overall to both SFT and the full model, but exact percentages are not provided in the text snippet.

6. Position within the DPO landscape and terminological distinctions

ReDPO belongs to a broader line of work that treats preference optimization as something requiring explicit control over imbalance, over-optimization, or coarse response-level supervision. In LLM alignment, "3D-Properties: Identifying Challenges in DPO and Charting a Path Forward" diagnoses DPO in terms of a drastic drop in rejected-response likelihood, degradation into suppression, and dispersion to unseen responses, and proposes asymmetric weighting and SFT-style regularization as remedies (Yan et al., 2024). "Gradient Imbalance in Direct Preference Optimization" formalizes winner–loser gradient asymmetry and proposes Balanced-DPO as a reweighting mechanism (Ma et al., 28 Feb 2025). Those papers are not about video diffusion, but they make the regularization logic behind ReDPO legible: preference learning alone can be structurally insufficient, and a stabilizing term can be necessary.

At the same time, ReDPO should not be conflated with similarly named methods. ReDiPO, introduced in "Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs," is an offline data-construction pipeline for standard DPO in LLMs; it is explicitly not a new preference-optimization loss and not ReDPO (Samuel et al., 28 May 2026). RePO, from "ReLU-based Preference Optimization," is a reference-free, ReLU max-margin objective for LLM preference learning; it is likewise not ReDPO (Wu et al., 10 Mar 2025). A different structural modification appears in ADPO, which changes the preference-model factorization by moving the summation outside the log-sigmoid in autoregressive LLM alignment (Oi et al., 10 Feb 2026). ReDPO, by contrast, is a diffusion-model objective for pruned text-to-video students that retains diffusion DPO structure while adding an SFT regularizer on the preferred sample.

The central contribution of ReDPO is therefore specific. It reformulates pruning distillation for text-to-video diffusion as a preference-learning problem under capacity mismatch. Rather than attempting exact teacher imitation, it compares teacher-preferred and student-failing samples and allocates the student’s limited capacity to the properties most harmed by pruning. Within V.I.P., that preference loss is embedded in an iterative pruning–diagnosis–curation–retraining loop, yielding compressed VideoCrafter2 and AnimateDiff systems that match or slightly exceed full-model quality while substantially reducing parameters and TFLOPs (Kim et al., 5 Aug 2025).