VDInstruct: Reward Fine-Tuning for Video Generation

• VDInstruct is a method that improves alignment in text-to-video diffusion models by recasting reward fine-tuning as localized video editing.
• It leverages sparse segmental sampling and temporally attenuated rewards to enhance computational efficiency and temporal stability in video generation.
• The approach fine-tunes pretrained models with minimal additional parameters, leading to superior visual quality and stronger prompt adherence.

Searching arXiv for “VDInstruct” and closely related papers to ground the article. VDInstruct, referred to as InstructVideo in the underlying paper, is a method for aligning text-to-video diffusion models with human feedback through reward fine-tuning. It targets a persistent failure mode of pretrained video generators: outputs may be visually unappealing, structurally unstable, temporally inconsistent, or misaligned with the textual prompt because of the noise and varying quality of web-scale pretraining data. The method addresses two practical obstacles in reward optimization for video diffusion: the cost of repeatedly traversing the full DDIM sampling chain, and the absence of a dedicated large-scale video reward model. Its central design is to recast reward fine-tuning as editing and to reuse image reward models for video supervision through sparse temporal aggregation (Yuan et al., 2023).

1. Problem formulation and conceptual basis

VDInstruct is situated in the broader problem of improving the human preference alignment of pretrained text-to-video diffusion models. The paper assumes a standard reward-fine-tuning objective,

$$\mathcal{L}_r (\theta) = \mathbb{E}_{\mathbb{P}(\bm{c})} \mathbb{E}_{\mathbb{P}_{\theta}(x_0|\bm{c})} [- r(\bm{x}_0, \bm{c})],$$

where $\bm{c}$ is the text prompt, $\bm{x}_0$ is the generated video, and $r(\bm{x}_0,\bm{c})$ is the reward score. In the standard setting, every optimization step requires sampling a video through the full DDIM chain, often about 50 steps, which makes reward optimization expensive. The second obstacle is supervisory: the paper states that no dedicated video reward model exists at large scale, and that collecting sufficient video preference annotations is costly (Yuan et al., 2023).

The method’s response to these constraints is deliberately conservative. Rather than retraining video generation from scratch, it fine-tunes a pretrained model by making preference-guided local edits. This suggests a design philosophy in which alignment should preserve the model’s prior generative competence while nudging outputs toward improved visual quality and prompt adherence.

2. Editing-based reward fine-tuning

The first main contribution is the recasting of reward fine-tuning as editing a sampled video. Instead of beginning each optimization step from pure text and denoising an entire video from scratch, the method starts from a real video-text pair drawn from the pretraining data, encodes the video into latent space, corrupts that latent to a chosen noise level, and then performs only a partial DDIM denoising trajectory. The edit is then evaluated by the reward model. The DDIM-to-DDPM step mapping is defined as

$d(i) = \frac{T}{D} \cdot (i-1) + 1,$

and, given a noise fraction $\tau$, the starting diffusion step is

$t_{\rm noi} = d(\tau \cdot D).$

The model then performs only $\tau \cdot D$ DDIM steps to recover an edited latent $\bm{z}_0$, rather than executing the full chain (Yuan et al., 2023).

This reformulation is important for both computational and statistical reasons. It reduces the number of sampling steps per update, lowers computation, and improves fine-tuning efficiency, while preserving the model’s ability to generate directly from text at inference time. The paper’s account implies that editing constrains optimization to a local region of the output space, which helps explain why the resulting system improves human preference alignment without obviously sacrificing generalization.

3. Segmental Video Reward and Temporally Attenuated Reward

The second main contribution is a reward mechanism that reuses off-the-shelf image reward models, especially HPSv2, for video preference supervision. The method is motivated by the assumption that video quality is largely determined by both the quality of individual frames and the smoothness of transitions across frames. Rather than score every frame densely, VDInstruct divides a video $$\bm{x}_0 \in \mathbb{R}^{F \times H \times W \times 3}$$ into $\bm{c}$0 temporal segments and samples one frame uniformly from each segment:

$\bm{c}$1

This yields a sparse subset $\bm{c}$2, and the reward is aggregated over sampled frames:

$\bm{c}$3

The simplest choice of aggregation is the mean (Yuan et al., 2023).

The paper calls this mechanism Segmental Video Reward (SegVR). Its purpose is not only computational economy but temporal stability. Dense frame scoring is reported to cause degraded motion continuity, structural twitching, and color jittering. Sparse segmental sampling therefore functions as a regularizer on reward density.

To further stabilize training, the method introduces Temporally Attenuated Reward (TAR), which weights central frames more heavily than peripheral frames. The attenuation coefficient is

$\bm{c}$4

and the reward becomes

$\bm{c}$5

The default setting is $\bm{c}$6. The stated intuition is that not all sampled frames are equally informative, and that forcing uniform optimization pressure across time can destabilize temporal learning (Yuan et al., 2023).

4. Training protocol and evaluation design

The base generator is ModelScopeT2V, trained on WebVid10M and implemented as a latent diffusion model. Default inference uses 20-step DDIM, and the paper also reports 50-step DDIM results for comparison. Fine-tuning follows DDPO-style evaluation on prompts describing 45 animal species, with about 20 video-text pairs per animal species selected from WebVid10M and no extra data introduced. About 6 prompts per animal are used for evaluation, and generalization is tested on new animals not seen during fine-tuning and on non-animal prompts (Yuan et al., 2023).

Parameter adaptation is performed with LoRA, applied to all Transformer linear layers for query, key, value, and output projections. The LoRA rank is 4, and the added parameters are about 1.58M, roughly 0.1% of the 1,347.44M-parameter base model. Training uses 4 NVIDIA A100s, a batch size of 8, learning rate $\bm{c}$7, and a default fine-tuning length of 20k steps. For efficiency, backpropagation is truncated to the last DDIM sampling step, following DRaFT-style efficiency tricks. At inference time, LoRA weights are merged back into the base model, so inference cost remains the same as ModelScopeT2V (Yuan et al., 2023).

Evaluation is organized around three axes: reward optimization performance using HPSv2 score on the first frames of all segments, pairwise human evaluation of video quality and video-text alignment, and out-of-domain generalization across unseen prompt distributions. The structure of the evaluation makes the method notable not only as a reward-maximization procedure but as a controlled study of whether preference fine-tuning can preserve broad prompt-conditioned generation.

5. Quantitative results, ablations, and limitations

On unseen prompts, VDInstruct outperforms the compared baselines, including ModelScopeT2V, DDPO, RWR, and DRaFT, with reported HPSv2 scores of 0.2717 ± 0.0137 for in-domain prompts, 0.2645 ± 0.0125 for new animals, and 0.2682 ± 0.0202 for non-animals. The paper interprets this as evidence of better generalization rather than overfitting to fine-tuning prompts. In human evaluation, two annotators with generative-model experience judge pairwise outputs. Against ModelScopeT2V, 75.5% prefer VDInstruct on quality, 15.5% tie, and 9.0% prefer the alternative; on alignment, 28.5% prefer VDInstruct, 57.0% tie, and 14.5% prefer the alternative. Against DRaFT, 76.0% prefer VDInstruct on quality and 30.0% on alignment. The strongest reported gain is in visual quality (Yuan et al., 2023).

The ablations are structurally important. For the corruption strength, small $\bm{c}$8 gives too little editable space, moderate $\bm{c}$9 gives the best tradeoff, and too large $\bm{x}_0$0 increases compute and slows improvement; the best reported choice is $\bm{x}_0$1. For temporal attenuation, large $\bm{x}_0$2 causes rewards to decay too quickly toward border frames, while small $\bm{x}_0$3 makes rewards too uniform and destabilizes training; the best reported choice is $\bm{x}_0$4. The paper also states that removing either SegVR or TAR reduces temporal modeling quality and causes visible collapse-like artifacts. From the reward-curve comparison, RWR and DDPO plateau after about 11 hours, whereas InstructVideo and DRaFT lag initially but later improve more effectively, with VDInstruct strongest among the direct backpropagation methods (Yuan et al., 2023).

The limitations are explicit. First, image reward models are not ideal video reward models, and a dedicated video reward model might assess the video more holistically. Second, as in other reward-based fine-tuning methods, excessive optimization can degrade quality even if the reward score rises. Third, temporal modeling remains delicate: if the reward signal is too dense or too strong, temporal collapse can occur. These constraints delimit the method’s scope. A plausible implication is that the main advance is not the full resolution of video preference learning, but a practically workable compromise between reward quality, temporal stability, and fine-tuning cost.

6. Relation to adjacent instruction-based visual systems

VDInstruct belongs to a wider turn toward instruction-conditioned visual systems, but its role is specific: it addresses alignment and preference optimization for video generation rather than instruction following in discriminative or interactive settings. In visual document understanding, InstructDoc aggregates 30 publicly available VDU datasets into 12 task types and pairs them with InstructDr, a document-aware model built on BLIP-2 and FlanT5 for zero-shot instruction-based VDU (Tanaka et al., 2024). In video understanding evaluation, VCIFBench introduces a benchmark with 306 satisfiable test instructions, a 540-pair DPO preference dataset, and a 30-item conflict diagnostic subset to measure whether multimodal models can satisfy explicit content, format, style, and structure constraints jointly (Xu et al., 3 Jun 2026).

Instruction conditioning also appears in physically embodied and interactive generation settings. Vega introduces InstructScene, containing around 100,000 scenes with driving instructions and corresponding trajectories, and proposes a unified Vision-Language-World-Action model for instruction-based generation and planning (Zuo et al., 26 Mar 2026). InstructVTON frames virtual try-on as natural-language-guided inpainting with automatic mask generation through VLMs and segmentation models, enabling multi-round execution for styling requests such as rolled sleeves or layered garments (Han et al., 24 Sep 2025). By contrast, Deferred Visual Ingestion in visually dense document QA adopts the principle “Index for locating, not understanding”, emphasizing demand-side page localization and query-time visual reasoning rather than full pre-ingestion understanding (Xu, 15 Feb 2026).

Taken together, these systems suggest that instruction-based visual modeling has diversified into at least three partially distinct regimes: controllable generation, constrained understanding, and task-specific interaction or planning. VDInstruct is most naturally placed in the first regime. Its distinctive contribution is to show that human-preference alignment for text-to-video diffusion can be made computationally tractable by combining editing-based reward fine-tuning with sparse, temporally attenuated supervision.