---
title: 'VDInstruct: Reward Fine-Tuning for Video Generation'
url: https://www.emergentmind.com/topics/vdinstruct
type: topic
---

# VDInstruct: Reward Fine-Tuning for Video Generation

Searching arXiv for “VDInstruct” and closely related papers to ground the article.
VDInstruct, referred to as **InstructVideo** in the underlying paper, is a method for aligning **text-to-video diffusion models** with **human feedback** through **reward fine-tuning**. It targets a persistent failure mode of pretrained video generators: outputs may be visually unappealing, structurally unstable, temporally inconsistent, or misaligned with the textual prompt because of the noise and varying quality of web-scale pretraining data. The method addresses two practical obstacles in reward optimization for video diffusion: the cost of repeatedly traversing the full DDIM sampling chain, and the absence of a dedicated large-scale video reward model. Its central design is to recast reward fine-tuning as **editing** and to reuse **image reward models** for video supervision through sparse temporal aggregation [2312.12490].

## 1. Problem formulation and conceptual basis

VDInstruct is situated in the broader problem of improving the human preference alignment of pretrained text-to-video diffusion models. The paper assumes a standard reward-fine-tuning objective,
$$
\mathcal{L}_r (\theta) = \mathbb{E}_{\mathbb{P}(\bm{c})} \mathbb{E}_{\mathbb{P}_{\theta}(x_0|\bm{c})} [- r(\bm{x}_0, \bm{c})],
$$
where \(\bm{c}\) is the text prompt, \(\bm{x}_0\) is the generated video, and \(r(\bm{x}_0,\bm{c})\) is the reward score. In the standard setting, every optimization step requires sampling a video through the full DDIM chain, often about 50 steps, which makes reward optimization expensive. The second obstacle is supervisory: the paper states that no dedicated video reward model exists at large scale, and that collecting sufficient video preference annotations is costly [2312.12490].

The method’s response to these constraints is deliberately conservative. Rather than retraining video generation from scratch, it fine-tunes a pretrained model by making preference-guided local edits. This suggests a design philosophy in which alignment should preserve the model’s prior generative competence while nudging outputs toward improved visual quality and prompt adherence.

## 2. Editing-based reward fine-tuning

The first main contribution is the recasting of reward fine-tuning as **editing a sampled video**. Instead of beginning each optimization step from pure text and denoising an entire video from scratch, the method starts from a real video-text pair drawn from the pretraining data, encodes the video into latent space, corrupts that latent to a chosen noise level, and then performs only a partial DDIM denoising trajectory. The edit is then evaluated by the reward model. The DDIM-to-DDPM step mapping is defined as
$$
d(i) = \frac{T}{D} \cdot (i-1) + 1,
$$
and, given a noise fraction \(\tau\), the starting diffusion step is
$$
t_{\rm noi} = d(\tau \cdot D).
$$
The model then performs only \(\tau \cdot D\) DDIM steps to recover an edited latent \(\bm{z}_0\), rather than executing the full chain [2312.12490].

This reformulation is important for both computational and statistical reasons. It reduces the number of sampling steps per update, lowers computation, and improves fine-tuning efficiency, while preserving the model’s ability to generate directly from text at inference time. The paper’s account implies that editing constrains optimization to a local region of the output space, which helps explain why the resulting system improves human preference alignment without obviously sacrificing generalization.

## 3. Segmental Video Reward and Temporally Attenuated Reward

The second main contribution is a reward mechanism that reuses **off-the-shelf image reward models**, especially **HPSv2**, for video preference supervision. The method is motivated by the assumption that video quality is largely determined by both the quality of individual frames and the smoothness of transitions across frames. Rather than score every frame densely, VDInstruct divides a video \(\bm{x}_0 \in \mathbb{R}^{F \times H \times W \times 3}\) into \(S\) temporal segments and samples one frame uniformly from each segment:
$$
g(i) = {\rm Uniform}\left( (i-1)\cdot \frac{F}{S},\; i\cdot \frac{F}{S} - 1 \right).
$$
This yields a sparse subset \(\bm{x}_0^g\), and the reward is aggregated over sampled frames:
$$
R = {\rm Agg}_i [r(\bm{x}_0^{g(i)}, \bm{c})], \quad i=1,\ldots,S.
$$
The simplest choice of aggregation is the mean [2312.12490].

The paper calls this mechanism **Segmental Video Reward (SegVR)**. Its purpose is not only computational economy but temporal stability. Dense frame scoring is reported to cause degraded motion continuity, structural twitching, and color jittering. Sparse segmental sampling therefore functions as a regularizer on reward density.

To further stabilize training, the method introduces **Temporally Attenuated Reward (TAR)**, which weights central frames more heavily than peripheral frames. The attenuation coefficient is
$$
f_i = {\rm e}^{-\lambda_{\rm tar} |g(i) - \frac{F}{2}|},
$$
and the reward becomes
$$
R = \frac{1}{S} \sum_{i = 1}^{S}{f_i \cdot r(\bm{x}_0^{g(i)}, \bm{c})}.
$$
The default setting is \(\lambda_{\rm tar} = 1.0\). The stated intuition is that not all sampled frames are equally informative, and that forcing uniform optimization pressure across time can destabilize temporal learning [2312.12490].

## 4. Training protocol and evaluation design

The base generator is **ModelScopeT2V**, trained on **WebVid10M** and implemented as a latent diffusion model. Default inference uses **20-step DDIM**, and the paper also reports **50-step DDIM** results for comparison. Fine-tuning follows DDPO-style evaluation on prompts describing **45 animal species**, with about **20 video-text pairs per animal species** selected from WebVid10M and no extra data introduced. About **6 prompts per animal** are used for evaluation, and generalization is tested on **new animals** not seen during fine-tuning and on **non-animal prompts** [2312.12490].

Parameter adaptation is performed with **LoRA**, applied to all Transformer linear layers for query, key, value, and output projections. The LoRA rank is **4**, and the added parameters are about **1.58M**, roughly **0.1%** of the **1,347.44M-parameter** base model. Training uses **4 NVIDIA A100s**, a **batch size of 8**, learning rate \(1 \times 10^{-5}\), and a default fine-tuning length of **20k steps**. For efficiency, backpropagation is truncated to the **last DDIM sampling step**, following DRaFT-style efficiency tricks. At inference time, LoRA weights are merged back into the base model, so inference cost remains the same as ModelScopeT2V [2312.12490].

Evaluation is organized around three axes: reward optimization performance using **HPSv2 score** on the first frames of all segments, pairwise human evaluation of **video quality** and **video-text alignment**, and out-of-domain generalization across unseen prompt distributions. The structure of the evaluation makes the method notable not only as a reward-maximization procedure but as a controlled study of whether preference fine-tuning can preserve broad prompt-conditioned generation.

## 5. Quantitative results, ablations, and limitations

On unseen prompts, VDInstruct outperforms the compared baselines, including **ModelScopeT2V**, **DDPO**, **RWR**, and **DRaFT**, with reported HPSv2 scores of **0.2717 ± 0.0137** for in-domain prompts, **0.2645 ± 0.0125** for new animals, and **0.2682 ± 0.0202** for non-animals. The paper interprets this as evidence of better generalization rather than overfitting to fine-tuning prompts. In human evaluation, two annotators with generative-model experience judge pairwise outputs. Against ModelScopeT2V, **75.5%** prefer VDInstruct on quality, **15.5%** tie, and **9.0%** prefer the alternative; on alignment, **28.5%** prefer VDInstruct, **57.0%** tie, and **14.5%** prefer the alternative. Against DRaFT, **76.0%** prefer VDInstruct on quality and **30.0%** on alignment. The strongest reported gain is in visual quality [2312.12490].

The ablations are structurally important. For the corruption strength, small \(\tau\) gives too little editable space, moderate \(\tau\) gives the best tradeoff, and too large \(\tau\) increases compute and slows improvement; the best reported choice is \(\tau = 0.6\). For temporal attenuation, large \(\lambda_{\rm tar}\) causes rewards to decay too quickly toward border frames, while small \(\lambda_{\rm tar}\) makes rewards too uniform and destabilizes training; the best reported choice is \(\lambda_{\rm tar} = 1.0\). The paper also states that removing either SegVR or TAR reduces temporal modeling quality and causes visible collapse-like artifacts. From the reward-curve comparison, **RWR** and **DDPO** plateau after about **11 hours**, whereas **InstructVideo** and **DRaFT** lag initially but later improve more effectively, with VDInstruct strongest among the direct backpropagation methods [2312.12490].

The limitations are explicit. First, image reward models are not ideal video reward models, and a dedicated video reward model might assess the video more holistically. Second, as in other reward-based fine-tuning methods, excessive optimization can degrade quality even if the reward score rises. Third, temporal modeling remains delicate: if the reward signal is too dense or too strong, temporal collapse can occur. These constraints delimit the method’s scope. A plausible implication is that the main advance is not the full resolution of video preference learning, but a practically workable compromise between reward quality, temporal stability, and fine-tuning cost.

## 6. Relation to adjacent instruction-based visual systems

VDInstruct belongs to a wider turn toward **instruction-conditioned** visual systems, but its role is specific: it addresses **alignment and preference optimization for video generation** rather than instruction following in discriminative or interactive settings. In visual document understanding, **InstructDoc** aggregates **30 publicly available VDU datasets** into **12 task types** and pairs them with **InstructDr**, a document-aware model built on BLIP-2 and FlanT5 for zero-shot instruction-based VDU [2401.13313]. In video understanding evaluation, **VCIFBench** introduces a benchmark with **306 satisfiable test instructions**, a **540-pair DPO preference dataset**, and a **30-item conflict diagnostic subset** to measure whether multimodal models can satisfy explicit content, format, style, and structure constraints jointly [2606.04588].

Instruction conditioning also appears in physically embodied and interactive generation settings. **Vega** introduces **InstructScene**, containing around **100,000 scenes** with driving instructions and corresponding trajectories, and proposes a unified **Vision-Language-World-Action** model for instruction-based generation and planning [2603.25741]. **InstructVTON** frames virtual try-on as natural-language-guided inpainting with automatic mask generation through VLMs and segmentation models, enabling multi-round execution for styling requests such as rolled sleeves or layered garments [2509.20524]. By contrast, **Deferred Visual Ingestion** in visually dense document QA adopts the principle **“Index for locating, not understanding”**, emphasizing demand-side page localization and query-time visual reasoning rather than full pre-ingestion understanding [2602.14162].

Taken together, these systems suggest that instruction-based visual modeling has diversified into at least three partially distinct regimes: controllable **generation**, constrained **understanding**, and task-specific **interaction or planning**. VDInstruct is most naturally placed in the first regime. Its distinctive contribution is to show that human-preference alignment for text-to-video diffusion can be made computationally tractable by combining editing-based reward fine-tuning with sparse, temporally attenuated supervision.

Source: https://www.emergentmind.com/topics/vdinstruct