Video Generation Models Are Good Latent Reward Models (2511.21541v1)

Abstract: Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-LLMs designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.

• The paper introduces Process Reward Feedback Learning (PRFL) that leverages latent space rewards to improve video generation with superior motion dynamics.
• It employs a Process-Aware Video Reward Model (PAVRM) that evaluates noisy latent representations across denoising steps for fine-grained temporal alignment.
• Empirical results demonstrate significant gains in motion metrics and training acceleration, validating its effectiveness over pixel-space approaches.

Video Generation Models as Latent Reward Models: Process-Aware Reinforcement Learning for Video Generation

Introduction

The paper "Video Generation Models Are Good Latent Reward Models" (2511.21541) presents a process-level framework for reward modeling and reward feedback learning (ReFL) in text-to-video and image-to-video generation. The work addresses key limitations of prior video reward modeling approaches, which depend heavily on vision-LLMs (VLMs) operating in pixel space. This reliance incurs substantial computational overhead–particularly with high-resolution videos–and limits reward signal propagation to late-stage denoising, inhibiting precise control over temporally grounded properties such as motion quality and structural consistency.

The authors introduce a new methodology: Process Reward Feedback Learning (PRFL), underpinned by a Process-Aware Video Reward Model (PAVRM) adapted from pre-trained video generation models (VGMs). This approach directly models rewards in the noisy latent space and applies timestep-aware supervision throughout the denoising process, substantially improving the effectiveness of preference-driven video generation alignment.

Figure 1: High-level comparison of RGB ReFL (pixel-space) and PRFL (latent-space) video reward modeling, highlighting PRFL's direct, memory-efficient supervision through latent-space modeling.

Background and Problem Statement

Traditionally, reward feedback learning for generative models operates in the outcome (pixel) space. For videos, this mandates near-complete denoising and VAE decoding before VLM reward models can evaluate outputs, resulting in:

• Evaluation latency: Complete denoising and decoding are needed per training step.
• Memory bottlenecks: Processing all video frames through a high-capacity VAE is computationally impractical, often resulting in memory exhaustion.
• Sparse supervision: Pixel-space rewards only affect late (output) stages, precluding fine-grained shaping of early generative steps critical for spatiotemporal and motion quality.

Initial latent reward modeling works for images, such as LPO [zhang2025diffusion], motivate the transition into process-aware latent-space rewards. However, previous explorations for video either focus on single frames (e.g., ContentV) or lack timestep-wise optimization (e.g., DOLLAR), failing to fully exploit the sequential, temporal nature of the video denoising trajectory.

PAVRM: Process-Aware Video Reward Modeling

The proposed PAVRM leverages a pre-trained video diffusion backbone, specifically employing the early DiT blocks to extract spatiotemporal features from noisy latent representations at arbitrary denoising timesteps. A learnable query-based attention mechanism aggregates features into a timestep-sensitive, quality-aware token, which is mapped to a reward score via a multi-layer perceptron (MLP) head.

Key properties:

• Timestep Sensitivity: PAVRM is trained to evaluate latent representations across the full denoising spectrum ([0,1] in rectified flow), providing reward signals not only at the clean output but mid-trajectory.
• Structure and Motion Encoding: Latent features from VGMs inherently encode both motion and structure as a result of sequential training objectives, yielding effective reward surrogates for preference modeling.
• Computational Efficiency: By operating entirely in latent space, PAVRM avoids VAE decoding overhead and enables full-batch, full-frame reward modeling.

  Figure 2: Empirical analysis of VGM feature usefulness—VGM-based reward modeling achieves robust accuracy across denoising timesteps, outperforming VLM-based models under timestep-aware fine-tuning.

PRFL: Latent-Space Reward Feedback Learning

PRFL operationalizes the PAVRM reward in an online post-training framework. At each training step, the model:

• Samples a random denoising timestep $t$.
• Constructs the corresponding noisy latent $\mathbf{x}_t$.
• Backpropagates the PAVRM reward loss from this intermediate state.

Supervised fine-tuning (SFT) regularization is alternated to prevent over-optimization and preserve motion diversity. This enables full-trajectory alignment, propagating preference rewards into both early (structural formation) and late (visual refinement) stages.

Figure 3: Schematic of the process-aware video generation alignment: training phases for PAVRM (reward modeling from noisy latents) and PRFL (reinforcement learning from latent-space rewards across random timesteps).

Experimental Evaluation

Quantitative Results

On the VBench and VBench2 test sets for text-to-video and image-to-video at 480P and 720P resolutions, PRFL yields substantial gains in motion-related metrics:

• Dynamic degree improves by up to +56.00 points (Inner Test Set, 720P).
• Human anatomy and subject consistency remain high or improve, indicating no degradation in other critical metrics.
• PRFL outperforms SFT, reward-weighted regression (RWR), and RGB ReFL baselines across all conditions, with the most pronounced effect in motion dynamics.

Human Evaluation and Qualitative Analysis

A rigorous user paper with 2,250 pairwise comparisons from professional annotators confirms that PRFL-trained models are consistently judged superior in holistic text-video alignment, visual fidelity, and especially motion realism compared to all baselines.

Figure 4: Human evaluation outcome; PRFL receives highest preference versus all competitive post-training methods.

Qualitative results on challenging benchmarks further showcase the artifact mitigation capacity of PRFL, with baseline models presenting deformations, temporal inconsistencies, or prompt misalignment in situations with complex motion or multi-agent interaction, whereas PRFL models maintain high visual coherence.

Figure 5: Frame comparison for baseline and PRFL models on complex motion scenarios. PRFL models mitigate common artifacts and better preserve subject integrity.

Training Efficiency

PRFL achieves notable training acceleration (1.4–1.5× speedup) and substantial GPU memory reductions over RGB ReFL, even when processing all frames (not just the first), with practical VRAM footprints on modern accelerators. In contrast, pixel-space approaches are untenable for high-resolution video due to memory exhaustion.

Timestep Sensitivity

Sampling reward steps across the full denoising chain is crucial—early-stage reward optimizes motion and dynamic degree, while late-stage focusing enhances anatomy and persistence. Full-range PRFL yields optimal aggregate performance, confirming the necessity for process-aware signals.

Architectural Insights and Generalization

Ablations demonstrate that moderate model depth (8–16 DiT layers) in PAVRM suffices for reward modeling—overly deep reward architectures risk overfitting to texture details and lose generalization. PAVRM also exhibits cross-model transferability: reward models trained solely on one generator (e.g., Wan2.1) retain strong discriminatory power on other models (e.g., HunyuanVideo, Veo3) when evaluated in the high-noise regime, reflecting the universality of latent space structural cues.

Practical and Theoretical Implications

The paper's results suggest that pre-trained video diffusion backbones are effective, computationally efficient reward models for RLHF-style alignment in generative video domains. This unlocks:

• Scalable preference alignment for long or high-resolution videos with commodity hardware.
• Fine-grained temporal supervision along the generative chain, essential for spatiotemporal and physical plausibility.
• The potential for hybrid reward systems that combine VGM-based process rewards, VLM outcome models, and richer multi-aspect signals (e.g., semantics, aesthetics).

Conclusion

This work establishes that video generation models are highly effective as latent reward models and that process-aware, timestep-sensitive reinforcement learning in latent space overcomes major computational and optimization deficiencies of existing video ReFL methods. By integrating the inherent inductive biases of VGMs and eschewing pixel-space bottlenecks, this paradigm achieves superior alignment with human motion preferences, artifact suppression, and practical acceleration for large-scale video generation.

Future development should explore composite, modular reward schemes encompassing multiple axes of human preference, reward decomposition for editability and controllability, and process-level alignment frameworks beyond motion (e.g., narrative structure, emotional flow). The underlying principles from this line of work are broadly extensible to other temporally-structured generative domains.