---
title: Video Prediction Policy (VPP)
url: https://www.emergentmind.com/topics/video-prediction-policy-vpp
type: topic
---

# Video Prediction Policy (VPP)

Searching arXiv for the cited VPP-related papers to ground the article in current literature.
Video Prediction Policy (VPP) denotes a family of policy-learning frameworks in which video prediction is not merely an auxiliary perceptual task but a primary mechanism for control, planning, or policy evaluation. In contemporary robotics, the term appears explicitly in "Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations" [2412.14803], while closely related formulations include joint image-action denoising [2411.18179], flow-based bimanual video prediction [2505.24156], latent-injection diffusion policies and planners [2601.16163], multi-view video action models [2604.03181], latent-action pretraining from actionless video [2511.07732], and action-conditional video world models for policy evaluation [2511.11520]. A historical precursor is the probabilistic future-prediction-and-control framework for autonomous driving, which jointly predicts semantics, depth, optical flow, and control from video [2003.06409].

## 1. Conceptual basis and lineage

The central premise of VPP is that predictive video models encode dynamics that static visual encoders often omit. Hu et al. argue that encoders pre-trained with single-image reconstruction or two-image contrastive learning tend to capture static information while neglecting dynamic aspects vital for embodied tasks, and therefore condition action learning on predicted future representations inside a video diffusion model [2412.14803]. PAD makes an even tighter coupling: image prediction and robotic action are treated as highly correlated because they share the same underlying dynamics of the physical world, so both are generated within a joint denoising process [2411.18179]. Cosmos Policy extends this logic by adapting a large pretrained video model into a robot policy that also predicts future state images and values, thereby combining policy learning, world modeling, and planning in one latent diffusion framework [2601.16163].

This research direction did not emerge ex nihilo. In autonomous driving, future prediction from video was already linked to control through a conditional-variational architecture that modeled a present distribution, a future distribution, probabilistic predictions of semantics, geometry, and motion, and a learned driving policy [2003.06409]. This suggests that VPP is best understood as a design principle: action selection is grounded in predictive visual dynamics rather than in static per-frame representations alone.

## 2. Architectural patterns

A common pattern is a two-stage architecture in which a video predictor supplies predictive features to a downstream action generator. In Hu et al.’s VPP, a text-guided video prediction model based on Stable Video Diffusion is fine-tuned on human and robot manipulation data, but the action policy does not fully denoise future video at inference time. Instead, it performs a single reverse-diffusion step, extracts intermediate U-Net features, aggregates them with a Video-Former, and conditions a diffusion policy head on the resulting predictive visual representations [2412.14803]. ViPRA uses a related decomposition, but its intermediate object is a motion-centric latent action space learned from passive video; a video-language model predicts future visual observations and latent actions, and a chunked flow-matching decoder maps those latent actions to robot-specific continuous action sequences [2511.07732].

A second pattern is joint modeling of visual prediction and action generation inside one denoising backbone. PAD defines parallel forward diffusion processes for image and action latents and uses a Diffusion Transformer to predict both modalities jointly; missing modalities can be handled by masked attention, which permits co-training on video data without action labels [2411.18179]. Cosmos Policy pushes this unification further by introducing no new MLP heads or dynamics modules at all: robot proprioception, an action chunk, future state frames, and a scalar value are each encoded as additional latent frames and interleaved with the image latents, so the pretrained diffusion transformer models them alongside images [2601.16163]. MV-VDP similarly keeps video generation central, but augments a pretrained single-view Video Foundation Model with a view-attention layer and jointly predicts multi-view RGB videos and heatmap videos, after which a dedicated action decoder reconstructs 3D trajectories, rotation, and gripper commands [2604.03181].

A third pattern is a cascaded planner-controller design. CogRobot treats bimanual manipulation as video prediction, first fine-tuning a text-to-flow model and then a flow-to-video model derived from a pretrained text-to-video backbone; a lightweight goal-conditioned diffusion policy then converts the predicted visual plan into joint commands [2505.24156]. In such systems, video prediction specifies an intended future, while a smaller controller handles embodiment-specific action realization.

## 3. Probabilistic formulations and training objectives

Most recent VPP variants inherit their learning dynamics from diffusion modeling. Cosmos Policy adopts a latent-diffusion formulation in which the clean latent sequence $z_0$ contains both image and non-image frames, and fine-tunes the denoiser with an EDM-style score-matching objective,
$$
\mathcal{L}_{\mathrm{diff}}=
\mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)}
\left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,
$$
while keeping the transformer architecture unchanged [2601.16163]. PAD defines separate forward diffusion processes over image and action latents and trains a multimodal denoiser with
$$
L(\theta)=\lambda_I L_{\mathrm{diff}}^I+\lambda_A L_{\mathrm{diff}}^A+\lambda_E L_{\mathrm{diff}}^E,
$$
where extra modalities such as depth can be appended directly to the token sequence [2411.18179]. Hu et al.’s VPP also splits training into video-model fine-tuning and a diffusion policy loss over short-horizon action chunks, using weighted dataset mixing across human video, open-source robot data, and downstream robot data [2412.14803].

Other VPPs modify the latent space or supervision rather than the diffusion paradigm itself. CogRobot fine-tunes independent text-to-flow and flow-to-video models using separate noise-prediction losses, with optical flow functioning as an intermediate variable that grounds language in concrete kinematic cues before RGB video prediction [2505.24156]. MV-VDP applies DDPM training to a joint latent sequence of RGB and heatmap videos and combines the two denoising losses as $L_{\mathrm{diff}}=\lambda L_{\mathrm{vid}}+(1-\lambda)L_{\mathrm{heat}}$, thereby aligning action-relevant 3D state with the representation format of video pretraining [2604.03181]. ViPRA departs from pure denoising by learning a VQ-VAE-style discrete latent action space from passive video using pixel reconstruction, LPIPS, and optical-flow consistency, then pairing cross-entropy pretraining over future frame tokens and latent action tokens with a continuous flow-matching objective during robot adaptation [2511.07732].

A broader probabilistic alternative appears in the autonomous-driving precursor, where future prediction is formulated as conditional variational inference. There, a present distribution and a future distribution parameterize a latent variable $\eta_t$, and training minimizes a KL divergence together with discounted reconstruction losses over segmentation, depth, and optical flow [2003.06409]. The underlying idea is the same: control should be conditioned on a distribution over plausible futures rather than on a static visual summary.

## 4. Action generation, planning, and deployment

In direct-control VPPs, predicted visual dynamics are converted into actions without an explicit world-model/planner split. Hu et al.’s VPP uses predictive visual representations extracted from a single reverse-diffusion step and conditions a diffusion policy head to denoise a short horizon of actions in $K=10$ steps, executing the first few commands before replanning [2412.14803]. PAD samples $k=3$ future steps with DDIM and repeats a receding-horizon loop in which the first predicted action is executed and the denoising process is run again from the updated observation [2411.18179].

Cosmos Policy replaces a conventional action head with latent-frame injection. The $K$-step action sequence is normalized to $[-1,+1]$, flattened, duplicated to fill a latent tensor, and inserted as a placeholder latent frame; at generation time the predicted latent frame is decoded by averaging the duplicated entries and un-normalizing the recovered values [2601.16163]. Because future proprioceptive state, future camera latents, and a scalar value are represented in the same way, the same model supports test-time planning via best-of-$N$ sampling. For each candidate action chunk, the planning model predicts the future state three times and the value five times, producing 15 total value estimates per candidate; a “majority mean” is then used to aggregate success or failure before selecting the action chunk with the highest imagined $V(s')$ [2601.16163].

Cascaded systems use predicted video more explicitly as a plan. In CogRobot, the flow-to-video module predicts future observations, and the final predicted frame is treated as a visual goal for a lightweight goal-conditioned diffusion policy; at test time the system repeatedly re-plans by generating future observations and sampling actions until the robot reaches the predicted frame [2505.24156]. ViPRA instead predicts latent action chunks and decodes them into continuous control via flow matching, integrating the learned flow in ten Euler steps and replanning every $H/2=7$ actions, which yields an effective closed-loop rate of up to 22 Hz [2511.07732]. MV-VDP decodes actions from two channels of predictive structure simultaneously: heatmap peaks are back-projected into 3D trajectories, while a small transformer plus MLP predicts discretized rotations and gripper state [2604.03181].

VPP can also operate as an evaluator rather than a controller. Tseng et al. fine-tune an action-conditional video world model and define a predicted policy value
$$
\hat V(\pi)\approx \frac{1}{M}\sum_{m=1}^M \mathbf{1}\bigl[\text{VLM judges rollout }m\text{ as success}\bigr],
$$
where sampled rollouts are generated autoregressively under a policy and then scored by a vision-language model from subsampled keyframes and a task prompt [2511.11520]. This formulation broadens the scope of VPP from action generation to scalable policy ranking and offline policy assessment.

## 5. Reported empirical performance

Across the cited literature, VPP-style methods are evaluated on long-horizon manipulation, multi-task simulation, bimanual control, dexterous hands, and real-world robotic platforms. Hu et al.’s VPP reports CALVIN ABC→D results of 0.957 on 1 task and 0.750 on 5 tasks, with average length $4.29\pm0.05$ versus 3.35 for the previous state of the art; on MetaWorld it reports 0.682 overall versus 0.574 for GR-1; on real Panda and Xhand manipulation it reports seen and unseen averages of 0.856 and 0.737 for Panda, and 0.749 and 0.605 for Xhand [2412.14803]. PAD reports 72.5% average success on the full MetaWorld benchmark versus 57.4% for the strongest baseline GR-1, and 72% average success on six real-world Panda tasks, rising to 78% for PAD-Depth [2411.18179]. Cosmos Policy reports 98.5% average success on LIBERO, 67.1% on RoboCasa, and 93.6% average direct success on real-world ALOHA, with planning adding a further +12.5 points in the two hardest tasks [2601.16163].

Other implementations show comparable breadth. CogRobot reports 100% ±0.0 on “Put Apple Cabinet,” 36% on “Block Handover,” 70% success on Lift Bag, and 75% on Pull Box [2505.24156]. MV-VDP reports 89.1% average success on Meta-World and 57.1% average success on a real-world setting where BridgeVLA reaches 41.4% and all other listed baselines are at or below 6% [2604.03181]. ViPRA reports 69.8% and 62.5% average success for its discrete and continuous variants on the 4-task SIMPLER suite, and 54.1% full success on real Franka Panda tasks, versus 23.8% for Scratch-FM and 41.8% for $\pi_0$ [2511.07732].

| System | Benchmark setting | Reported result |
|---|---|---|
| Cosmos Policy | LIBERO / RoboCasa / real-world ALOHA | 98.5% / 67.1% / 93.6% avg direct |
| VPP (Hu et al.) | CALVIN ABC→D / MetaWorld | avg length 4.29 ± 0.05 / overall 0.682 |
| PAD | full MetaWorld / real Panda | 72.5% / 72%, PAD-Depth 78% |
| CogRobot | RoboTwin “Put Apple Cabinet” / Lift Bag / Pull Box | 100% ±0.0 / 70% / 75% |
| MV-VDP | Meta-World / real-world | 89.1% / 57.1% |
| ViPRA | SIMPLER / real Franka Panda | 69.8%, 62.5% / 54.1% |

The data regimes are also notable. Cosmos Policy reports 67.1% on RoboCasa with only 50 demos versus 300+ in other methods [2601.16163]. MV-VDP reports 89.1% average success on Meta-World with 5 demo trajectories per task and real-world performance with 10 demos per task [2604.03181]. CogRobot trains its task-specific controller with 100 tele-operated demonstrations per task [2505.24156], while ViPRA adapts to continuous control using only 100 to 200 teleoperated demonstrations [2511.07732]. A plausible implication is that video-based predictive representations can materially reduce the amount of action-labeled data required for downstream control, although the exact savings remain method-dependent.

## 6. Limitations, failure modes, and research directions

The literature repeatedly identifies objective mismatch and rollout degradation as central obstacles. Tseng et al. report hallucinations and object permanence errors, multi-view inconsistencies, “replication” under small gripper displacements, cumulative drift on long-horizon tasks such as Tool Hang, and VLM annotation noise with 65–80% accuracy [2511.11520]. Their mitigations include adding more out-of-distribution rollouts, leveraging larger physically aligned pretraining, and proposing future view-consistency loss, object permanence constraints, and hybrid physics-diffusion models [2511.11520]. PAD emphasizes a different limitation: diffusion sampling is computationally expensive, with a control rate of approximately 1 Hz, and longer horizons may require higher-capacity models or hierarchical planners [2411.18179]. Hu et al.’s VPP likewise notes that the system is heavier than pure one-frame encoders and that inference at 7–10 Hz remains slower than classic PD loops [2412.14803]. CogRobot reports that its diffusion policy is still task-specific and that frame-rate replanning can be computationally heavy; longer horizons beyond 17 frames and richer modalities such as depth and haptics are stated as future directions [2505.24156].

A more recent line argues that standard diffusion training itself under-optimizes the precision-critical dynamics required for manipulation. VAMPO formulates multi-step denoising in a latent video prediction model as an MDP, introduces an Euler Hybrid sampler that injects stochasticity only at the first denoising step, and applies GRPO with a verifiable non-adversarial reward defined by latent $L_1$ deviation and cosine similarity [2603.19370]. Reported effects include an increase in effective rank from 29.28 to 43.88, a rise in normalized ER-ratio from 0.0603 to 0.0814, improvement on CALVIN ABC→D 5-step completion from 74.7% to 83.1%, and an increase in average length from 4.28 to 4.56; on L-CALVIN, average length rises from 5.53 to 6.73 [2603.19370]. This suggests that future VPP research may increasingly separate two questions that early work often conflated: learning a plausible video model and learning a control-adequate video model.

Cosmos Policy provides a related direction through rollout refinement. After deploying the direct policy and collecting successes, failures, and returns, the method fine-tunes a separate planning model with 90% of each batch devoted to world-model and value objectives and 10% to policy, leaving the original checkpoint as the action policy [2601.16163]. In the broader VPP landscape, this indicates an emerging shift from fixed predictive representations toward post-trained predictive representations specialized for planning, evaluation, or robustness.

Source: https://www.emergentmind.com/topics/video-prediction-policy-vpp