Video Prediction Policy (VPP)

• Video Prediction Policy is a framework that uses predicted video dynamics to encode essential motion cues for effective control and planning in robotics.
• It employs diffusion-based models to jointly generate visual predictions and corresponding actions, enhancing long-horizon manipulation and efficiency.
• Empirical studies report improved success rates and reduced action-label requirements compared to traditional static vision-based controllers.

Searching arXiv for the cited VPP-related papers to ground the article in current literature. Video Prediction Policy (VPP) denotes a family of policy-learning frameworks in which video prediction is not merely an auxiliary perceptual task but a primary mechanism for control, planning, or policy evaluation. In contemporary robotics, the term appears explicitly in "Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations" (Hu et al., 2024), while closely related formulations include joint image-action denoising (Guo et al., 2024), flow-based bimanual video prediction (Fan et al., 30 May 2025), latent-injection diffusion policies and planners (Kim et al., 22 Jan 2026), multi-view video action models (Li et al., 3 Apr 2026), latent-action pretraining from actionless video (Routray et al., 11 Nov 2025), and action-conditional video world models for policy evaluation (Tseng et al., 14 Nov 2025). A historical precursor is the probabilistic future-prediction-and-control framework for autonomous driving, which jointly predicts semantics, depth, optical flow, and control from video (Hu et al., 2020).

1. Conceptual basis and lineage

The central premise of VPP is that predictive video models encode dynamics that static visual encoders often omit. Hu et al. argue that encoders pre-trained with single-image reconstruction or two-image contrastive learning tend to capture static information while neglecting dynamic aspects vital for embodied tasks, and therefore condition action learning on predicted future representations inside a video diffusion model (Hu et al., 2024). PAD makes an even tighter coupling: image prediction and robotic action are treated as highly correlated because they share the same underlying dynamics of the physical world, so both are generated within a joint denoising process (Guo et al., 2024). Cosmos Policy extends this logic by adapting a large pretrained video model into a robot policy that also predicts future state images and values, thereby combining policy learning, world modeling, and planning in one latent diffusion framework (Kim et al., 22 Jan 2026).

This research direction did not emerge ex nihilo. In autonomous driving, future prediction from video was already linked to control through a conditional-variational architecture that modeled a present distribution, a future distribution, probabilistic predictions of semantics, geometry, and motion, and a learned driving policy (Hu et al., 2020). This suggests that VPP is best understood as a design principle: action selection is grounded in predictive visual dynamics rather than in static per-frame representations alone.

2. Architectural patterns

A common pattern is a two-stage architecture in which a video predictor supplies predictive features to a downstream action generator. In Hu et al.’s VPP, a text-guided video prediction model based on Stable Video Diffusion is fine-tuned on human and robot manipulation data, but the action policy does not fully denoise future video at inference time. Instead, it performs a single reverse-diffusion step, extracts intermediate U-Net features, aggregates them with a Video-Former, and conditions a diffusion policy head on the resulting predictive visual representations (Hu et al., 2024). ViPRA uses a related decomposition, but its intermediate object is a motion-centric latent action space learned from passive video; a video-LLM predicts future visual observations and latent actions, and a chunked flow-matching decoder maps those latent actions to robot-specific continuous action sequences (Routray et al., 11 Nov 2025).

A second pattern is joint modeling of visual prediction and action generation inside one denoising backbone. PAD defines parallel forward diffusion processes for image and action latents and uses a Diffusion Transformer to predict both modalities jointly; missing modalities can be handled by masked attention, which permits co-training on video data without action labels (Guo et al., 2024). Cosmos Policy pushes this unification further by introducing no new MLP heads or dynamics modules at all: robot proprioception, an action chunk, future state frames, and a scalar value are each encoded as additional latent frames and interleaved with the image latents, so the pretrained diffusion transformer models them alongside images (Kim et al., 22 Jan 2026). MV-VDP similarly keeps video generation central, but augments a pretrained single-view Video Foundation Model with a view-attention layer and jointly predicts multi-view RGB videos and heatmap videos, after which a dedicated action decoder reconstructs 3D trajectories, rotation, and gripper commands (Li et al., 3 Apr 2026).

A third pattern is a cascaded planner-controller design. CogRobot treats bimanual manipulation as video prediction, first fine-tuning a text-to-flow model and then a flow-to-video model derived from a pretrained text-to-video backbone; a lightweight goal-conditioned diffusion policy then converts the predicted visual plan into joint commands (Fan et al., 30 May 2025). In such systems, video prediction specifies an intended future, while a smaller controller handles embodiment-specific action realization.

3. Probabilistic formulations and training objectives

Most recent VPP variants inherit their learning dynamics from diffusion modeling. Cosmos Policy adopts a latent-diffusion formulation in which the clean latent sequence $z_0$ contains both image and non-image frames, and fine-tunes the denoiser with an EDM-style score-matching objective,

$$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$

while keeping the transformer architecture unchanged (Kim et al., 22 Jan 2026). PAD defines separate forward diffusion processes over image and action latents and trains a multimodal denoiser with

$$L(\theta)=\lambda_I L_{\mathrm{diff}}^I+\lambda_A L_{\mathrm{diff}}^A+\lambda_E L_{\mathrm{diff}}^E,$$

where extra modalities such as depth can be appended directly to the token sequence (Guo et al., 2024). Hu et al.’s VPP also splits training into video-model fine-tuning and a diffusion policy loss over short-horizon action chunks, using weighted dataset mixing across human video, open-source robot data, and downstream robot data (Hu et al., 2024).

Other VPPs modify the latent space or supervision rather than the diffusion paradigm itself. CogRobot fine-tunes independent text-to-flow and flow-to-video models using separate noise-prediction losses, with optical flow functioning as an intermediate variable that grounds language in concrete kinematic cues before RGB video prediction (Fan et al., 30 May 2025). MV-VDP applies DDPM training to a joint latent sequence of RGB and heatmap videos and combines the two denoising losses as $$L_{\mathrm{diff}}=\lambda L_{\mathrm{vid}}+(1-\lambda)L_{\mathrm{heat}}$$, thereby aligning action-relevant 3D state with the representation format of video pretraining (Li et al., 3 Apr 2026). ViPRA departs from pure denoising by learning a VQ-VAE-style discrete latent action space from passive video using pixel reconstruction, LPIPS, and optical-flow consistency, then pairing cross-entropy pretraining over future frame tokens and latent action tokens with a continuous flow-matching objective during robot adaptation (Routray et al., 11 Nov 2025).

A broader probabilistic alternative appears in the autonomous-driving precursor, where future prediction is formulated as conditional variational inference. There, a present distribution and a future distribution parameterize a latent variable $\eta_t$, and training minimizes a KL divergence together with discounted reconstruction losses over segmentation, depth, and optical flow (Hu et al., 2020). The underlying idea is the same: control should be conditioned on a distribution over plausible futures rather than on a static visual summary.

4. Action generation, planning, and deployment

In direct-control VPPs, predicted visual dynamics are converted into actions without an explicit world-model/planner split. Hu et al.’s VPP uses predictive visual representations extracted from a single reverse-diffusion step and conditions a diffusion policy head to denoise a short horizon of actions in $K=10$ steps, executing the first few commands before replanning (Hu et al., 2024). PAD samples $k=3$ future steps with DDIM and repeats a receding-horizon loop in which the first predicted action is executed and the denoising process is run again from the updated observation (Guo et al., 2024).

Cosmos Policy replaces a conventional action head with latent-frame injection. The $K$-step action sequence is normalized to $[-1,+1]$, flattened, duplicated to fill a latent tensor, and inserted as a placeholder latent frame; at generation time the predicted latent frame is decoded by averaging the duplicated entries and un-normalizing the recovered values (Kim et al., 22 Jan 2026). Because future proprioceptive state, future camera latents, and a scalar value are represented in the same way, the same model supports test-time planning via best-of-$N$ sampling. For each candidate action chunk, the planning model predicts the future state three times and the value five times, producing 15 total value estimates per candidate; a “majority mean” is then used to aggregate success or failure before selecting the action chunk with the highest imagined $$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$0 (Kim et al., 22 Jan 2026).

Cascaded systems use predicted video more explicitly as a plan. In CogRobot, the flow-to-video module predicts future observations, and the final predicted frame is treated as a visual goal for a lightweight goal-conditioned diffusion policy; at test time the system repeatedly re-plans by generating future observations and sampling actions until the robot reaches the predicted frame (Fan et al., 30 May 2025). ViPRA instead predicts latent action chunks and decodes them into continuous control via flow matching, integrating the learned flow in ten Euler steps and replanning every $$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$1 actions, which yields an effective closed-loop rate of up to 22 Hz (Routray et al., 11 Nov 2025). MV-VDP decodes actions from two channels of predictive structure simultaneously: heatmap peaks are back-projected into 3D trajectories, while a small transformer plus MLP predicts discretized rotations and gripper state (Li et al., 3 Apr 2026).

VPP can also operate as an evaluator rather than a controller. Tseng et al. fine-tune an action-conditional video world model and define a predicted policy value

$$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$2

where sampled rollouts are generated autoregressively under a policy and then scored by a vision-LLM from subsampled keyframes and a task prompt (Tseng et al., 14 Nov 2025). This formulation broadens the scope of VPP from action generation to scalable policy ranking and offline policy assessment.

5. Reported empirical performance

Across the cited literature, VPP-style methods are evaluated on long-horizon manipulation, multi-task simulation, bimanual control, dexterous hands, and real-world robotic platforms. Hu et al.’s VPP reports CALVIN ABC→D results of 0.957 on 1 task and 0.750 on 5 tasks, with average length $$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$3 versus 3.35 for the previous state of the art; on MetaWorld it reports 0.682 overall versus 0.574 for GR-1; on real Panda and Xhand manipulation it reports seen and unseen averages of 0.856 and 0.737 for Panda, and 0.749 and 0.605 for Xhand (Hu et al., 2024). PAD reports 72.5% average success on the full MetaWorld benchmark versus 57.4% for the strongest baseline GR-1, and 72% average success on six real-world Panda tasks, rising to 78% for PAD-Depth (Guo et al., 2024). Cosmos Policy reports 98.5% average success on LIBERO, 67.1% on RoboCasa, and 93.6% average direct success on real-world ALOHA, with planning adding a further +12.5 points in the two hardest tasks (Kim et al., 22 Jan 2026).

Other implementations show comparable breadth. CogRobot reports 100% ±0.0 on “Put Apple Cabinet,” 36% on “Block Handover,” 70% success on Lift Bag, and 75% on Pull Box (Fan et al., 30 May 2025). MV-VDP reports 89.1% average success on Meta-World and 57.1% average success on a real-world setting where BridgeVLA reaches 41.4% and all other listed baselines are at or below 6% (Li et al., 3 Apr 2026). ViPRA reports 69.8% and 62.5% average success for its discrete and continuous variants on the 4-task SIMPLER suite, and 54.1% full success on real Franka Panda tasks, versus 23.8% for Scratch-FM and 41.8% for $$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$4 (Routray et al., 11 Nov 2025).

System	Benchmark setting	Reported result
Cosmos Policy	LIBERO / RoboCasa / real-world ALOHA	98.5% / 67.1% / 93.6% avg direct
VPP (Hu et al.)	CALVIN ABC→D / MetaWorld	avg length 4.29 ± 0.05 / overall 0.682
PAD	full MetaWorld / real Panda	72.5% / 72%, PAD-Depth 78%
CogRobot	RoboTwin “Put Apple Cabinet” / Lift Bag / Pull Box	100% ±0.0 / 70% / 75%
MV-VDP	Meta-World / real-world	89.1% / 57.1%
ViPRA	SIMPLER / real Franka Panda	69.8%, 62.5% / 54.1%

The data regimes are also notable. Cosmos Policy reports 67.1% on RoboCasa with only 50 demos versus 300+ in other methods (Kim et al., 22 Jan 2026). MV-VDP reports 89.1% average success on Meta-World with 5 demo trajectories per task and real-world performance with 10 demos per task (Li et al., 3 Apr 2026). CogRobot trains its task-specific controller with 100 tele-operated demonstrations per task (Fan et al., 30 May 2025), while ViPRA adapts to continuous control using only 100 to 200 teleoperated demonstrations (Routray et al., 11 Nov 2025). A plausible implication is that video-based predictive representations can materially reduce the amount of action-labeled data required for downstream control, although the exact savings remain method-dependent.

6. Limitations, failure modes, and research directions

The literature repeatedly identifies objective mismatch and rollout degradation as central obstacles. Tseng et al. report hallucinations and object permanence errors, multi-view inconsistencies, “replication” under small gripper displacements, cumulative drift on long-horizon tasks such as Tool Hang, and VLM annotation noise with 65–80% accuracy (Tseng et al., 14 Nov 2025). Their mitigations include adding more out-of-distribution rollouts, leveraging larger physically aligned pretraining, and proposing future view-consistency loss, object permanence constraints, and hybrid physics-diffusion models (Tseng et al., 14 Nov 2025). PAD emphasizes a different limitation: diffusion sampling is computationally expensive, with a control rate of approximately 1 Hz, and longer horizons may require higher-capacity models or hierarchical planners (Guo et al., 2024). Hu et al.’s VPP likewise notes that the system is heavier than pure one-frame encoders and that inference at 7–10 Hz remains slower than classic PD loops (Hu et al., 2024). CogRobot reports that its diffusion policy is still task-specific and that frame-rate replanning can be computationally heavy; longer horizons beyond 17 frames and richer modalities such as depth and haptics are stated as future directions (Fan et al., 30 May 2025).

A more recent line argues that standard diffusion training itself under-optimizes the precision-critical dynamics required for manipulation. VAMPO formulates multi-step denoising in a latent video prediction model as an MDP, introduces an Euler Hybrid sampler that injects stochasticity only at the first denoising step, and applies GRPO with a verifiable non-adversarial reward defined by latent $$\mathcal{L}_{\mathrm{diff}}= \mathbb{E}_{z_0,c,n\sim\mathcal{N}(0,\sigma^2 I)} \left\|D_\theta(z_0+n;\sigma,c)-z_0\right\|^2,$$5 deviation and cosine similarity (Ge et al., 19 Mar 2026). Reported effects include an increase in effective rank from 29.28 to 43.88, a rise in normalized ER-ratio from 0.0603 to 0.0814, improvement on CALVIN ABC→D 5-step completion from 74.7% to 83.1%, and an increase in average length from 4.28 to 4.56; on L-CALVIN, average length rises from 5.53 to 6.73 (Ge et al., 19 Mar 2026). This suggests that future VPP research may increasingly separate two questions that early work often conflated: learning a plausible video model and learning a control-adequate video model.

Cosmos Policy provides a related direction through rollout refinement. After deploying the direct policy and collecting successes, failures, and returns, the method fine-tunes a separate planning model with 90% of each batch devoted to world-model and value objectives and 10% to policy, leaving the original checkpoint as the action policy (Kim et al., 22 Jan 2026). In the broader VPP landscape, this indicates an emerging shift from fixed predictive representations toward post-trained predictive representations specialized for planning, evaluation, or robustness.