VIDAR: Video Diffusion for Action Reasoning

• The paper presents a two-stage VIDAR framework that fuses large-scale video diffusion pre-training with a Masked Inverse Dynamics Model for action prediction.
• The approach uses rich spatiotemporal representations to encode physical continuity, object permanence, and motion dynamics, outperforming static image-text methods.
• Empirical results on bimanual robotic manipulation demonstrate improved success rates and data efficiency, highlighting VIDAR’s potential in embodied control systems.

Searching arXiv for the cited VIDAR paper and closely related video-diffusion-for-robotics work. Video Diffusion for Action Reasoning (VIDAR) denotes a family of robot-learning approaches in which a video diffusion model provides the primary spatiotemporal prior for downstream control. In its canonical formulation, VIDAR is a two-stage framework for generalist bimanual manipulation that combines large-scale diffusion-based video pre-training with a Masked Inverse Dynamics Model (MIDM) for action prediction (Feng et al., 17 Jul 2025). In later work, the same term also came to describe a broader paradigm in which policy learning is organized around future video generation, intermediate denoising states, or jointly generated geometric trajectories, with the central claim that video diffusion models encode physical continuity, object permanence, and motion dynamics more effectively than static image-text backbones for manipulation (Ma et al., 11 Mar 2026).

1. Conceptual basis and scope

VIDAR emerged from a specific data bottleneck in robot learning: action-labelled trajectories are scarce, whereas large corpora of videos depicting physical interaction are comparatively abundant. Several papers frame this asymmetry as the main reason to use video diffusion as a control prior. In DiT4DiT, Vision-Language-Action models are described as still relying on representations inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data, whereas generative video models encode rich spatiotemporal structure and implicit physics (Ma et al., 11 Mar 2026). AVID makes a closely related argument from the world-modeling side, noting that action-labelled data is often scarce while pretrained image-to-video diffusion models already generate realistic video, but are not action-conditioned by default (Rigter et al., 2024).

Within this literature, VIDAR has two distinct but related meanings. In the narrow sense, it refers to the framework introduced in "Generalist Bimanual Manipulation via Foundation Video Diffusion Models" (Feng et al., 17 Jul 2025): a foundation video model produces or evaluates future visual trajectories, and MIDM extracts action-relevant regions for inverse dynamics. In the broader sense, VIDAR denotes a modeling strategy in which policy learning is cast as future-trajectory generation followed by control inference. DiT4DiT states this explicitly as the “Broader VIDAR Insight”: policy learning is framed as joint latent trajectory generation, “first predict a plausible future, then invert it into controls” (Ma et al., 11 Mar 2026).

This broader usage places VIDAR at the intersection of policy learning, inverse dynamics, forward dynamics, and world modeling. Unified World Models (UWM) formalize this by coupling action and video diffusion within a single transformer whose independent diffusion timesteps can realize a policy, a forward dynamics model, an inverse dynamics model, or a video generator (Zhu et al., 3 Apr 2025). This suggests that VIDAR is less a single architecture than a design family organized around diffusion-based spatiotemporal reasoning.

2. Canonical two-stage VIDAR architecture

The canonical VIDAR pipeline consists of two stages: large-scale diffusion-based video pre-training and a Masked Inverse Dynamics Model (MIDM) for action prediction (Feng et al., 17 Jul 2025). The pre-training corpus contains 746,533 episodes (approximately 750 K), each with 3 views and language/task metadata, drawn from Agibot (approximately 726 K episodes), RoboMind-Franka, RoboMind-Aloha, and RDT (Feng et al., 17 Jul 2025). The target platform described for adaptation is an Aloha bimanual arm with 3 calibrated views, using 20 minutes of human demonstrations, approximately 232 episodes covering 81 tasks (Feng et al., 17 Jul 2025).

The observation model is explicitly multi-view and metadata-conditioned. Raw streams $\{\mathbf I_t^{(k)}\}_{k=1}^V$ are fused into a unified observation tensor

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$

while the conditioning vector is

$\mathbf l=l_r+l_c+l_t,$

where $l_r$, $l_c$, and $l_t$ encode robot identity, camera type, and task description, respectively (Feng et al., 17 Jul 2025). The video backbone is a U-Net style video diffusion model, Vidu 2.0, with cross-frame attention and spatio-temporal residual blocks (Feng et al., 17 Jul 2025).

At inference, the pre-trained generator $G_\theta$ samples $K$ candidate rollouts

$$\tilde{\mathbf v}_{1:T}^{(i)}=\{\tilde x_1,\dots,\tilde x_T\},$$

and Test-Time Scaling (TTS) ranks them with an external evaluator $q_\eta$, such as CLIP or GPT-4o, selecting $$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$0 (Feng et al., 17 Jul 2025). In the reported implementation, TTS uses $$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$1 samples ranked by GPT-4o, while video generation produces 60 frames at 8 fps in approximately 25 s on 8 × A100 80 GB (Feng et al., 17 Jul 2025).

MIDM then maps generated or real frames to actions. It first predicts a soft spatial mask

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$2

forms $$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$3, and applies an action regressor

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$4

(Feng et al., 17 Jul 2025). The mask network is a U-Net, and the action regressor is a ResNet or MLP; crucially, the masks are learned without pixel-level labels (Feng et al., 17 Jul 2025). The paper reports that the $$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$5 regularizer drives the mask to cover only the robot and manipulated object, which in turn supports generalization to unseen backgrounds (Feng et al., 17 Jul 2025).

3. Mathematical formulations and training objectives

The canonical VIDAR video model is trained in a rectified-flow formulation. The generator learns a velocity field

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$6

with training loss

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$7

and classifier-free guidance

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$8

(Feng et al., 17 Jul 2025). MIDM is trained with

$$\mathbf o_t=\bigoplus_{k=1}^V\gamma_k\,\phi_{r_k}(\mathbf I_t^{(k)}),$$9

with a straight-through estimator through $\mathbf l=l_r+l_c+l_t,$0 (Feng et al., 17 Jul 2025).

A later and more tightly coupled formulation appears in DiT4DiT, which replaces explicit video-to-action staging with a cascaded Video DiT to Action DiT pipeline (Ma et al., 11 Mar 2026). A frozen spatiotemporal VAE encodes past and future observations into latents, and the video transformer denoises noisy future latents via continuous-time flow matching. Instead of using only the final reconstructed frames, DiT4DiT hooks an intermediate hidden activation at a fixed flow timestep:

$\mathbf l=l_r+l_c+l_t,$1

The action transformer then predicts the action velocity field

$\mathbf l=l_r+l_c+l_t,$2

and both modules are trained jointly with

$\mathbf l=l_r+l_c+l_t,$3

under decoupled timesteps $\mathbf l=l_r+l_c+l_t,$4, $\mathbf l=l_r+l_c+l_t,$5, and $\mathbf l=l_r+l_c+l_t,$6 (Ma et al., 11 Mar 2026). This is a direct realization of video diffusion for action reasoning in latent space rather than pixel space.

Parallel unification strategies appear in UWM and UVA. UWM integrates action and video diffusion in one transformer with independently sampled timesteps for each modality, so the same network can represent policy learning, forward dynamics, inverse dynamics, and video generation (Zhu et al., 3 Apr 2025). UVA instead learns a joint video-action latent representation and uses decoupled diffusion heads, allowing action inference to bypass video generation entirely at test time while still benefiting from joint optimization (Li et al., 28 Feb 2025). Together these formulations show that VIDAR can be instantiated as two-stage rollout-plus-inversion, joint latent flow matching, or a shared multimodal denoising transformer.

4. Reported empirical performance

The reported quantitative results span bimanual real robots, simulated manipulation, humanoid control, multi-view policies, and 4D geometry-conditioned transfer.

Method	Setting	Reported result
VIDAR (Feng et al., 17 Jul 2025)	16 tasks: seen tasks/backgrounds, unseen tasks, unseen backgrounds	68.2 / 66.7 / 55.6 success
MIDM vs. ResNet (Feng et al., 17 Jul 2025)	Inverse dynamics generalization	49.0% vs. 24.3% test acc; 0.0308 vs. 0.0430 test $\mathbf l=l_r+l_c+l_t,$7
DiT4DiT (Ma et al., 11 Mar 2026)	LIBERO overall average	98.6%
DiT4DiT (Ma et al., 11 Mar 2026)	RoboCasa-GR1 average	50.8%
PointAction (Tong et al., 2 Jun 2026)	RoboCasa365 ID / OOD-Env / OOD-Task	47.7 / 44.1 / 17.0
MV-VDP (Li et al., 3 Apr 2026)	Meta-World / real world average	89.1% / 57.1%
Vidarc (Feng et al., 19 Dec 2025)	Real-world average; latency	56.0%; 3.03 s vs. 34.3 s

Several papers emphasize not only final success rate but also data efficiency and optimization behavior. The canonical VIDAR paper highlights “only 20 min of demos vs. 27 h in RDT-1B” as a strength of the approach (Feng et al., 17 Jul 2025). DiT4DiT reports that video diffusion serves as a scaling proxy for policy learning, improving sample efficiency by over $\mathbf l=l_r+l_c+l_t,$8 and speeding convergence by up to $\mathbf l=l_r+l_c+l_t,$9; on RoboCasa it requires approximately $l_r$0th the action data to reach comparable performance versus static grounding or FLARE-style baselines (Ma et al., 11 Mar 2026). MV-VDP reports successful real-world manipulation “with only ten demonstration trajectories and without additional pretraining,” while still producing realistic future videos and strong robustness across hyperparameters (Li et al., 3 Apr 2026).

Ablation results also clarify which internal components matter. In VIDAR, removing TTS degrades success from $l_r$1 to $l_r$2, and removing MIDM degrades it to $l_r$3 (Feng et al., 17 Jul 2025). In DiT4DiT, hidden-state extraction performs best at transformer block 18, exactly 1 denoise step is best for $l_r$4 extraction, and joint training yields smooth temporal clusters in t-SNE with silhouette increased by $l_r$5 over decoupled training (Ma et al., 11 Mar 2026). These findings indicate that action-relevant information often resides in intermediate denoising structure rather than only in fully synthesized pixels.

5. Variants and extensions

One line of work adapts pretrained or closed-source video generators into action-conditioned world models. AVID trains a small adapter, rather than modifying backbone weights, and combines the backbone prediction $l_r$6 and adapter prediction $l_r$7 with a learned mask,

$l_r$8

so that action-relevant regions can be overridden while static background detail is preserved (Rigter et al., 2024). Vid2World takes a different route: it causalizes bidirectional video diffusion by replacing temporal attention with causal attention, converting temporal convolutions to causal convolutions, and training under diffusion forcing with per-frame noise levels, then adds classifier-free action guidance for controllable autoregressive rollouts (Huang et al., 20 May 2025).

A second line of work pursues tighter video-action coupling. UVA learns a joint video-action latent representation and uses masked-input training so that one model can perform policy learning, video generation, forward dynamics, inverse dynamics, and combined planning-plus-policy (Li et al., 28 Feb 2025). UWM similarly couples both modalities inside a unified transformer, but emphasizes independent diffusion timesteps and register tokens for information exchange between action and video substreams (Zhu et al., 3 Apr 2025). CoVAR preserves a pretrained video DiT, adds a parallel dedicated action DiT, introduces Bridge Attention for bidirectional cross-modal interaction, and appends an action refinement module for low-resolution datasets such as Libero90 (Yang et al., 17 Dec 2025).

A third line adds explicit spatial structure to reduce the ambiguity of RGB-only grounding. PointAction fine-tunes a foundation video model to jointly predict future RGB and dense pointmaps $l_r$9, then conditions a diffusion action decoder on robot-mask 3D points, using an explicit 4D point interface as an embodiment-agnostic action representation (Tong et al., 2 Jun 2026). MV-VDP instead predicts multi-view RGB videos and multi-view heatmap videos from a colored point-cloud state and end-effector pose, aligning video pretraining with action finetuning while making the predicted future directly interpretable for human oversight (Li et al., 3 Apr 2026).

A fourth line targets deployment constraints. Video2Act extracts structure-aware and motion-aware cues from a pretrained VDM using Sobel filtering on high-resolution features and FFT-based high-pass filtering on long-horizon low-resolution features, then feeds compressed tokens into a DiT action head in an asynchronous dual-system design where the VDM is the slow System 2 and the action model is the fast System 1 (Jia et al., 2 Dec 2025). Vidarc pushes this further into closed-loop control by combining autoregressive embodied video diffusion, MIDM, causal training, and KV-cache re-prefilling, thereby reducing end-to-end per-chunk latency from $l_c$0 s to $l_c$1 s while improving real-world average success from $l_c$2 to $l_c$3 (Feng et al., 19 Dec 2025).

6. Limitations, misconceptions, and open directions

The original two-stage VIDAR formulation has several explicit limitations. It is open-loop, so errors accumulate over long horizons; video inference is slow, at approximately $l_c$4 s for a $l_c$5 s rollout; and TTS depends on an external evaluator such as GPT-4o (Feng et al., 17 Jul 2025). These constraints are important because they delimit where generated video can be used directly as a control substrate and where it remains primarily a planning or representation tool.

A recurring misconception is that video diffusion alone functions as the controller. The surveyed methods indicate otherwise. In practice, video diffusion is paired with MIDM, an action diffusion transformer, an inverse dynamics model, classifier-free action guidance, or a point-based action decoder rather than used as a standalone policy (Feng et al., 17 Jul 2025). The same literature also shows that RGB-only rollouts under-specify metric 3D motion, contact geometry, and fine-grained spatial constraints; PointAction states this directly as the reason for introducing explicit point-based 4D modeling (Tong et al., 2 Jun 2026). CoVAR identifies a related issue from another angle: monocular generation lacks explicit 3D geometry, even when joint video-action diffusion is strong (Yang et al., 17 Dec 2025).

Compute remains a major systems bottleneck. Vid2World reports post-training costs of approximately 7 days on 4 A100s for 100 k steps, and CoVAR describes its DiTs as compute-heavy at 1.4 B total parameters (Huang et al., 20 May 2025). Closed-loop variants reduce some of the latency problem, but do not remove the underlying cost of high-capacity diffusion backbones (Feng et al., 19 Dec 2025). Open problems named directly in the literature include receding-horizon closed-loop control, hierarchical diffusion for longer horizons, force/torque or tactile conditioning, mobile manipulators, larger-scale causal adapters, and longer temporal memory (Feng et al., 17 Jul 2025).

A plausible implication is that VIDAR is evolving from a two-stage bimanual manipulation framework into a broader class of embodied generative control systems. The trajectory across these papers is consistent: future video prediction remains central, but increasingly serves as an intermediate representation for masked inverse dynamics, latent flow matching, 3D point-trajectory grounding, multi-view heatmap decoding, or low-latency closed-loop action generation rather than as an end product in itself (Ma et al., 11 Mar 2026).