Mask2IV: Interaction-Centric Video Generation

• Mask2IV is a framework for interaction-centric video generation that decouples motion planning from appearance synthesis via a two-stage pipeline.
• It first predicts dense mask trajectories from high-level cues and then synthesizes temporally coherent videos conditioned on these trajectories.
• The design supports both natural language and spatial cues, enhancing applications in robot learning, manipulation policy optimization, and affordance reasoning.

Mask2IV is a framework for interaction-centric video generation that emphasizes explicit, fine-grained control of actor–object interactions—such as human hands or robotic manipulators manipulating specified objects—without requiring per-frame mask annotations at inference. Unlike fully text-driven or naively mask-conditioned approaches, Mask2IV decouples motion planning from appearance synthesis via a two-stage pipeline: first predicting the motion trajectories of actors and objects as dense mask sequences from high-level cues, and then synthesizing temporally coherent, physically plausible videos conditioned on those trajectories. This design supports versatile user control, fosters policy learning and affordance reasoning in embodied AI, and rigorously addresses the challenges of generating videos with precise, controllable contact dynamics (Li et al., 3 Oct 2025).

1. Motivation and Problem Formulation

Interaction-centric video generation aims to synthesize short video clips where an articulated actor (e.g., a hand or robot) interacts with an object in a physically meaningful and temporally coherent manner. Such data is essential for downstream embodied intelligence tasks, including robot learning (e.g., imitation from demonstration), manipulation policy optimization (e.g., pick-and-place training), and affordance reasoning (e.g., determining graspable regions). Despite advances in large-scale diffusion models for text-to-video or image-to-video, these methods often fail to accurately render fine-grained actor–object contacts.

Previous approaches are commonly categorized as (a) text-conditioned generation (e.g., DynamiCrafter), which offers broad diversity of outputs but little control over which object is manipulated or the specifics of the actor’s trajectory, and (b) mask-conditioned generation (e.g., InterDyn, CosHand), which delivers explicit spatial supervision but requires impractical, dense frame-wise annotations from users. Mask2IV is designed to eliminate the dependency on such manual mask inputs at test time while retaining explicit, flexible control over the interaction process (Li et al., 3 Oct 2025).

2. Pipeline Architecture

Mask2IV employs a two-stage, decoupled pipeline for interaction-centric video synthesis:

Stage I (Interaction Trajectory Generation):

Inputs consist of an initial RGB image $I$, a user-annotated object mask $M$, and a control cue $C$ (either a text action prompt $T$ for humans or a spatial mask $P$ for robotic targets). The output is a temporally dense sequence of actor-and-object masks $S = \{\hat{M}_t\}_{t=1}^N$, with each $\hat{M}_t\in\{0,1\}^{H\times W\times 3}$ giving per-pixel assignments.

Stage II (Trajectory-conditioned Video Generation):

The same initial image $I$ and trajectory $S$ condition a video generator, yielding $V\in\mathbb{R}^{N\times H\times W\times 3}$ consistent with the predicted mask dynamics.

This decomposition simplifies the modeling of complex interactions, enabling fine-grained control and tractable training.

Stage	Input(s)	Output
Trajectory	$M$0, $M$1, $M$2 ($M$3 or $M$4)	Mask trajectory $M$5
Video	$M$6, predicted $M$7	Video $M$8

3. Mask Trajectory Prediction Module

The trajectory generator is built atop a pretrained image-to-video latent diffusion model. The process is as follows:

• The initial frame $M$9 is encoded via a VAE encoder $C$0 to latent $C$1.
• The object mask $C$2 is color-encoded and mapped to $C$3.
• The control cue $C$4 is either a text prompt $C$5 (encoded using CLIP and injected via cross-attention) or a target-position mask $C$6 (encoded to $C$7 and applied to the final timestep).
• The inputs $C$8 are concatenated, expanded across timesteps, and combined with noise latents $C$9 for the diffusion process.
• The UNet-based backbone uses frozen temporal attention layers to preserve motion priors, while other parameters are fine-tuned.

Mathematically, the predicted mask at timestep $T$0 is given by

$T$1

with losses computed as standard denoising diffusion objectives on the mask latents: $T$2 No additional cross-entropy or explicit temporal consistency is required.

4. Video Generation and Robustness Strategies

The second stage employs a DynamiCrafter-style diffusion network $T$3 conditioned on the predicted mask trajectory. Key features include:

• Mask Conditioning: The trajectory $T$4 is encoded by the VAE; its latent $T$5 is concatenated with the tiled image latent $T$6 and the noise $T$7 at each diffusion timestep.
• Random Dilation/Erosion: With probability $T$8, masks are randomly morphed, enhancing robustness to spatial variations and segmentation imperfections.
• Contact Loss: To enforce accurate rendering at contact regions, a contact mask $T$9 is computed using binary dilation. The denoising loss is re-weighted with a factor $P$0 at contact pixels, as

$P$1

with $P$2.

5. User Interface and Control Modalities

Mask2IV enables several forms of user control:

• Object Specification: The user sketches a single binary mask $P$3 on $P$4 to select the manipulated object.
• Action Description: For human–object interaction, users provide a brief, natural-language prompt (e.g., “a hand picking up a mug”), encoded via CLIP.
• Spatial Cues: For robotic scenarios, users supply a target-position mask $P$5 to designate the final object pose.

Changing $P$6, $P$7, or $P$8 at inference recomputes the trajectory and synthesized video without further annotation, supporting rapid, flexible manipulation scenario prototyping.

6. Datasets and Evaluation Protocols

Two benchmarks are curated:

• Human-Object Interaction (HOI4D): 19,000 egocentric hand-object clips with actions such as grasping or lifting, paired with verbal descriptions.
• Robotic Manipulation (BridgeData V2): 30,000 robot sessions in diverse tabletop settings, with masks extracted via GroundingDINO and SAM2. Object annotation relies on temporal change in IoU; target positions are specified by spatial masks.

Generation quality is assessed via Fréchet Video Distance (FVD), LPIPS, PSNR, and SSIM. Semantic alignment is evaluated with text–video and video–video embedding cosine similarities (T2V-Sim, V2V-Sim), utilizing EgoVLP and ViCLIP for the respective domains.

Method	FVD ↓	LPIPS ↓	PSNR ↑	SSIM ↑	V2V‐Sim ↑	T2V‐Sim ↑
DynamiCrafter	554.5/860.5	0.516/0.375	13.48/14.21	0.553/0.571	0.473/0.867	0.146/0.215
DynamiCrafter‐ft	168.7/197.8	0.206/0.166	20.49/19.80	0.721/0.775	0.814/0.957	0.199/0.223
CosHand	162.9/174.8	0.209/0.123	20.67/21.81	0.725/0.809	0.837/0.969	0.191/0.220
InterDyn	172.4/207.8	0.207/0.145	20.71/21.16	0.730/0.802	0.794/0.955	0.172/0.219
Mask2IV	149.7/155.7	0.178/0.111	21.48/22.30	0.741/0.815	0.847/0.971	0.200/0.220

Mask2IV delivers state-of-the-art scores in all reported metrics across both HOI and robotic benchmarks (Li et al., 3 Oct 2025).

7. Ablation and Implementation Details

Component-wise ablation reveals incremental improvements at each stage, notably with direct latent concatenation, object mask trajectories, random dilation/erosion, and contact-region weighting. For instance, on HOI4D, FVD improves from 157.4 (ControlNet adapter) to 104.6 (full Mask2IV pipeline with contact loss).

Training is performed with two NVIDIA A100 (80 GB) GPUs, 16-frame videos at $P$9 resolution, AdamW optimizer with learning rate $S = \{\hat{M}_t\}_{t=1}^N$0, batch size 8, and 50-step DDIM sampler at inference. Training time is approximately 24 hours per benchmark.

8. Strengths, Limitations, and Prospects

Mask2IV’s primary strengths are its explicit, annotation-free control over both actor and object trajectories, decoupled pipeline facilitating both flexibility and interpretability, and versatile cue modalities (text or spatial), enabling a wide range of task configurations. Limitations include dependency on the quality of mask segmentation during dataset curation, susceptibility to errors under large camera motion or occlusions, and a lack of real-time inference capability.

Potential future work includes joint end-to-end training of both pipeline stages to tighten mask–video consistency, integration of 3D geometry or physical priors, and extension to multi-actor or long-horizon scenarios. This suggests Mask2IV positions itself as a foundational solution for grounding embodied intelligence pipelines with fine-grained, controllable, and semantically meaningful interaction video data (Li et al., 3 Oct 2025).