MTV-World: Multi-View Visuomotor Control

• MTV-World is an embodied world modeling framework that uses multi-view trajectory videos to translate low-level robotic actions into visually realistic outcomes.
• It achieves geometric consistency by rendering explicit 2D end-effector trajectories across synchronized camera views, thereby mitigating 2D projection ambiguities.
• Evaluations on dual-arm manipulation tasks show state-of-the-art improvements with over 30 points gain in the Jaccard index and reduced FID/FVD scores.

MTV-World is an embodied world modeling framework that introduces Multi-View Trajectory-Video control for precise and consistent visuomotor prediction in robotics. It addresses a fundamental limitation in prior embodied world models: the inability to translate low-level action (e.g., joint angles or joint-space velocities) into visually and physically realistic arm and object interactions in video-prediction rollouts. MTV-World leverages explicit image-space end-effector trajectory renderings, synchronized across multiple calibrated views, as the control signal for visuomotor prediction, enabling tight geometric alignment between predicted actions and observable outcomes in generated frames. This strategy is quantitatively validated in complex dual-arm robot manipulation setups, yielding state-of-the-art consistency and precision compared to prior single-view, action-conditioned approaches (Su et al., 17 Nov 2025).

1. Motivation and Core Problem

Traditional embodied world models typically condition video prediction on low-level robot actions such as sequences of joint angles $q_t$ or actuator commands. In articulated robots or complex manipulators, these low-level actions are high-dimensional, highly coupled, and inherently nonlinear when projected into the observation domain via the kinematic chain. Prediction errors in joint space are amplified when mapped to the rendered image plane, often causing implausible, jittery, or non-physical arm trajectories in generated video. As a result, interaction-critical events—such as grasps, pushes, or stacking actions—cannot be reliably simulated, and predicted outcomes become inconsistent with real-world physics. This decoupling undermines the credibility of model-based planning or sim-to-real learning pipelines relying on such world models (Su et al., 17 Nov 2025).

MTV-World replaces direct low-level action conditioning with a trajectory-video-based representation: rather than operating in the joint or end-effector pose space, it renders the robot’s planned end-effector path as an explicit 2D curve in each video frame, using the camera’s intrinsic and extrinsic matrices. This “trajectory video” tightly aligns the control signal with the observation domain, guaranteeing geometric consistency between commanded actions and pixel-space forecasts across all views.

2. Mathematical Formulation of Multi-View Trajectory Videos

The central control signal in MTV-World is the multi-view trajectory video, computed as follows:

• For each timestep $t$, forward kinematics produce the 3D Cartesian pose $p_w = (x_w, y_w, z_w)$ from joint angles $q_t$ via Denavit–Hartenberg parameters.
• For each calibrated camera with extrinsic parameters $(R, t)$ and intrinsic matrix $$K = \begin{bmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{bmatrix}$$, the 3D world point is projected into camera coordinates by

$p_c = R^{\top}(p_w - t) = (x_c, y_c, z_c)$

• The pixel coordinates are then:

$$u = f_x \frac{x_c}{z_c} + c_x, \quad v = f_y \frac{y_c}{z_c} + c_y$$

• These $(u_t, v_t)$ points are rendered on each frame as a glowing path, with trailing points faded to indicate temporal evolution, producing a trajectory video $$X_v^{\text{traj}} \in \mathbb{R}^{(1+T)\times 3 \times h \times w}$$ per view $v$, for $V$ different views. The synchronized set $\{X_1^{\text{traj}}, \ldots, X_V^{\text{traj}}\}$ forms the multi-view trajectory video control signal (Su et al., 17 Nov 2025).

This approach preserves the geometric fidelity of the control signal in the observation domain and compensates for 2D spatial information loss (projection ambiguity) by aggregating information across views.

3. MTV-World Model Architecture

MTV-World couples multi-view trajectory videos and visual observations in a multimodal conditional generative framework with the following components:

• Inputs: Multi-view reference frames $X^{\text{video}}$, multi-view trajectory videos $X^{\text{traj}}$, and a text-based instruction $l$ if used.
• Encoder: A shared VAE encoder $Enc(\cdot)$ maps both $X^{\text{video}}$ and $X^{\text{traj}}$ to per-view latent sequences (dim $V \times c' \times (1+T) \times h' \times w'$). The first frame from each view is further adapted as a reference latent.
• Multimodal Context: Each reference frame is embedded using CLIP, and instructions are embedded (umT5). These embeddings are concatenated and projected for cross-attention conditioning.
• Latent Fusion and Diffusion Backbone: Latents are concatenated temporally for all views and passed into a DiT (Diffusion Transformer) backbone. Cross-attention layers integrate appearance priors and instruction semantics.
• Decoder: Decoded future-frame latents are mapped back to output video frames per view.

Optional components include a cross-view consistency loss enforced via latent warping (e.g., geometrically transforming one predicted latent into the other view and penalizing mismatch).

4. Training Objectives and Evaluation

The training pipeline involves multiple objectives:

• VAE Reconstruction Loss: $$L_{\text{VAE}} = \mathbb{E}_{I\sim\text{real}}\left[\|I - Dec(Enc(I))\|_1\right] + \beta \cdot \text{KL}(q(z|I) \,\|\, p(z))$$
• Diffusion Denoising Loss: Standard score-matching: $$L_{\text{diff}} = \mathbb{E}_{t,\epsilon \sim \mathcal{N}(0,1)}\left[\|\epsilon - \epsilon_\theta(z_t, c)\|_2^2\right]$$, with $z_t$ the noisy latent and $c$ the conditioning signal.
• Optional Multi-View Consistency Loss: $$L_\text{consist} = \sum_{v \neq v'} \| W_{v \rightarrow v'}(X^{\text{pred}}_v) - X^{\text{pred}}_{v'} \|_1$$.
• Object-Centric Reconstruction: Weighted inside object masks $M$: $$L_\text{mask} = \mathbb{E}[ \| M \odot (I - \hat{I}) \|_1 ]$$.
• Total Loss: $$L_\text{total} = L_{\text{VAE}} + L_{\text{diff}} + \lambda_1 L_{\text{consist}} + \lambda_2 L_\text{mask}$$
• Object Interaction Accuracy: Jaccard Index per frame and per-video, using object and arm masks generated via an auto-evaluation pipeline leveraging pretrained vision-language and referring video object segmentation models.

5. Auto-Evaluation and Experimental Results

The auto-evaluation system operates without manual annotation:

• A VLM generates object descriptions from the first predicted frame.
• These descriptions are provided to a referring video object segmentation (RVOS) model to segment both ground-truth and predicted videos.
• Per-frame Jaccard indices ($$J_t = \left| M^{\text{pred}}_t \cap M^{\text{gt}}_t \right| / \left| M^{\text{pred}}_t \cup M^{\text{gt}}_t \right|$$) are computed and averaged across time.

MTV-World was validated on a dual-arm YAM robot with 6-DoF manipulators and two calibrated RGBD cameras, over 1,492 real-world complex manipulation trials. Metrics included FID, FVD, and the Jaccard index $J$.

• Performance: Over single-view Policy2Vec baselines, MTV-World improves the Jaccard index by over 30 points on the secondary view and lowers FID/FVD by 2–8 points under both maskless and mask-focused losses.
• Ablations: Multi-view inputs prevent severe view-dependent collapse observed in single-view models. Object-masked losses add another 2–3 points of $J$.
• Zero-shot: The model generalizes to held-out tasks, unseen physical objects, and novel viewpoints.

6. Advantages, Design Insights, and Limitations

Key advantages:

• Multi-view trajectory video input mitigates 2D projection ambiguity by providing complementary perspectives, ensuring that the complete 3D path of the manipulator is recoverable through fusion.
• Geometric grounding leads to greater consistency in predicting arm–object interactions, reliably modeling contact events crucial for physical reasoning in robotic planning.
• The multi-view design allows a single model to achieve high consistency across all observational views, rather than being specialized per-camera.

Limitations:

• MTV-World requires accurate camera calibration. Errors in extrinsic/intrinsic parameters or segmentation can lead to degraded control fidelity.
• By operating in the 2D image domain, explicit depth variation is not directly modeled beyond what is recoverable through the 2D path and multi-view fusion.
• The system is computationally intensive, with full two-view diffusion rollouts at 81 frames.

Future directions include augmenting trajectory control with explicit 3D representations (e.g., depth maps or point clouds), adding closed-loop feedback for adaptive control, scaling to more cameras or dynamic viewpoints, and unifying with reinforcement or model-based planning in the latent space.

7. Impact and Broader Context in Multi-View Visuomotor Control

MTV-World represents a paradigm shift away from low-level, kinematics-centric control toward visually grounded, observation-aligned world modeling with explicit geometric fidelity. By directly mapping end-effector trajectories into multi-view, temporally synchronized image-space controls, it enables world models to predict physical interaction with high precision and robustness. This framework is well-aligned with recent multi-view, camera-control, and geometry-grounded video generation literature, which seek to enforce explicit spatial consistency and realistic cross-view predictions in both robotic and general video synthesis contexts (Su et al., 17 Nov 2025). The approach is broadly applicable to scenarios requiring compositional visuomotor planning, multi-camera simulation, and real-to-sim transfer in complex robotics and embodied AI tasks.