Video-Point Model (VPM) Overview

• Video-Point Model (VPM) is a camera-conditioned, multi-view world model that simultaneously predicts future RGB videos and persistent 3D trajectories of physical points, enhancing both temporal and cross-view consistency and making it suitable for tasks like 4D scene modeling and robot control.
• The model utilizes a mixture-of-transformers architecture, combining video prediction with persistent point-trajectory forecasting, improving cross-view consistency by interconnected camera viewpoints, and establishes robustly correlated physical trajectories across time.
• VPM employs a specialized training curriculum to improve tenacity for point estimation and video prediction along with emphasizing trajectories improving accuracy in practical prediction and generalization for extended utilization across artificial and authentic environments.

Video-Point Model (VPM) is a camera-conditioned, multi-view world model for forecasting future RGB videos and persistent three-dimensional trajectories of physical scene points. Given synchronized multi-view video histories, camera parameters, a language instruction, and point queries selected from observed frames, VPM jointly predicts future videos from requested viewpoints and the 3D motion of the queried points. Its central premise is that persistent 3D trajectories provide a stronger constraint on temporal and cross-view consistency than per-frame depth or point-map supervision alone. The model is built on the Cosmos 3 / Cosmos3-Nano mixture-of-transformers architecture and extends video forecasting toward 4D scene modeling and robot control (Jeon et al., 20 Oct 2026).

1. Problem formulation and representational basis

Let $\mathcal V_{\mathrm{in}}$ denote the input cameras and $\mathcal V_{\mathrm{out}}$ the requested output cameras. For camera $v$ and time $\tau$, an RGB frame is represented as $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$, with camera intrinsic matrix $\mathbf K^v$ and camera-to-world pose $\mathbf T_\tau^v\in SE(3)$. Given an observed history of $h'$ frames, camera parameters, and a language instruction $\ell$, VPM predicts the next $h$ frames,

$\mathcal V_{\mathrm{out}}$0

together with trajectories for $\mathcal V_{\mathrm{out}}$1 points selected from the observed history.

A point query is

$\mathcal V_{\mathrm{out}}$2

where $\mathcal V_{\mathrm{out}}$3 is an observed time, $\mathcal V_{\mathrm{out}}$4 is the source camera, and $\mathcal V_{\mathrm{out}}$5 is the image location. The corresponding physical point has three-dimensional position $\mathcal V_{\mathrm{out}}$6 at future time $\mathcal V_{\mathrm{out}}$7. Trajectories are expressed in the coordinate frame of a reference camera $\mathcal V_{\mathrm{out}}$8 at the first observed time $\mathcal V_{\mathrm{out}}$9.

Conceptually, VPM models the conditional distribution

$v$0

The model therefore combines video prediction with point-trajectory forecasting. A trajectory preserves the identity of one physical point across time and cameras, unlike independent per-frame depth estimates or unrelated two-dimensional tracks. If a point moves from $v$1 to $v$2, its three-dimensional displacement is represented directly by

$v$3

This shared identity constrains generated appearances in different cameras to correspond to the same evolving three-dimensional event. It also allows a point to disappear from one camera while remaining visible in another.

The principal distinction between VPM and conventional video world models is representational. A video-only model can produce individually plausible frames while allowing object identity, geometry, and motion to disagree across views. VPM instead forecasts RGB observations and persistent 3D trajectories jointly. Its point supervision is intended to anchor appearance changes to a common description of scene motion.

2. Camera-conditioned multi-view architecture

VPM starts from Cosmos3-Nano, whose mixture-of-transformers architecture contains a reasoner and a generator. The reasoner processes language and high-level conditioning, whereas the generator denoises future video tokens. The reasoner remains frozen during VPM training. The generation pathway is modified for multi-view input, and a trajectory head is attached to intermediate generator representations (Jeon et al., 20 Oct 2026).

Latent video tokens from all cameras are placed on a common temporal grid containing clean observed-history tokens, noisy future tokens, and tokens associated with requested output cameras. Attention can therefore operate across time, image space, and camera views. A point visible in one camera can influence the prediction for another camera, including when it is temporarily occluded in the second view.

For every video token, VPM computes the camera ray corresponding to the represented image region. In the reference frame, the ray is encoded with Plücker coordinates,

$v$4

where $v$5 is the ray origin and $v$6 is the unit ray direction. A lightweight encoder $v$7 produces the camera embedding

$v$8

The embedding enters each transformer layer additively,

$v$9

and through camera-dependent query and key rotations,

$\tau$0

VPM also introduces a view-specific axis into rotary positional encoding. If $\tau$1, $\tau$2, and $\tau$3 denote temporal and spatial token coordinates, and $\tau$4 denotes the camera index, the rotary phase is

$\tau$5

The view frequencies are initialized to zero, allowing the model initially to recover the pretrained three-dimensional positional encoding. The camera-conditioning mechanism consequently consists of additive Plücker embeddings, camera-dependent query/key rotations, and a learnable view axis in rotary positional encoding. Removing these components substantially harms performance, particularly for moving-camera urban scenes.

Rectified-flow video generation

Let $\tau$6 be clean future-video latent tokens and $\tau$7. At noise level $\tau$8, VPM forms

$\tau$9

The generator denoiser $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$0 predicts the rectified-flow velocity $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$1. The video-generation objective is

$\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$2

where $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$3 contains the observed video history, camera conditions, language, and other conditioning tokens. Clean conditioning tokens are not included in the generation loss. At inference, observed-history tokens remain clean, whereas future tokens are initialized from Gaussian noise at $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$4 and iteratively denoised to $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$5.

3. Persistent trajectory prediction

Each queried point retains one identity across time, camera views, and the observed-to-future transition. The trajectory head predicts, for every view and timestep:

• a two-dimensional image position $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$6;
• a depth $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$7;
• a confidence $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$8.

The head operates on query and key features from an intermediate generator self-attention layer. In the implementation, these features are taken from layer 18 and reshaped into per-view latent grids $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$9 and $\mathbf K^v$0. For each queried point, a source query feature is sampled and correlated with key features around the current point estimate in every view and timestep.

A transformer refiner updates point estimates over $\mathbf K^v$1 iterations, with the default $\mathbf K^v$2. Its attention is factorized over spatial positions, time, and camera views. This allows refinement to integrate local image evidence, temporal motion patterns, and cross-view correspondence. Cross-view attention is particularly important; removing it produces the largest degradation among the refiner ablations.

Predicted image positions and depths are lifted into the reference coordinate system. Let

$\mathbf K^v$3

be the camera-to-reference transformation at time $\mathbf K^v$4, with rotation $\mathbf K^v$5 and translation $\mathbf K^v$6. The predicted point is

$\mathbf K^v$7

Predictions from multiple views thus describe the same physical point in a shared reference coordinate system.

Trajectory supervision

The trajectory loss is

$\mathbf K^v$8

The sequence term uses truncated Huber supervision for image positions and frame-to-frame displacements. The depth term uses truncated Huber supervision for log-depth. The confidence term is binary cross-entropy for whether the predicted point lies within a prescribed error threshold. The loss is summed over the refinement iterations and averaged over visible points. The reported configuration uses

$\mathbf K^v$9

The trajectory targets in every view are generated by projecting the same ground-truth three-dimensional trajectory with known camera parameters. Consequently, VPM is not trained with unrelated two-dimensional tracks in each view; each view provides a geometrically coupled projection of one shared three-dimensional trajectory.

Cross-view correspondence supervision

VPM also supervises the attention features from which the trajectory head derives correspondence. Let $\mathbf T_\tau^v\in SE(3)$0 denote the attention map from latent frame $\mathbf T_\tau^v\in SE(3)$1 in view $\mathbf T_\tau^v\in SE(3)$2 to latent frame $\mathbf T_\tau^v\in SE(3)$3 in view $\mathbf T_\tau^v\in SE(3)$4, normalized over target spatial cells and averaged over attention heads. For visible query-target pairs,

$\mathbf T_\tau^v\in SE(3)$5

This objective encourages attention to connect the same physical point across time and views. It directly modifies generator features rather than supervising only the final trajectory head.

The combined forecasting objective is

$\mathbf T_\tau^v\in SE(3)$6

with default weights

$\mathbf T_\tau^v\in SE(3)$7

4. Training curriculum and data

VPM uses a three-stage curriculum.

Multi-view video adaptation trains the generator, camera embeddings, and view frequencies with $\mathbf T_\tau^v\in SE(3)$8 using multi-view video sources, including samples without trajectory annotations.

Trajectory-head warm-up freezes the remaining model parameters and trains the trajectory head and view frequencies with $\mathbf T_\tau^v\in SE(3)$9. This allows the trajectory head to learn point motion before the visual backbone is jointly modified.

Joint optimization jointly trains the generator and trajectory head with $h'$0. The reasoner remains frozen throughout. Removing the warm-up stage is particularly harmful: on real robot clips, $h'$1 decreases from $h'$2 with the full schedule to $h'$3 without Stage 2.

VPM-Data

VPM-Data contains 10,000 simulated clips with synchronized multi-view videos and persistent point trajectories.

Domain	Clips	Views	Tracks per clip
Tabletop manipulation	8,000	3	768–2,048
Urban dynamics	2,000	4	256

Tabletop episodes replay physics-simulated pick-and-place behavior of a Fourier GR-1 T2 robot in Unreal Engine 5. Scenes randomize clutter, object placement, surface appearance, camera poses, and target or receptacle configurations. Urban clips contain simulated pedestrians and vehicles in traffic and crowd environments, with randomized camera intrinsics, camera paths, fields of view, target appearance, pedestrian models, and vehicle models.

The full training mixture contains 604,913 samples from VPM-Data, DROID, HRDexDB, MolmoMotion-1M, and Ego-Exo4D. Training snippets contain $h'$4 frames. Longer snippets improve generalization to complex motions. At test time, the model generalizes to approximately 50 frames, although fine-tuning was limited by available hardware to snippets of at most 20 frames.

For each annotated object, the dataset samples 256 surface points. Tabletop clips additionally contain 512 robot points distributed across the arms and hands. For deformable or articulated meshes, point identity is maintained using triangle identity, barycentric coordinates, and skeletal transforms or skinning weights. The dataset stores RGB video, metric depth, camera intrinsics, per-frame extrinsics, visibility and validity masks, object visibility states, captions, and point trajectories.

VP-WorldBench

VP-WorldBench evaluates video fidelity, geometric accuracy, cross-view consistency, and downstream robot control. Its simulation benchmark contains 500 tabletop clips and 500 urban clips. Hard subsets contain sustained camera-specific visibility loss while maintaining collective multi-camera coverage.

The real-world benchmark contains 100 robot manipulation episodes and 100 ten-second indoor sequences. The indoor set uses 48 synchronized RGB cameras and reconstructed dynamic surfaces. It includes people moving, carrying objects, opening drawers, and interacting with furniture.

5. Evaluation and empirical results

VPM evaluates generated video with FVD, PSNR, SSIM, and LPIPS, depending on the benchmark. Cross-view consistency is measured with MEt3R, while three-dimensional trajectory accuracy is evaluated with a depth-adaptive metric. For point error

$h'$5

the metric is

$h'$6

where $h'$7, $h'$8 is ground-truth depth, and $h'$9 is focal length.

Forecasting results

Representative results on VP-WorldBench are:

Model	FVD, simulation robot	PSNR, simulation robot	MEt3R, simulation robot	$\ell$0
Cosmos3-Nano	104.74	20.05	0.040	39.03
VPM (V)	78.52	21.41	0.022	43.62
VPM (V+T)	67.35	22.08	0.016	50.05

On simulation general scenes, VPM (V+T) obtains FVD $\ell$1, PSNR $\ell$2, MEt3R $\ell$3, and $\ell$4, compared with FVD $\ell$5, PSNR $\ell$6, MEt3R $\ell$7, and $\ell$8 for VPM (V).

On real robot scenes, VPM (V+T) obtains FVD $\ell$9, PSNR $h$0, MEt3R $h$1, and $h$2, compared with FVD $h$3, PSNR $h$4, MEt3R $h$5, and $h$6 for VPM (V).

On ReCamMaster, VPM (V+T) improves over VPM (V) from FVD $h$7 to $h$8 and PSNR $h$9 to $\mathcal V_{\mathrm{out}}$00. On CMU Panoptic, it obtains FVD $\mathcal V_{\mathrm{out}}$01, PSNR $\mathcal V_{\mathrm{out}}$02, MEt3R $\mathcal V_{\mathrm{out}}$03, and $\mathcal V_{\mathrm{out}}$04 as reported for that benchmark. On RoboCasa Kitchen, VPM (V+T) obtains PSNR $\mathcal V_{\mathrm{out}}$05, SSIM $\mathcal V_{\mathrm{out}}$06, LPIPS $\mathcal V_{\mathrm{out}}$07, and MEt3R $\mathcal V_{\mathrm{out}}$08.

The reported ablation indicates that persistent trajectories provide more than a generic geometric auxiliary task. On simulation-general clips:

• no geometry supervision: FVD $\mathcal V_{\mathrm{out}}$09, $\mathcal V_{\mathrm{out}}$10;
• per-frame pointmaps: FVD $\mathcal V_{\mathrm{out}}$11, $\mathcal V_{\mathrm{out}}$12;
• persistent trajectories without correspondence loss: FVD $\mathcal V_{\mathrm{out}}$13, $\mathcal V_{\mathrm{out}}$14;
• full trajectory supervision: FVD $\mathcal V_{\mathrm{out}}$15, $\mathcal V_{\mathrm{out}}$16.

These results support the interpretation that temporal identity, rather than geometry supervision alone, is a major source of improvement.

PointMotionBench

On PointMotionBench, VPM uses three frames and language:

Dataset	ADE	FDE	PWT
HOT3D	0.098	0.209	0.396
WorldTrack	0.129	0.248	0.470
DAVIS	1.124	1.804	0.171

VPM achieves the lowest ADE on HOT3D and the best results on all three reported metrics for WorldTrack. It outperforms MolmoMotion-AR under the same three-frame input on ADE and FDE across the reported datasets.

6. Action-enabled VPM and limitations

VPM can be extended to robot control by adding action tokens to the shared denoising backbone. A future action sequence is represented as

$\mathcal V_{\mathrm{out}}$17

with unified actions represented as nine-dimensional vectors. Clean action tokens $\mathcal V_{\mathrm{out}}$18 are independently corrupted,

$\mathcal V_{\mathrm{out}}$19

The action loss is

$\mathcal V_{\mathrm{out}}$20

The robot-learning objective is

$\mathcal V_{\mathrm{out}}$21

Video, trajectory, and action tokens share the predictive backbone and are denoised jointly. Point supervision provides dense three-dimensional future positions for the robot and manipulated objects, complementing low-dimensional action commands.

Method	LIBERO	LIBERO-Plus	RoboCasa	Banana	Cup	Cube
Cosmos3-Nano	96.8	81.6	72.0	50.0	54.2	41.7
VPM (V+A)	96.8	82.0	73.4	58.3	64.6	43.8
VPM (V+T+A)	97.2	84.6	76.2	79.2	70.8	54.2

Point supervision improves over VPM (V+A) on every reported simulation and real-world task.

VPM also supports trajectory-only zero-shot control. Predicted image trajectories and confidences are triangulated into three-dimensional points, and point-to-point inverse kinematics fits the robot’s forward-kinematics model to those points. The controller executes only an initial segment of the predicted trajectory and replans from a new observation.

The method depends on calibrated camera geometry or sufficiently accurate camera estimates, persistent-point supervision, and manageable multi-view token sequences. Its limitations include incomplete scene coverage, limited modeling of fine non-rigid motion, calibration dependence, computational scaling with the number of cameras, simulation-to-real differences, and the difficulty of obtaining large-scale real multi-view trajectory annotations. When future camera poses are unavailable, the model must infer camera motion from observed history, although explicit camera conditioning can improve results when poses are supplied.

VPM is therefore distinguished from video-only world models by its joint prediction of appearance and persistent three-dimensional motion. It is also distinct from per-frame point-map reconstruction: the defining object is not merely a sequence of 3D estimates, but a temporally and cross-view identified trajectory for each physical point. This trajectory-level representation supplies a shared coordinate system for video forecasting, 4D reconstruction, occlusion reasoning, cross-view consistency, and robot manipulation (Jeon et al., 20 Oct 2026).