---
title: Video-Point Model (VPM) Overview
url: https://www.emergentmind.com/topics/video-point-model-vpm
type: topic
---

# Video-Point Model (VPM) Overview

Video-Point Model (VPM) is a camera-conditioned, multi-view world model for forecasting future RGB videos and persistent three-dimensional trajectories of physical scene points. Given synchronized multi-view video histories, camera parameters, a language instruction, and point queries selected from observed frames, VPM jointly predicts future videos from requested viewpoints and the 3D motion of the queried points. Its central premise is that persistent 3D trajectories provide a stronger constraint on temporal and cross-view consistency than per-frame depth or point-map supervision alone. The model is built on the Cosmos 3 / Cosmos3-Nano mixture-of-transformers architecture and extends video forecasting toward 4D scene modeling and robot control [2610.28594].

## 1. Problem formulation and representational basis

Let $\mathcal V_{\mathrm{in}}$ denote the input cameras and $\mathcal V_{\mathrm{out}}$ the requested output cameras. For camera $v$ and time $\tau$, an RGB frame is represented as $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$, with camera intrinsic matrix $\mathbf K^v$ and camera-to-world pose $\mathbf T_\tau^v\in SE(3)$. Given an observed history of $h'$ frames, camera parameters, and a language instruction $\ell$, VPM predicts the next $h$ frames,

$$
\left\{\widehat{\mathbf O}_{t+1:t+h}^{v}\right\}_{v\in\mathcal V_{\mathrm{out}}},
$$

together with trajectories for $N$ points selected from the observed history.

A point query is

$$
q_n=(t_n,v_n,\mathbf u_n),
$$

where $t_n$ is an observed time, $v_n$ is the source camera, and $\mathbf u_n\in\mathbb R^2$ is the image location. The corresponding physical point has three-dimensional position $\mathbf P_{\tau,n}\in\mathbb R^3$ at future time $\tau$. Trajectories are expressed in the coordinate frame of a reference camera $v_{\mathrm{ref}}$ at the first observed time $t_0=t-h'+1$.

Conceptually, VPM models the conditional distribution

$$
p\left(
\mathbf O_{t+1:t+h}^{\mathcal V_{\mathrm{out}}},
\mathbf P_{t+1:t+h,1:N}
\mid
\mathbf O_{t-h'+1:t}^{\mathcal V_{\mathrm{in}}},
\mathbf K,\mathbf T,\ell,q_{1:N}
\right).
$$

The model therefore combines video prediction with point-trajectory forecasting. A trajectory preserves the identity of one physical point across time and cameras, unlike independent per-frame depth estimates or unrelated two-dimensional tracks. If a point moves from $\mathbf P_{\tau,n}$ to $\mathbf P_{\tau+1,n}$, its three-dimensional displacement is represented directly by

$$
\Delta\mathbf P_{\tau,n}
=
\mathbf P_{\tau+1,n}-\mathbf P_{\tau,n}.
$$

This shared identity constrains generated appearances in different cameras to correspond to the same evolving three-dimensional event. It also allows a point to disappear from one camera while remaining visible in another.

The principal distinction between VPM and conventional video world models is representational. A video-only model can produce individually plausible frames while allowing object identity, geometry, and motion to disagree across views. VPM instead forecasts RGB observations and persistent 3D trajectories jointly. Its point supervision is intended to anchor appearance changes to a common description of scene motion.

## 2. Camera-conditioned multi-view architecture

VPM starts from Cosmos3-Nano, whose mixture-of-transformers architecture contains a reasoner and a generator. The reasoner processes language and high-level conditioning, whereas the generator denoises future video tokens. The reasoner remains frozen during VPM training. The generation pathway is modified for multi-view input, and a trajectory head is attached to intermediate generator representations [2610.28594].

Latent video tokens from all cameras are placed on a common temporal grid containing clean observed-history tokens, noisy future tokens, and tokens associated with requested output cameras. Attention can therefore operate across time, image space, and camera views. A point visible in one camera can influence the prediction for another camera, including when it is temporarily occluded in the second view.

For every video token, VPM computes the camera ray corresponding to the represented image region. In the reference frame, the ray is encoded with Plücker coordinates,

$$
\mathbf r_i=
\left(
\mathbf d_i,\,
\mathbf o_i\times\mathbf d_i
\right),
$$

where $\mathbf o_i$ is the ray origin and $\mathbf d_i$ is the unit ray direction. A lightweight encoder $\phi$ produces the camera embedding

$$
\mathbf e_i=\phi(\mathbf r_i).
$$

The embedding enters each transformer layer additively,

$$
\mathbf h_i\leftarrow \mathbf h_i+\mathbf W\mathbf e_i,
$$

and through camera-dependent query and key rotations,

$$
\mathbf q_i\leftarrow
\mathbf R\bigl(\psi(\mathbf e_i)\bigr)\mathbf q_i,
\qquad
\mathbf k_i\leftarrow
\mathbf R\bigl(\psi(\mathbf e_i)\bigr)\mathbf k_i.
$$

VPM also introduces a view-specific axis into rotary positional encoding. If $p_{t,i}$, $p_{h,i}$, and $p_{w,i}$ denote temporal and spatial token coordinates, and $p_{\mathrm{view},i}=v$ denotes the camera index, the rotary phase is

$$
\theta_{i,k}
=
p_{t,i}\omega_k^t+
p_{h,i}\omega_k^h+
p_{w,i}\omega_k^w+
p_{\mathrm{view},i}\omega_k^{\mathrm{view}}.
$$

The view frequencies are initialized to zero, allowing the model initially to recover the pretrained three-dimensional positional encoding. The camera-conditioning mechanism consequently consists of additive Plücker embeddings, camera-dependent query/key rotations, and a learnable view axis in rotary positional encoding. Removing these components substantially harms performance, particularly for moving-camera urban scenes.

### Rectified-flow video generation

Let $\mathbf x_0$ be clean future-video latent tokens and $\boldsymbol\epsilon\sim\mathcal N(0,I)$. At noise level $\sigma\in[0,1]$, VPM forms

$$
\mathbf x_\sigma
=
(1-\sigma)\mathbf x_0+\sigma\boldsymbol\epsilon.
$$

The generator denoiser $v_\theta$ predicts the rectified-flow velocity $\boldsymbol\epsilon-\mathbf x_0$. The video-generation objective is

$$
\mathcal L_{\mathrm{FM}}
=
\mathbb E_{\mathbf x_0,c,\boldsymbol\epsilon,\sigma}
\left[
\left\|
v_\theta(\mathbf x_\sigma,\sigma,c)
-
(\boldsymbol\epsilon-\mathbf x_0)
\right\|_2^2
\right],
$$

where $c$ contains the observed video history, camera conditions, language, and other conditioning tokens. Clean conditioning tokens are not included in the generation loss. At inference, observed-history tokens remain clean, whereas future tokens are initialized from Gaussian noise at $\sigma=1$ and iteratively denoised to $\sigma=0$.

## 3. Persistent trajectory prediction

Each queried point retains one identity across time, camera views, and the observed-to-future transition. The trajectory head predicts, for every view and timestep:

- a two-dimensional image position $\widehat{\mathbf u}_{\tau,n}^v$;
- a depth $\widehat d_{\tau,n}^v$;
- a confidence $\widehat c_{\tau,n}^v$.

The head operates on query and key features from an intermediate generator self-attention layer. In the implementation, these features are taken from layer 18 and reshaped into per-view latent grids $\mathbf F_q^v$ and $\mathbf F_k^v$. For each queried point, a source query feature is sampled and correlated with key features around the current point estimate in every view and timestep.

A transformer refiner updates point estimates over $M$ iterations, with the default $M=4$. Its attention is factorized over spatial positions, time, and camera views. This allows refinement to integrate local image evidence, temporal motion patterns, and cross-view correspondence. Cross-view attention is particularly important; removing it produces the largest degradation among the refiner ablations.

Predicted image positions and depths are lifted into the reference coordinate system. Let

$$
\mathbf G_r^v
=
(\mathbf T_{t_0}^{v_{\mathrm{ref}}})^{-1}\mathbf T_r^v
$$

be the camera-to-reference transformation at time $r$, with rotation $\mathbf R_r^v$ and translation $\mathbf b_r^v$. The predicted point is

$$
\widehat{\mathbf P}_{r,n}^v
=
\mathbf R_r^v
\left(
\widehat d_{r,n}^v
(\mathbf K^v)^{-1}
\begin{bmatrix}
\widehat{\mathbf u}_{r,n}^v\\
1
\end{bmatrix}
\right)
+
\mathbf b_r^v.
$$

Predictions from multiple views thus describe the same physical point in a shared reference coordinate system.

### Trajectory supervision

The trajectory loss is

$$
\mathcal L_{\mathrm{track}}
=
\lambda_{\mathrm{seq}}\mathcal L_{\mathrm{seq}}
+
\lambda_{\mathrm{depth}}\mathcal L_{\mathrm{depth}}
+
\lambda_{\mathrm{conf}}\mathcal L_{\mathrm{conf}}.
$$

The sequence term uses truncated Huber supervision for image positions and frame-to-frame displacements. The depth term uses truncated Huber supervision for log-depth. The confidence term is binary cross-entropy for whether the predicted point lies within a prescribed error threshold. The loss is summed over the refinement iterations and averaged over visible points. The reported configuration uses

$$
\lambda_{\mathrm{seq}}
=
\lambda_{\mathrm{depth}}
=
\lambda_{\mathrm{conf}}
=
1.
$$

The trajectory targets in every view are generated by projecting the same ground-truth three-dimensional trajectory with known camera parameters. Consequently, VPM is not trained with unrelated two-dimensional tracks in each view; each view provides a geometrically coupled projection of one shared three-dimensional trajectory.

### Cross-view correspondence supervision

VPM also supervises the attention features from which the trajectory head derives correspondence. Let $\mathbf C_{i,j}^{v_1,v_2}$ denote the attention map from latent frame $i$ in view $v_1$ to latent frame $j$ in view $v_2$, normalized over target spatial cells and averaged over attention heads. For visible query-target pairs,

$$
\mathcal L_{\mathrm{corr}}
=
-\frac{1}{|\Omega|}
\sum_{(n,v_2,j)\in\Omega}
\log
\mathbf C_{t_n,j}^{v_n,v_2}
\left[
\mathbf u_n,\mathbf u_{j,n}^{v_2}
\right].
$$

This objective encourages attention to connect the same physical point across time and views. It directly modifies generator features rather than supervising only the final trajectory head.

The combined forecasting objective is

$$
\mathcal L_{\mathrm{forecast}}
=
\mathcal L_{\mathrm{FM}}
+
\lambda_{\mathrm{track}}\mathcal L_{\mathrm{track}}
+
\lambda_{\mathrm{corr}}\mathcal L_{\mathrm{corr}},
$$

with default weights

$$
\lambda_{\mathrm{track}}=0.01,
\qquad
\lambda_{\mathrm{corr}}=0.01.
$$

## 4. Training curriculum and data

VPM uses a three-stage curriculum.

**Multi-view video adaptation** trains the generator, camera embeddings, and view frequencies with $\mathcal L_{\mathrm{FM}}$ using multi-view video sources, including samples without trajectory annotations.

**Trajectory-head warm-up** freezes the remaining model parameters and trains the trajectory head and view frequencies with $\mathcal L_{\mathrm{track}}$. This allows the trajectory head to learn point motion before the visual backbone is jointly modified.

**Joint optimization** jointly trains the generator and trajectory head with $\mathcal L_{\mathrm{forecast}}$. The reasoner remains frozen throughout. Removing the warm-up stage is particularly harmful: on real robot clips, $\delta_{\mathrm{avg}}^{3D}$ decreases from $64.75$ with the full schedule to $49.77$ without Stage 2.

### VPM-Data

VPM-Data contains 10,000 simulated clips with synchronized multi-view videos and persistent point trajectories.

| Domain | Clips | Views | Tracks per clip |
|---|---:|---:|---:|
| Tabletop manipulation | 8,000 | 3 | 768–2,048 |
| Urban dynamics | 2,000 | 4 | 256 |

Tabletop episodes replay physics-simulated pick-and-place behavior of a Fourier GR-1 T2 robot in Unreal Engine 5. Scenes randomize clutter, object placement, surface appearance, camera poses, and target or receptacle configurations. Urban clips contain simulated pedestrians and vehicles in traffic and crowd environments, with randomized camera intrinsics, camera paths, fields of view, target appearance, pedestrian models, and vehicle models.

The full training mixture contains 604,913 samples from VPM-Data, DROID, HRDexDB, MolmoMotion-1M, and Ego-Exo4D. Training snippets contain $N\in\{5,9,19\}$ frames. Longer snippets improve generalization to complex motions. At test time, the model generalizes to approximately 50 frames, although fine-tuning was limited by available hardware to snippets of at most 20 frames.

For each annotated object, the dataset samples 256 surface points. Tabletop clips additionally contain 512 robot points distributed across the arms and hands. For deformable or articulated meshes, point identity is maintained using triangle identity, barycentric coordinates, and skeletal transforms or skinning weights. The dataset stores RGB video, metric depth, camera intrinsics, per-frame extrinsics, visibility and validity masks, object visibility states, captions, and point trajectories.

### VP-WorldBench

VP-WorldBench evaluates video fidelity, geometric accuracy, cross-view consistency, and downstream robot control. Its simulation benchmark contains 500 tabletop clips and 500 urban clips. Hard subsets contain sustained camera-specific visibility loss while maintaining collective multi-camera coverage.

The real-world benchmark contains 100 robot manipulation episodes and 100 ten-second indoor sequences. The indoor set uses 48 synchronized RGB cameras and reconstructed dynamic surfaces. It includes people moving, carrying objects, opening drawers, and interacting with furniture.

## 5. Evaluation and empirical results

VPM evaluates generated video with FVD, PSNR, SSIM, and LPIPS, depending on the benchmark. Cross-view consistency is measured with MEt3R, while three-dimensional trajectory accuracy is evaluated with a depth-adaptive metric. For point error

$$
e_{\tau,n}
=
\left\|
\widehat{\mathbf P}_{\tau,n}
-
\mathbf P_{\tau,n}
\right\|_2,
$$

the metric is

$$
\delta_{\mathrm{avg}}^{3D}
=
\frac{1}{|\mathcal D|}
\sum_{\delta\in\mathcal D}
\frac{1}{Nh}
\sum_{n=1}^{N}
\sum_{\tau=t+1}^{t+h}
\mathbf 1
\left[
e_{\tau,n}
<
\delta
\frac{d_{\tau,n}}{f}
\right],
$$

where $\mathcal D=\{1,2,4,8,16\}$, $d_{\tau,n}$ is ground-truth depth, and $f$ is focal length.

### Forecasting results

Representative results on VP-WorldBench are:

| Model | FVD, simulation robot | PSNR, simulation robot | MEt3R, simulation robot | $\delta_{\mathrm{avg}}^{3D}$ |
|---|---:|---:|---:|---:|
| Cosmos3-Nano | 104.74 | 20.05 | 0.040 | 39.03 |
| VPM (V) | 78.52 | 21.41 | 0.022 | 43.62 |
| VPM (V+T) | **67.35** | **22.08** | **0.016** | **50.05** |

On simulation general scenes, VPM (V+T) obtains FVD $81.92$, PSNR $19.16$, MEt3R $0.050$, and $\delta_{\mathrm{avg}}^{3D}=36.80$, compared with FVD $101.37$, PSNR $16.95$, MEt3R $0.057$, and $\delta_{\mathrm{avg}}^{3D}=28.71$ for VPM (V).

On real robot scenes, VPM (V+T) obtains FVD $255.19$, PSNR $20.57$, MEt3R $0.064$, and $\delta_{\mathrm{avg}}^{3D}=64.75$, compared with FVD $296.83$, PSNR $18.42$, MEt3R $0.074$, and $\delta_{\mathrm{avg}}^{3D}=50.83$ for VPM (V).

On ReCamMaster, VPM (V+T) improves over VPM (V) from FVD $103.12$ to $79.63$ and PSNR $15.09$ to $17.06$. On CMU Panoptic, it obtains FVD $128.64$, PSNR $23.24$, MEt3R $0.029$, and $\delta_{\mathrm{avg}}^{3D}=0.029$ as reported for that benchmark. On RoboCasa Kitchen, VPM (V+T) obtains PSNR $22.83$, SSIM $0.8801$, LPIPS $0.0537$, and MEt3R $0.020$.

The reported ablation indicates that persistent trajectories provide more than a generic geometric auxiliary task. On simulation-general clips:

- no geometry supervision: FVD $101.37$, $\delta_{\mathrm{avg}}^{3D}=28.71$;
- per-frame pointmaps: FVD $99.35$, $\delta_{\mathrm{avg}}^{3D}=29.69$;
- persistent trajectories without correspondence loss: FVD $85.73$, $\delta_{\mathrm{avg}}^{3D}=31.54$;
- full trajectory supervision: FVD $81.92$, $\delta_{\mathrm{avg}}^{3D}=36.80$.

These results support the interpretation that temporal identity, rather than geometry supervision alone, is a major source of improvement.

### PointMotionBench

On PointMotionBench, VPM uses three frames and language:

| Dataset | ADE | FDE | PWT |
|---|---:|---:|---:|
| HOT3D | 0.098 | 0.209 | 0.396 |
| WorldTrack | 0.129 | 0.248 | 0.470 |
| DAVIS | 1.124 | 1.804 | 0.171 |

VPM achieves the lowest ADE on HOT3D and the best results on all three reported metrics for WorldTrack. It outperforms MolmoMotion-AR under the same three-frame input on ADE and FDE across the reported datasets.

## 6. Action-enabled VPM and limitations

VPM can be extended to robot control by adding action tokens to the shared denoising backbone. A future action sequence is represented as

$$
\mathbf A_t\in\mathbb R^{h_A\times d_A},
$$

with unified actions represented as nine-dimensional vectors. Clean action tokens $\mathbf a_0$ are independently corrupted,

$$
\mathbf a_{\sigma_a}
=
(1-\sigma_a)\mathbf a_0
+
\sigma_a\boldsymbol\epsilon_a,
\qquad
\boldsymbol\epsilon_a\sim\mathcal N(0,I).
$$

The action loss is

$$
\mathcal L_{\mathrm{act}}
=
\mathbb E
\left[
\left\|
v_\theta^a
\left(
\mathbf a_{\sigma_a},
\sigma_a,
\mathbf x_\sigma,
c
\right)
-
\left(
\boldsymbol\epsilon_a-\mathbf a_0
\right)
\right\|_2^2
\right].
$$

The robot-learning objective is

$$
\mathcal L_{\mathrm{robot}}
=
\mathcal L_{\mathrm{forecast}}
+
\lambda_{\mathrm{act}}\mathcal L_{\mathrm{act}},
\qquad
\lambda_{\mathrm{act}}=0.1.
$$

Video, trajectory, and action tokens share the predictive backbone and are denoised jointly. Point supervision provides dense three-dimensional future positions for the robot and manipulated objects, complementing low-dimensional action commands.

| Method | LIBERO | LIBERO-Plus | RoboCasa | Banana | Cup | Cube |
|---|---:|---:|---:|---:|---:|---:|
| Cosmos3-Nano | 96.8 | 81.6 | 72.0 | 50.0 | 54.2 | 41.7 |
| VPM (V+A) | 96.8 | 82.0 | 73.4 | 58.3 | 64.6 | 43.8 |
| VPM (V+T+A) | **97.2** | **84.6** | **76.2** | **79.2** | **70.8** | **54.2** |

Point supervision improves over VPM (V+A) on every reported simulation and real-world task.

VPM also supports trajectory-only zero-shot control. Predicted image trajectories and confidences are triangulated into three-dimensional points, and point-to-point inverse kinematics fits the robot’s forward-kinematics model to those points. The controller executes only an initial segment of the predicted trajectory and replans from a new observation.

The method depends on calibrated camera geometry or sufficiently accurate camera estimates, persistent-point supervision, and manageable multi-view token sequences. Its limitations include incomplete scene coverage, limited modeling of fine non-rigid motion, calibration dependence, computational scaling with the number of cameras, simulation-to-real differences, and the difficulty of obtaining large-scale real multi-view trajectory annotations. When future camera poses are unavailable, the model must infer camera motion from observed history, although explicit camera conditioning can improve results when poses are supplied.

VPM is therefore distinguished from video-only world models by its joint prediction of appearance and persistent three-dimensional motion. It is also distinct from per-frame point-map reconstruction: the defining object is not merely a sequence of 3D estimates, but a temporally and cross-view identified trajectory for each physical point. This trajectory-level representation supplies a shared coordinate system for video forecasting, 4D reconstruction, occlusion reasoning, cross-view consistency, and robot manipulation [2610.28594].

Source: https://www.emergentmind.com/topics/video-point-model-vpm