---
title: 'VP-WorldBench: Benchmark for 3D Video Prediction'
url: https://www.emergentmind.com/topics/vp-worldbench
type: topic
---

# VP-WorldBench: Benchmark for 3D Video Prediction

VP-WorldBench is a forecasting benchmark for evaluating whether video world models can predict a coherent, evolving three-dimensional environment across multiple cameras, time, language conditions, and, in extended settings, robot actions. Introduced with the Video-Point Model (VPM), it pairs synchronized multi-view videos with persistent metric 3D trajectories of scene points. The benchmark evaluates future-video fidelity, cross-view consistency, 3D motion accuracy, visibility and occlusion handling, camera-conditioned forecasting, and the relationship between predicted world states and downstream robot control [2610.28594].

## 1. Purpose and conceptual foundations

VP-WorldBench addresses a limitation of conventional video-generation evaluation: independently plausible frames do not necessarily describe the same physical world. A model may produce visually realistic images while changing object identity across cameras, violating temporal correspondence, misrepresenting depth, or predicting inconsistent motion after occlusion.

The benchmark is based on the premise that a world model should forecast both appearance and structured 3D evolution. Its central annotation is a persistent point trajectory: a physical surface point is identified once and tracked through future frames and camera views, including periods during which it is occluded in an individual view. The model must therefore predict not only where a point appears, but where the same physical point moves in the future.

VP-WorldBench evaluates six related capabilities:

1. future RGB-video forecasting;
2. cross-view consistency;
3. metric 3D trajectory accuracy;
4. robustness to camera-specific disappearance and re-acquisition;
5. generalization across simulated and real environments;
6. usefulness of the representation for robot control.

The benchmark distinguishes ordinary visual plausibility from shared-world coherence. A video can achieve strong perceptual quality while producing poor 3D motion or inconsistent cross-view geometry. Conversely, a model can preserve coarse appearance without correctly forecasting physical trajectories.

## 2. Forecasting tasks and input-output protocols

For camera $v$ and time $\tau$, an RGB frame is denoted $\mathbf O_\tau^v\in\mathbb R^{H\times W\times 3}$, camera intrinsics by $\mathbf K^v$, and camera-to-world pose by $\mathbf T_\tau^v\in SE(3)$. The model receives observed multi-view histories, camera parameters, a language instruction, and point queries sampled from the observations.

A point query is defined as $(t_n,v_n,\mathbf u_n)$, where $t_n$ is an observed timestep, $v_n$ is an observed camera, and $\mathbf u_n$ is a two-dimensional image coordinate. Its persistent three-dimensional position at time $\tau$ is represented by $\mathbf P_{\tau,n}\in\mathbb R^3$. Trajectories are expressed in the coordinate system of a reference camera at the first observed timestep.

The principal protocols are:

- **Image-to-video (I2V)**: one frame per input view is provided, and the model forecasts 64 future frames.
- **Video-to-video (V2V)**: 16 observed frames per view are provided, and the model forecasts the next 64 frames.
- **Camera-conditioned forecasting**: future camera trajectories are additionally provided.
- **Two-view forecasting**: a restricted protocol evaluates performance using only two input views.
- **Trajectory prediction**: the trajectory head directly forecasts persistent 3D point motion.
- **Action-conditioned forecasting**: robot actions are represented as an additional token modality.

Generated videos are resized to 480p for evaluation. The main benchmark results use all available views without supplying future camera trajectories at inference, requiring the model to infer camera motion from the observed history. Each clip also has an English caption or instruction generated and filtered by a vision-language model. Instructions describe the scene and activity while avoiding numerical claims and camera descriptions.

## 3. VPM architecture and trajectory supervision

VPM extends the Cosmos3-Nano mixture-of-transformers world model with multi-view processing and a trajectory head. All camera streams are packed into a shared token sequence. Camera geometry is encoded using Plücker rays. For a token with ray origin $\mathbf o_i$ and unit direction $\mathbf d_i$, the ray representation is

$$
\mathbf r_i=(\mathbf d_i,\mathbf o_i\times \mathbf d_i).
$$

A camera encoder maps this representation to an embedding that is added to token features and used to rotate attention queries and keys. VPM also adds a view-specific rotary-position axis, preventing tokens at identical temporal and spatial positions in different cameras from being treated as identical.

Video generation uses rectified flow matching. Given clean future-video tokens $\mathbf x_0$, Gaussian noise $\boldsymbol\epsilon$, and noise level $\sigma$, the noisy representation is

$$
\mathbf x_\sigma=(1-\sigma)\mathbf x_0+\sigma\boldsymbol\epsilon.
$$

The denoising objective predicts the flow from the noisy representation toward the clean future video.

The trajectory head reads query and key features from the generator’s 18th self-attention layer. It samples a source feature at a queried point, computes correlations with nearby features across views and time, iteratively refines the estimate four times, predicts two-dimensional position, depth, and confidence, and unprojects the estimates into a fixed reference coordinate system.

Trajectory supervision combines:

- a truncated Huber loss on two-dimensional positions and frame-to-frame displacements;
- a truncated Huber loss on log-depth;
- binary cross-entropy for prediction confidence;
- a cross-view correspondence loss over visible query-target pairs.

The default trajectory and correspondence weights are $\lambda_{\mathrm{track}}=\lambda_{\mathrm{corr}}=0.01$, while sequence, depth, and confidence components use unit weights. The full forecasting objective combines video flow matching, trajectory supervision, and correspondence supervision.

For action-conditioned forecasting, future actions are represented as a nine-dimensional token modality. The robot objective adds an action-denoising loss with $\lambda_{\mathrm{act}}=0.1$. This shared representation is intended to connect predicted visual dynamics, persistent 3D motion, and robot control.

## 4. Dataset and benchmark composition

### VPM-Data

The training corpus contains 10,000 simulated multi-view clips:

| Domain | Samples | Views | FPS | Frames | Tracks per clip |
|---|---:|---:|---:|---:|---:|
| Tabletop manipulation | 8,000 | 3 | 20 | 81–181 | 768–2,048 |
| Urban dynamics | 2,000 | 4 | 15 | 81–181 | 256 |

The broader training mixture contains 604,913 samples from simulated tabletop and urban data, DROID, HRDexDB, MolmoMotion-1M, and Ego-Exo4D. Training proceeds in three stages: adapting the generator and camera embeddings with video flow matching; training the trajectory head while freezing the remaining model; and jointly training video generation and trajectory supervision.

### VP-WorldBench splits

The benchmark includes simulated and real environments:

| Domain | Samples | Views | FPS | Frames | Tracks per clip | Hard clips |
|---|---:|---:|---:|---:|---:|---:|
| Simulated tabletop manipulation | 500 | 3 | 20 | 81–680 | 768–1,536 | 27 |
| Simulated urban dynamics | 500 | 4 | 15 | 81–181 | 256 | 95 |
| Real tabletop manipulation | 100 | 2 | 15 | 139–325 | 768 | — |
| Real indoor scenes | 100 | 48 | 15 | 150 | 512 | — |

The simulation benchmark contains 661,071 RGB images and 716,544 persistent point trajectories.

Simulated tabletop sequences replay physics-simulated Fourier GR-1 T2 robot pick-and-place episodes in Unreal Engine 5. They contain one to four manipulation targets or receptacles, three cameras, 1–12 distractors, 306 possible prop types, robot arms and hands, receptacles, and clutter. The evaluation includes ordinary fixed-camera clips and hard clips with camera rotations and occlusions. The tabletop set contains 1,299 tracked objects, comprising 693 manipulation targets and 606 receptacles.

Simulated urban sequences use an Unreal-based driving and crowd simulator with 332 pedestrians, 168 vehicles, 55 target models, six city tiles, approximately 30 background vehicles, and approximately 40 pedestrians. Target entities may be moving or stationary pedestrians or vehicles. Cameras use independently sampled horizontal fields of view from $25^\circ$ to $90^\circ$ and include static, arc, translation, random-polyline, and pan-tilt trajectories.

The real robot portion contains 100 teleoperated tabletop manipulation episodes using a Franka Research 3 arm, Robotiq 2F-85 gripper, one wrist-mounted Zed Mini stereo camera, and two external Zed 2i stereo cameras. Tasks involve moving a banana to a basket, a green cup to a plate, or a blue cube onto a red cube while avoiding a box pile.

The real indoor split contains 100 ten-second sequences captured in a $23.1\,\mathrm m^2$ home-like environment with three participants, household furniture and objects, and 48 synchronized FLIR RGB cameras. The released benchmark uses 15 FPS and 150 frames per sequence.

## 5. Persistent trajectories, visibility, and hard cases

Persistent points are sampled on object or robot surfaces once and retain their identity throughout a sequence. For a local point $\mathbf p_n^\ell$ on link or object $\ell$, its world-space position is

$$
\mathbf p_{\tau,n}^w=\mathbf R_{\ell,\tau}^w\mathbf p_n^\ell+\mathbf b_{\ell,\tau}^w.
$$

Because $\mathbf p_n^\ell$ remains fixed, the trajectory index identifies the same physical surface location over time. Sampling uses 256 points per annotated object, 512 robot points in simulated tabletop scenes, approximately 256 robot points in real-robot extraction, and representation based on triangle indices, barycentric coordinates, skinning weights, and bone transforms for urban scenes.

Camera-coordinate trajectories are obtained through world-to-camera transformations. The stored benchmark convention uses metres and the OpenCV coordinate system:

$$
+X:\text{right},\qquad +Y:\text{down},\qquad +Z:\text{forward}.
$$

Visibility is annotated independently from point identity. For tabletop points, a projected point is visible when its camera-space depth $z$ satisfies

$$
z\leq d+\epsilon,
$$

where $d$ is rendered depth and $\epsilon=3$ cm. An object is considered visible when more than 10% of its in-frustum points pass this test. Urban visibility distinguishes visible, occluded, out-of-frustum, and absent or invalid states.

The point trajectory remains defined during occlusion. This permits evaluation of whether the model predicts motion through intervals in which a point is not directly observed by a particular camera.

Hard clips enforce sustained camera-specific disappearance while maintaining collective multi-view coverage. At least one camera sees the target at every timestep, while each affected camera loses the target for at least 30 consecutive frames. The target must also move during the hidden interval. In tabletop hard clips, all three cameras satisfy the disappearance criterion. In urban hard clips, two or three of four cameras do so, with all four cameras performing departure and re-acquisition cycles.

These cases distinguish simple interpolation from multi-view world forecasting. A model must transfer motion information across cameras, preserve point identity through occlusion, and reconstruct the target when it reappears.

## 6. Evaluation metrics and empirical findings

VP-WorldBench evaluates visual fidelity, cross-view consistency, persistent 3D motion, and downstream control.

### Video fidelity

The benchmark reports FVD using I3D features and PSNR against ground-truth frames. Metrics are computed independently per output view and averaged across views. External evaluations additionally report SSIM and LPIPS.

### Cross-view consistency

MEt3R compares feature-rendered point maps between image pairs. VP-WorldBench reports deviation from ground-truth MEt3R rather than raw MEt3R, because raw scores can incorrectly reward non-overlapping views. Lower cross-view MEt3R deviation is better.

### 3D trajectory accuracy

Generated videos are processed with MVTracker and MoGe-2 depth. For predicted point $\widehat{\mathbf P}_{\tau,n}$ and ground truth $\mathbf P_{\tau,n}$, the Euclidean error is

$$
e_{\tau,n}=\left\|\widehat{\mathbf P}_{\tau,n}-\mathbf P_{\tau,n}\right\|_2.
$$

The benchmark reports the fraction of points within depth-adaptive thresholds $\mathcal D=\{1,2,4,8,16\}$, scaled using ground-truth depth and focal length. Higher values indicate more accurate 3D forecasting. Ground-truth videos are passed through the same tracking and reconstruction pipeline to estimate an upper bound below an ideal score of 100.

### Robot control

Robot evaluations report task success rates. In the real-robot benchmark, a placement that reaches the goal while touching the box pile receives 0.5, successful placement without contact receives 1, and other outcomes receive 0.

The main all-view I2V results show that VPM with video and trajectory supervision, denoted VPM (V+T), improves over video-only VPM across all reported splits:

| Split | FVD | PSNR | Cross-view MEt3R | $\delta^{3D}$ |
|---|---:|---:|---:|---:|
| Simulated robot | 67.35 | 22.08 | .016 | 50.05 |
| Simulated general | 81.92 | 19.16 | .050 | 36.80 |
| Real robot | 255.19 | 20.57 | .064 | 64.75 |
| Real general | 56.48 | 22.10 | .007 | 33.92 |

VPM (V) without trajectory supervision obtains FVD values of 78.52, 101.37, 296.83, and 62.85 on the same four splits. Its corresponding 3D trajectory scores are 43.62, 28.71, 50.83, and 26.54. Thus, trajectory supervision improves both video quality and geometric forecasting under the reported protocol.

The ablations indicate that persistent trajectories provide a stronger signal than per-frame depth or point maps. Full trajectory supervision achieves a simulated-robot $\delta^{3D}$ of 50.05, compared with 45.21 for depth supervision and 44.90 for point-map supervision. Removing correspondence supervision reduces performance further.

Action-token extensions improve downstream control. VPM (V+T+A) reaches 97.2 on LIBERO, 84.6 on LIBERO-Plus, and 76.2 on RoboCasa. On the real robot tasks, it achieves 79.2 for the banana task, 70.8 for the cup task, and 54.2 for the cube task. These results are reported as task success rates or fractional success scores according to the corresponding protocol.

The benchmark also supports a training-free trajectory-to-control procedure. It samples robot surface points, reads predicted two-dimensional trajectories, triangulates them into three dimensions, fits robot forward kinematics, and executes the resulting joint trajectory in a receding-horizon manner. The paper reports qualitative zero-shot real-robot successes but does not regard the method as sufficiently reliable for quantitative evaluation.

## 7. Limitations and relation to adjacent benchmarks

VP-WorldBench combines capabilities usually evaluated separately by multi-view tracking, point-motion forecasting, camera-controlled generation, 4D generation, and robot manipulation benchmarks. Its distinctive focus is future shared-world evolution: synchronized multi-view forecasting, persistent metric trajectories, camera geometry, occlusion handling, language conditioning, and action-relevant motion.

Several limitations constrain interpretation. The benchmark relies on calibrated cameras, model-specific conditioning conversions, segmentation and tracking, depth reconstruction, and pose estimation. Real-world trajectories depend on calibration and reconstruction quality. Simulation does not reproduce all real sensor noise, and the benchmark’s point annotations primarily cover robots and selected objects rather than every scene element. Cloth and hair motion are not represented by the skeletal annotations.

The number of cameras, frame rates, resolutions, and clip lengths varies across domains. Computational cost grows with the number of camera streams because they are processed jointly. Urban hard clips are disproportionately pedestrian-heavy, and live traffic and crowd simulation are not necessarily bit-exactly reproducible from random seeds.

The benchmark also does not establish that a model contains an explicit persistent-memory module. A model may rely on implicit reconstruction, visual priors, or language conditioning. Conversely, a failure may reflect camera-following error, occlusion, tracking failure, or reconstruction noise rather than a failure of world memory.

VP-WorldBench is therefore best understood as a structured evaluation suite for multi-view 4D forecasting rather than a single scalar test. Its principal contribution is to align video prediction with persistent 3D identity and motion. The reported results support the conclusion that point-trajectory supervision improves both future-video fidelity and 3D motion accuracy, while action-token integration can transfer these representations to robot control.

Source: https://www.emergentmind.com/topics/vp-worldbench