PointZero: Dynamic 3D Prediction | Transformers

• PointZero is a transformer-based model designed for robot-free pre-training of transferable 3D dynamics through 3D point-track completion, leveraging synthetic datasets for various object deformations, articulations, and rigid motions.
• Its key method is to predict future 3D trajectories for all observed points from a single RGB-D observation and a small set of partial trajectories, using direct regression, flow matching, or JiT-style prediction.
• Evaluated on synthetic and real-world tasks, PointZero outperforms baselines in multiple metrics, showing potential for action-conditioned dynamics and imitation learning

PointZero is a transformer-based model for robot-free pre-training of transferable 3D dynamics through 3D point-track completion. Given a single RGB-D observation, a sparse set of partial 3D trajectories, and the visible point cloud, PointZero predicts future 3D trajectories for all observed points. The model is trained on approximately 2.9 million synthetic image frames spanning deformable, articulated, and rigid objects, without robot action labels, and is subsequently adapted to action-conditioned dynamics prediction and imitation learning (Duisterhof et al., 16 Sep 2026). The name has also been used descriptively for several unrelated “zero-point” concepts, including gravitational vacuum observables, zero-phase-delay photonic propagation, dominant partition-function zeros, localized non-Hermitian zero-energy modes, and numerical zero-finding methods; these usages do not denote the PointZero dynamics model.

1. Scope and conceptual basis

PointZero addresses the data and embodiment constraints of conventional action-conditioned world models. Such models typically learn a mapping from scene state and robot action to future scene state, requiring robot demonstrations, calibrated robot states, and action labels. These requirements exclude much human and web video, where object motion may be visible but robot actions are absent.

PointZero replaces explicit action conditioning during pre-training with a small set of observed 3D point trajectories. Its input consists of:

• one RGB-D observation;
• visible 3D points reconstructed from the RGB-D data;
• one to three sparse 3D point trajectories;
• the RGB image and camera information.

The model predicts a dense future point-track field. In abstract form,

$(I,D,M,K,A)\longrightarrow \widehat{P}_{2:T},$

where $I$ is the RGB image, $D$ is the depth map, $M$ is a foreground mask, $K$ is the camera intrinsic matrix, $A$ is the sparse trajectory set, and $P_{2:T}$ contains future positions for all observed points.

The pre-training objective is based on the premise that sparse point trajectories act as an action-like conditioning signal. Unlike a robot action, a point trajectory is directly observable in video or reconstructed RGB-D data. The resulting representation is intended to cover rigid motion, articulation, cloth deformation, collisions, and other high-dimensional dynamics. It can subsequently be adapted by replacing sparse tracks with robot end-effector states or by adding an action-prediction head.

PointZero should be distinguished from several other terms involving “PointZero.” In gravitational research, the term has been used editorially for a gravitational vacuum baseline from which induced cosmological and charge-like observables may arise through graviton zero-point fluctuations (Garattini, 2010). In photonics, it describes a zero-phase-delay condition satisfying $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$ (Guoyan et al., 2012). In statistical mechanics, it denotes a scaled Newton–Raphson procedure for finding the dominant energy-probability-distribution zero (Carvalho et al., 2019). These are conceptual or method-specific usages unrelated to the 3D dynamics architecture.

2. Input representation and point-track completion

The observed RGB-D frame is represented by

$$I\in\mathbb{R}^{H\times W\times 3},\qquad D\in\mathbb{R}^{H\times W},\qquad M\in\{0,1\}^{H\times W},\qquad K\in\mathbb{R}^{3\times 3}.$$

Masked depth pixels are unprojected into an observed point cloud $P^{\mathrm{obs}}\in\mathbb{R}^{N_p\times3}$. For a pixel with homogeneous image coordinate $I$0, the camera-coordinate point is

$I$1

PointZero normalizes each point cloud using the initial-frame centroid and its $I$2th-percentile radius. If

$I$3

is the centroid and $I$4 is the $I$5th-percentile radial distance from $I$6, the normalized coordinates are

$I$7

The inverse transformation is applied before reporting physical errors. Random rotations and Gaussian noise are applied to point clouds and conditioning tracks. RGB images receive brightness, contrast, salt-and-pepper, and Gaussian-noise augmentation.

The conditioning trajectories have the form

$I$8

with $I$9 and typically $D$0 physical frames. Each $D$1 specifies the position of conditioning point $D$2 at time $D$3. The output is a dense tensor

$D$4

with the first frame fixed to the observed point cloud:

$D$5

For each observed point, PointZero predicts a flattened future trajectory

$D$6

The formulation is parallel rather than autoregressive: the model predicts an entire future trajectory per point instead of recursively predicting only the next state. This design avoids compounding one-step prediction errors and permits a direct representation of temporally correlated motion.

3. Learning objectives

PointZero evaluates direct regression, flow matching, and JiT-style $D$7-prediction. All objectives use a coordinate-averaged, temporally weighted squared error. For trajectory error $D$8,

$D$9

The increasing temporal weight emphasizes long-horizon prediction.

Direct regression

With dynamics model $M$0, direct regression minimizes

$M$1

This objective predicts the future trajectory directly from the observation and sparse tracks.

Flow matching

For each point, a Gaussian source trajectory is sampled:

$M$2

At flow time $M$3,

$M$4

The target velocity is

$M$5

The flow-matching objective is

$M$6

At inference, trajectories are generated from noise using four deterministic forward-Euler updates.

JiT-style prediction

The JiT-style objective predicts the denoised target trajectory,

$M$7

with implied velocity

$M$8

The denominator $M$9 is clipped to at least $K$0 during training. Time values are sampled using a logit-normal distribution, with $K$1 explicitly sampled with probability $K$2. At test time, a deterministic logit-normal quantile grid with $K$3 and $K$4 is used.

The reported results show that flow matching and JiT-style training generally outperform direct regression. Their advantage is consistent with multimodal future dynamics: a given RGB-D observation and sparse trajectory can admit multiple physically plausible completions. “Oracle-10” results, which select the best of ten generated samples using ground truth, measure distributional coverage rather than deployable prediction quality.

4. Transformer architecture

PointZero is a diffusion transformer that combines point geometry, sparse trajectories, visual features, and noisy future trajectories. Its implementation uses ViT-Base-sized transformer components with token dimension $K$5, $K$6 attention heads, and $K$7 transformer layers. A three-layer Perceiver-IO visual encoder compresses image features into $K$8 visual query tokens in the pre-training checkpoints.

Point tokens

For each observed point, the initial coordinate is repeated across the future horizon:

$K$9

A shared multilayer perceptron embeds this representation:

$A$0

These tokens encode the initial spatial configuration and provide cross-attention context for trajectory prediction.

Sparse-track tokens

Each sparse trajectory coordinate is embedded with a shared MLP and augmented with temporal and trajectory-index embeddings:

$A$1

The flattened sparse-track sequence is

$A$2

The learned time embedding $A$3 and trajectory-index embedding $A$4 allow the transformer to distinguish temporal position from the identity of each conditioning trajectory.

Visual tokens

Dense DINOv2 features are extracted from the RGB image:

$A$5

A Perceiver-IO module compresses these features. Learned latent queries

$A$6

are processed using self-attention over the latent sequence and cross-attention from the latent queries to $A$7:

$A$8

During downstream post-training, DINO visual conditioning is omitted for efficiency.

Query tokens and attention

For each observed point, the current noisy future trajectory $A$9 is concatenated with the repeated initial position $P_{2:T}$0:

$P_{2:T}$1

where

$P_{2:T}$2

A diffusion-time embedding is added to each query. The context sequence is

$P_{2:T}$3

The transformer alternates self-attention and cross-attention layers. Point queries form the query stream, while point, sparse-track, and visual tokens provide context. The architecture does not use an explicitly specified spatial graph, voxel representation, or dedicated local message-passing kernel. Spatial and temporal structure is encoded through point tokens, trajectory tokens, time embeddings, repeated initial coordinates, and global transformer attention.

The principal architectural properties are parallel full-trajectory prediction, mixed geometric and visual conditioning, flexible global attention, and compatibility with both deterministic regression and generative trajectory objectives.

5. Synthetic training data

PointZero introduces a synthetic 4D dataset containing approximately 2.9 million image frames with dense point trajectories. The data cover deformable objects, articulated objects, rigid objects, and collisions. The reported mixture is approximately $P_{2:T}$4 for deformable, articulated, and rigid examples, with sampling probabilities of approximately $P_{2:T}$5, $P_{2:T}$6, and $P_{2:T}$7, respectively.

Evaluation is reported on approximately 32,000 held-out scenes. Exact train, validation, and test split cardinalities are not specified in the supplied material.

Deformable objects

The deformable subset procedurally generates towels, T-shirts, and shorts. Mesh scale and proportions are randomized. NVIDIA FleX simulates cloth dynamics, with one or two action trajectories applied through kinematic constraints. Interactions include folding, lifting, dropping, and pushing. Stiffness, drag, and other physics parameters are randomized. Blender renders scenes with randomized cloth and background textures from PolyHaven.

For deformable objects, the conditioning trajectories are the action trajectories used as FleX kinematic constraints.

Articulated objects

Articulated objects are drawn from PartNet-Mobility and simulated in Genesis. A revolute or prismatic joint is selected, and a random feasible configuration is generated. Samples with negligible 2D optical flow are rejected to remove visually uninformative motion.

For articulated objects, sparse conditioning tracks are selected from the $P_{2:T}$8 points with the largest displacements.

Rigid objects

Rigid-body scenes are generated with a modified Kubric engine. These scenes contain up to three objects rather than up to 23 in the original setting. The data target rigid motion, collisions, and multi-object dynamics. Sparse tracks are again selected from points with large displacement.

Cameras, camera positions, and intrinsics are randomized. Each example begins with a single-view RGB-D observation, exposing the model to viewpoint variation and partial visibility.

6. Evaluation, transfer, and limitations

PointZero is evaluated using final-timestep point clouds. The reported metrics include mean distance error, mean squared error, bidirectional Chamfer distance, and Earth Mover’s Distance:

$P_{2:T}$9

$\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$0

The baselines are GBND, ParticleFormer, PGND, and PTv3. PointZero outperforms every listed baseline across the reported synthetic deformable, articulated, and rigid categories and metrics. On synthetic data, the PointZero flow-matching and JiT variants outperform direct regression. For example, on rigid scenes, PointZero regression obtains an MDE of $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$1 cm, compared with $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$2 cm for flow matching using ten mean samples and $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$3 cm for flow matching with oracle selection.

The real-world zero-shot evaluation contains 124 human-object interactions involving 14 objects: six articulated, five deformable, and three rigid. FoundationStereo provides depth and CoTracker3 supplies image point tracks. Human contact points are manually annotated in the first frame and forward-propagated to produce action-like conditioning tracks. PointZero-FM and PointZero-JiT outperform the baselines on 11 of 12 reported category–metric combinations. PTv3 achieves the best rigid-object MSE, $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$4, compared with $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$5 for PointZero-JiT.

Action-conditioned dynamics

For downstream scene-specific dynamics, sparse point-track conditioning is replaced by robot end-effector conditioning. Each end-effector token contains a 6-DoF pose and gripper state, embedded by a new MLP:

$\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$6

The compatible PointZero components are initialized from the pre-trained checkpoint and fine-tuned with JiT training and LoRA.

On the six-scene PGND benchmark—Bread, Paperbag, Cloth, Box, Rope, and Sloth—PointZero fine-tuning outperforms application-specific baselines on four of six scenes and substantially outperforms PointZero trained from scratch. In a zero-shot PGND evaluation, PointZero-FM-ZS obtains an average MDE of approximately $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$7 cm, compared with approximately $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$8 cm for the strongest baseline, PTv3.

Imitation learning

For imitation learning, sparse-track conditioning is removed and an action head $\mathbf{k}\cdot\mathbf{v}_{\mathrm{gr}}=0$9 is added to predict an end-effector action trajectory. The model can jointly predict actions and point trajectories:

$$I\in\mathbb{R}^{H\times W\times 3},\qquad D\in\mathbb{R}^{H\times W},\qquad M\in\{0,1\}^{H\times W},\qquad K\in\mathbb{R}^{3\times 3}.$$0

The protocol uses 20 expert demonstrations with action labels per task and 100 additional expert videos without action labels. The evaluated tasks are Blockstack, Microwave opening, Glass righting, Drawer opening, Cup/glass righting, Paper pickup and disposal, and Sock folding. PointZero is best or tied for best on six of seven tasks. With point-track supervision during downstream training, average simulation success increases from $$I\in\mathbb{R}^{H\times W\times 3},\qquad D\in\mathbb{R}^{H\times W},\qquad M\in\{0,1\}^{H\times W},\qquad K\in\mathbb{R}^{3\times 3}.$$1 to $$I\in\mathbb{R}^{H\times W\times 3},\qquad D\in\mathbb{R}^{H\times W},\qquad M\in\{0,1\}^{H\times W},\qquad K\in\mathbb{R}^{3\times 3}.$$2 when comparing the scratch and pretrained recipes reported in the paper.

Limitations

PointZero has several stated limitations. The synthetic-to-real gap remains significant because synthetic data do not fully model real materials, contact dynamics, clutter, hand-object interaction, sensor noise, or long-horizon behavior. Point-track completion does not explicitly encode forces, contact locations, friction, or robot embodiment. A single RGB-D view leaves occluded geometry and latent physical properties underdetermined.

The sparse conditioning interface is also simplified. Pre-training uses point tracks, whereas action-conditioned post-training uses end-effector pose and gripper state. Richer interaction representations may be required for complex manipulation. Downstream experiments cover seven manipulation tasks and do not establish broad multi-task robot generalization.

Generative evaluations using best-of-ten oracle selection should not be interpreted as deployable inference performance because the selection uses ground truth. The model may generate multiple plausible futures, creating a downstream requirement for trajectory selection, planning, or uncertainty-aware control.

PointZero’s principal contribution is therefore a transferable representation and pre-training strategy rather than a complete robotic world model. Its central proposition is that dense 3D motion prediction conditioned on sparse observable trajectories can provide a robot-independent dynamics prior. The reported results support transfer to action-conditioned prediction and imitation learning, while the remaining limitations concern physical contact modeling, partial observability, synthetic-to-real transfer, and practical selection among multimodal futures.