Point4D: Long-Range 4D Reconstruction

• Point4D is a feed-forward model designed for long-range 4D reconstruction from a single monocular video, focusing on robust 3D trajectory prediction.
• The model's central innovation combines a 3D query motion decoder with chunk-wise $\mathrm{Sim}(3)$ alignment, enabling direct 3D coordinate propagation, which substantially improves accuracy and robustness for long video sequences.
• Point4D’s performance is validated through extensive evaluation on dynamic and static datasets, consistently outperforming state-of-the-art methods in endpoint accuracy, survival rate, and other critical metrics across 200-, 300-, and 500-frame sequences.

Point4D is a feed-forward model for long-range 4D reconstruction from monocular video. It predicts dense per-point 3D trajectories across multi-hundred-frame sequences by representing query points in 3D rather than as image-plane pixels. Its central mechanism is a 3D query-based motion decoder: predicted 3D endpoints are transformed into the coordinate system of the next temporal chunk and directly reused as subsequent queries, eliminating reprojection, pixel matching, and visibility at the exact handoff frame. Point4D combines a visual geometry transformer, depth and camera-pose estimation, arbitrary-frame visual descriptors, chunk-wise $\operatorname{Sim}(3)$ alignment, and independently decoded 3D trajectories (Jeon et al., 8 Sep 2026).

1. Problem formulation and objectives

Point4D addresses long-range 4D motion reconstruction from a monocular video

$V=\{I_t\}_{t=0}^{T-1},$

where $I_t$ is an RGB frame. For query point $i$, the model predicts a temporally ordered 3D trajectory

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$

with all positions expressed in one globally aligned coordinate system.

The objective differs from conventional optical-flow or 2D point-tracking formulations. A conventional query is associated with a pixel coordinate $(u,v)$ in a source frame and must be reprojected or matched when temporal processing is divided into multiple chunks. Point4D instead defines a query by a 3D coordinate $\mathbf{p}$, together with source, target, and camera-frame indices and an appearance descriptor:

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$

The semantic query is: given a point at 3D position $\mathbf{p}$ in the coordinate system of $t_{\mathrm{src}}$, where is that point at $V=\{I_t\}_{t=0}^{T-1},$0, expressed in the coordinate system of $V=\{I_t\}_{t=0}^{T-1},$1?

This formulation is designed for points that are occluded, leave the image, or are not visible at a chunk boundary. A 3D coordinate remains a valid state variable even when no corresponding image pixel exists. The approach does not guarantee accurate prediction when a point is absent from an entire chunk, because no visual evidence is then available within that chunk.

The paper’s principal interpretation is that Point4D’s advantage on long videos arises primarily from robust cross-chunk propagation rather than from a uniformly superior single-chunk motion decoder. On short sequences, Point4D is competitive with modern feed-forward 4D methods; on long sequences, its direct 3D-query handoff substantially improves trajectory survival and endpoint accuracy (Jeon et al., 8 Sep 2026).

2. Scene encoding and geometric initialization

Visual geometry transformer

A video chunk containing $V=\{I_t\}_{t=0}^{T-1},$2 frames is processed by an encoder $V=\{I_t\}_{t=0}^{T-1},$3 that produces patch-level features, camera tokens, temporal tokens, depth maps, and camera poses:

$V=\{I_t\}_{t=0}^{T-1},$4

The scene representation is

$V=\{I_t\}_{t=0}^{T-1},$5

where

$V=\{I_t\}_{t=0}^{T-1},$6

is the set of visual patch tokens for frame $V=\{I_t\}_{t=0}^{T-1},$7. Here, $V=\{I_t\}_{t=0}^{T-1},$8 is the number of image patches and $V=\{I_t\}_{t=0}^{T-1},$9 is the feature dimension. The encoder also produces a camera token $I_t$0, a learned temporal token $I_t$1, a depth map $I_t$2, and a camera pose $I_t$3.

The architecture is based on visual geometry transformers with alternating frame-wise and global self-attention. The encoder and geometry heads are initialized from Depth Anything 3, whereas the motion decoder is trained from scratch. Temporal information is introduced through a normalized frame-time encoding:

$I_t$4

Depth is predicted through a DPT-style decoder,

$I_t$5

and camera poses through

$I_t$6

These predictions initialize 3D query coordinates and provide the transformations required for chunk alignment and camera-coordinate conversion.

Initial 3D queries

Suppose $I_t$7 is a pixel sampled in the initial frame $I_t$8. Its predicted depth is used to unproject the pixel into 3D:

$I_t$9

For a pinhole camera with intrinsic matrix $i$0,

$i$1

The initial query is then formed as

$i$2

The initial 3D point therefore depends on predicted depth. Errors in depth can corrupt the query location and consequently affect subsequent trajectory predictions.

3. 3D query-based motion decoding

Query representation

The 3D query replaces the image-plane coordinate used by prior query-based 4D models. Its spatial component is embedded using a sinusoidal positional encoding:

$i$3

followed by a learned projection to dimension $i$4.

The complete query embedding is conceptually

$i$5

where $i$6 embeds the appearance descriptor. The source-time token identifies the coordinate system of the input point, the target-time token specifies the requested time, and the camera token specifies the output coordinate system.

Arbitrary-frame appearance descriptors

The descriptor $i$7 need not be extracted from the source frame. Instead, Point4D selects a reference frame $i$8 in which the point is visible and extracts an RGB patch:

$i$9

The descriptor is reused across temporal chunks and is not re-extracted during handoff. This allows the model to retain appearance evidence even when the point is occluded or absent from the source frame. The reported ablations compare three formulations:

1. a 2D query;
2. a 3D query using only a source-frame patch;
3. the Point4D formulation using a 3D query and an arbitrary visible-frame descriptor.

The third formulation produces the strongest long-range results.

Cross-attention decoder

For a batch of $$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$0 queries, query embeddings are represented as

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$1

The scene features are flattened across the chunk:

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$2

The motion decoder applies cross-attention from the query embeddings to the scene representation:

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$3

The decoder is a lightweight D4RT-style cross-attention architecture. It does not use self-attention between queries; queries are decoded independently. This supports arbitrary query sets and flexible batching.

The decoder predicts a 3D endpoint, a confidence value, and a visibility logit. For query $$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$4,

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$5

For $$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$6 queries and $$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$7 target times, the predicted positions can be represented as

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$8

Visibility is estimated independently from trajectory position. A point may remain a valid 3D query even when the visibility prediction indicates that it is not visible at a target time.

4. Training objective

Point4D uses a total loss composed of point-position, confidence, reprojection, and visibility terms:

$$\mathcal{T}_i=\left\{\mathbf{P}_i(t)\in\mathbb{R}^3\right\}_{t=0}^{T-1},$$9

Signed-log 3D point loss

To reduce the influence of distant points, each coordinate is transformed using

$(u,v)$0

The point loss is an $(u,v)$1 loss in transformed coordinates:

$(u,v)$2

For $(u,v)$3 queries and $(u,v)$4 target times, the averaged loss is

$(u,v)$5

Confidence modulates the point loss so that uncertain predictions are penalized less heavily. The supplied description does not specify the exact scalar parameterization of this modulation.

Reprojection consistency

Predicted 3D points are projected into the target camera:

$(u,v)$6

The auxiliary reprojection loss is

$(u,v)$7

This term is used during training as a consistency constraint. It is not used for cross-chunk re-querying at inference.

Visibility loss

Let $(u,v)$8 denote whether point $(u,v)$9 is visible at target time $\mathbf{p}$0, and let $\mathbf{p}$1 be its predicted visibility logit. The visibility loss is binary cross-entropy:

$\mathbf{p}$2

Visibility is an output variable rather than a prerequisite for trajectory propagation. The system retains the point state even when the point is currently invisible.

5. Chunk-wise long-range reconstruction

Temporal partitioning

Long videos are divided into overlapping chunks:

$\mathbf{p}$3

The principal configuration uses 48-frame chunks with 8-frame overlap. The reported long-video settings include:

• PointOdyssey and Dynamic Replica: 200 frames;
• Panoptic Studio: 150 frames;
• extended PointOdyssey evaluation: 500 frames;
• extended Dynamic Replica evaluation: 300 frames.

Each chunk is encoded independently, avoiding the memory cost of jointly processing hundreds of frames.

Similarity alignment

Independent chunks have separate coordinate systems and potentially separate scales. Adjacent chunks are aligned using a similarity transformation:

$\mathbf{p}$4

where $\mathbf{p}$5 is scale, $\mathbf{p}$6 is rotation, and $\mathbf{p}$7 is translation. Thus,

$\mathbf{p}$8

The transformation is estimated from dense depth predictions on shared overlap frames using an Umeyama-style alignment:

$\mathbf{p}$9

Pairwise transformations are composed to place all chunks in a global coordinate system:

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$0

Repeated alignment introduces a potential source of accumulated error. Inaccuracies in depth, camera pose, or overlap registration can therefore affect long-range trajectories.

Direct 3D endpoint handoff

The central operation occurs at the boundary between chunks. Suppose point $$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$1 has query coordinate $$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$2 in chunk $$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$3. The decoder predicts its endpoint at the handoff frame $$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$4:

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$5

The endpoint is transformed into the coordinate system of the next chunk:

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$6

or, explicitly,

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$7

The transformed endpoint is inserted directly into the next query:

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$8

The handoff does not require:

• projection into image coordinates;
• selection of a matching pixel;
• trajectory matching between chunks;
• visibility in the overlap frame;
• re-extraction of the appearance descriptor.

This is the principal architectural distinction between Point4D and 2D-query alternatives. Point identity is propagated as a 3D coordinate state rather than as an image-plane location.

6. Training data, evaluation, and results

Training data and implementation

Point4D is trained on dynamic and static datasets.

Dynamic datasets include:

• PointOdyssey;
• Dynamic Replica;
• BEDLAM2;
• Kubric Movi-F;
• CoTracker-Kubric;
• Waymo DriveTrack;
• OmniWorld.

Static datasets include:

• ScanNet;
• ScanNet++;
• BlendedMVS;
• Co3Dv2;
• WildRGBD.

Static points are treated as stationary trajectories. OmniWorld lacks trajectory ground truth, so its queries are restricted to

$$q_{\mathrm{3D}}=(\mathbf{p},t_{\mathrm{src}},t_{\mathrm{tgt}},t_{\mathrm{cam}},S).$$9

reducing those samples to geometry and relative-pose supervision.

Training clips contain 16–64 frames, with image width sampled from 252 to 518 pixels. Each frame provides 750 query pixels, approximately 40% of which are sampled from edge regions detected with Sobel filtering. The source, target, and camera indices are sampled uniformly. In 40% of queries,

$\mathbf{p}$0

Dynamic-point losses are upweighted relative to static-point losses.

The optimization uses AdamW, a peak learning rate of $\mathbf{p}$1, a 10-epoch warm-up, cosine decay, and 150 training epochs on eight H100 GPUs. The pretrained encoder uses a learning rate scaled by $\mathbf{p}$2. Augmentations include color jitter, Gaussian blur, random rescaling, and aspect-ratio augmentation.

Metrics

For predicted trajectory position $\mathbf{p}$3 and ground-truth position $\mathbf{p}$4, endpoint error is

$\mathbf{p}$5

The average position accuracy at threshold $\mathbf{p}$6 is

$\mathbf{p}$7

with thresholds $\mathbf{p}$8 meters.

Long-video survival measures the fraction of the sequence completed before error exceeds a selected threshold:

$\mathbf{p}$9

Ground-truth and predicted trajectories are median-scale aligned. Long-video experiments use one global scale per sequence, whereas single-chunk dynamic evaluation follows the Any4D protocol and rescales each frame pair independently.

Long-video performance

On 200-frame PointOdyssey sequences, Point4D reports:

Method	EPE	APD	Survival
TraceAnything	2.147	0.121	0.070
Any4D	1.026	0.400	0.291
4RC	0.789	0.559	0.463
VDPM	0.736	0.559	0.461
Point4D	0.616	0.585	0.514

SpatialTrackerV2 obtains lower EPE on this benchmark, with EPE $t_{\mathrm{src}}$0, APD $t_{\mathrm{src}}$1, and survival $t_{\mathrm{src}}$2, whereas Point4D has better survival than the listed iterative methods.

On Dynamic Replica, Point4D is best on all three reported metrics:

Method	EPE	APD	Survival
TraceAnything	0.767	0.464	0.391
Any4D	0.364	0.711	0.629
4RC	0.336	0.733	0.654
VDPM	0.386	0.666	0.587
Point4D	0.155	0.856	0.812

On Panoptic Studio, Point4D obtains the highest APD and survival, while VDPM has lower EPE:

Method	EPE	APD	Survival
TraceAnything	0.665	0.408	0.312
Any4D	0.497	0.495	0.389
4RC	0.379	0.613	0.534
VDPM	0.280	0.719	0.634
Point4D	0.236	0.731	0.664

On longer sequences, Point4D maintains the strongest reported results. For 500-frame PointOdyssey sequences, it obtains EPE $t_{\mathrm{src}}$3, APD $t_{\mathrm{src}}$4, and survival $t_{\mathrm{src}}$5, compared with EPE $t_{\mathrm{src}}$6, APD $t_{\mathrm{src}}$7, and survival $t_{\mathrm{src}}$8 for 4RC. On 300-frame Dynamic Replica sequences, Point4D obtains EPE $t_{\mathrm{src}}$9, APD $V=\{I_t\}_{t=0}^{T-1},$00, and survival $V=\{I_t\}_{t=0}^{T-1},$01.

Single-chunk performance

Point4D is competitive rather than uniformly dominant when the entire sequence fits in one chunk. Reported values include:

• LSFOdyssey: EPE $V=\{I_t\}_{t=0}^{T-1},$02, APD $V=\{I_t\}_{t=0}^{T-1},$03;
• Dynamic Replica: EPE $V=\{I_t\}_{t=0}^{T-1},$04, APD $V=\{I_t\}_{t=0}^{T-1},$05;
• Panoptic Studio: EPE $V=\{I_t\}_{t=0}^{T-1},$06, APD $V=\{I_t\}_{t=0}^{T-1},$07.

VDPM and 4RC often obtain stronger single-chunk numbers. This supports the distinction between local motion-decoding accuracy and long-range chunk-propagation reliability.

Query-formulation ablations

On long-video PointOdyssey:

Query formulation	EPE	Survival
2D	0.891	0.283
3D, source patch	0.869	0.380
3D, arbitrary visible-frame descriptor	0.616	0.514

On Dynamic Replica:

Query formulation	EPE	Survival
2D	0.712	0.422
3D, source patch	0.825	0.266
3D, arbitrary visible-frame descriptor	0.155	0.812

These results indicate that both changes matter: replacing the 2D query with a 3D coordinate and allowing the appearance descriptor to come from an arbitrary visible frame.

Visibility-specific handoff behavior

Point4D’s EPE by handoff condition is:

Dataset	Visible	Occluded	Out of frame
PointOdyssey	0.548	0.597	1.147
Dynamic Replica	0.140	0.202	0.309
Panoptic Studio	0.228	0.265	0.409

The largest advantage over 2D-query methods occurs for occluded points. Out-of-frame points remain more difficult, particularly when they are absent throughout the next chunk. The results therefore distinguish between removing the requirement for visibility at the handoff and solving motion prediction without any visual evidence.

7. Significance, limitations, and relation to 4D point modeling

Point4D converts long-range 4D tracking from an image-plane correspondence problem into a coordinate-state propagation problem. A 2D formulation propagates a pixel and must repeatedly perform projection, matching, or pixel selection. Point4D propagates

$V=\{I_t\}_{t=0}^{T-1},$08

where $V=\{I_t\}_{t=0}^{T-1},$09 is a 3D state and $V=\{I_t\}_{t=0}^{T-1},$10 is an appearance descriptor extracted once from an arbitrary visible frame.

This design provides:

• a geometry-based point identity;
• appearance information independent of a single image-plane location;
• direct endpoint propagation across chunks;
• improved robustness to occluded handoffs;
• improved robustness when points leave the image;
• arbitrary sparse or dense query support.

The approach is related to broader 4D point-cloud and pointmap research. Trajectory-level refinement methods such as Auto4D separate persistent object geometry from time-varying pose and use sequential LiDAR observations for temporal refinement (Yang et al., 2021). Hierarchical point-cloud video backbones such as Point Primitive Transformer apply long-range attention to geometrically meaningful primitive tokens rather than directly to all raw points (Wen et al., 2022). Query-based panoptic systems such as Mask4Former represent spatio-temporal instances with learned queries and jointly predict masks and temporal associations (Yilmaz et al., 2023). Cross-view pointmap generation emphasizes shared 3D coordinate frames and geometric alignment across camera views (Liu et al., 1 Jul 2025). C4D combines pointmaps with optical flow, long-term tracking, mobility estimation, and trajectory smoothness for dynamic monocular reconstruction (Wang et al., 16 Oct 2025). These systems address different tasks and use different state parameterizations; Point4D is specifically concerned with long-range dense 3D trajectory reconstruction.

The principal limitations are:

1. No persistent scene or feature memory: only the 3D query coordinate and appearance descriptor are propagated between chunks.
2. Entire-chunk disappearance: if a point is occluded or out of frame for the full next chunk, the model lacks direct visual evidence.
3. Accumulated alignment error: repeated $V=\{I_t\}_{t=0}^{T-1},$11 transformations can compound depth, pose, and overlap-registration errors.
4. Dependence on depth initialization: incorrect initial depth corrupts the first 3D query.
5. No theoretical guarantee across unobserved intervals: 3D-query validity does not imply accurate motion prediction without supporting observations.
6. Query-dependent computation: unlike dense prediction heads, decoding cost increases with the number of requested points, although this is advantageous for sparse query sets.
7. Single-chunk trade-offs: the method’s strongest advantage concerns long-range chaining, not necessarily per-chunk endpoint accuracy.

The central contribution is therefore the combination of a 3D query-based decoder, arbitrary-frame appearance descriptors, and direct transformed-endpoint re-querying. By preserving point identity in 3D rather than in image coordinates, Point4D enables feed-forward reconstruction of dense trajectories across 200-, 300-, and 500-frame videos, with its strongest empirical gains appearing as the number of temporal handoffs increases (Jeon et al., 8 Sep 2026).