Point4D: Long-range 4D Motion Reconstruction

Abstract: We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io

• The paper introduces Point4D, a method that replaces 2D image-plane queries with 3D coordinate queries for 4D motion reconstruction, significantly improving handle occlusions and boundary challenges.
• Point4D’s approach enhances long-range tracking with key metrics such as End Point Error (EPE) reduced to 0.616 on PointOdyssey and survival rates improving to 0.514, showcasing its effectiveness compared to baseline methods.
• A crucial advantage of Point4D is that it provides a generally comparable single-chunk accuracy while especially excelling in long-video tracking conditions, where feed-forward and iterative baselines exhibit performance decline.

Point4D addresses a specific scalability failure in feed-forward 4D reconstruction: existing models can infer 3D motion efficiently over short video windows, but their 2D-query formulations are difficult to chain when a tracked point becomes occluded or leaves the field of view. The central claim is that long-range reconstruction is enabled not primarily by a larger temporal encoder, but by changing the representation used to propagate point identity. Instead of querying a source pixel, Point4D queries a 3D point together with a visual descriptor extracted from any frame in which that point is visible. This permits the predicted 3D endpoint of one chunk to be passed directly into the next chunk without reprojection or image-space correspondence search (2609.09145).

Problem formulation and motivation

The target task is dense 4D reconstruction from monocular video: for query points initialized in a reference frame, estimate their 3D positions over time, expressed in a common coordinate system. The difficulty is not merely estimating motion within a short temporal window. Long sequences introduce two coupled problems. First, the self-attention cost of jointly encoding hundreds of frames is prohibitive. Second, trajectory continuation requires preserving point identity across chunk boundaries, including cases in which the point is occluded or outside the camera’s field of view.

Prior feed-forward methods such as TraceAnything, Any4D, 4RC, and V-DPM generally decode motion from image-plane queries. A point is associated with a pixel, an image patch, and temporal and camera indices; its 3D position at a target time is then predicted by a query decoder. This design is well suited to short-range inference, but it makes chunk handoff dependent on visibility. To continue a trajectory, the predicted endpoint must typically be projected into a later frame and re-queried at a pixel location. Projection is undefined when the point is occluded or outside the image, and even when visibility is preserved, camera and depth errors introduce additional correspondence noise.

Point4D replaces this image-plane dependency with a 3D query. Its conceptual distinction is summarized below.

Property	2D-query reconstruction	Point4D
Query identity	Source pixel and image patch	3D coordinate and visual descriptor
Handoff between chunks	Reprojection or trajectory matching	Direct 3D re-query
Occluded point at handoff	Query may be undefined	Query remains geometrically defined
Out-of-frame point	Requires extrapolation or clipping	Propagated through transformed 3D coordinate
Descriptor source	Usually source frame	Any frame in which the point is visible
Decoder output	3D point at a target time	3D point at a target time

The implication is important: the method separates point identity from instantaneous image-plane visibility. This does not make an unobserved point fully observable; it instead avoids making visibility a prerequisite for representing and propagating the query.

Figure 1: Point4D chains 3D motion predictions across overlapping chunks, whereas 2D-query methods require image reprojection or matching at chunk boundaries.

Model architecture

Point4D uses a ViT-based visual geometry encoder initialized from Depth Anything 3. Given a video chunk, the encoder produces patch-level scene tokens, camera tokens, temporal tokens, depth maps, and camera poses. These outputs provide the geometric and temporal context required by a lightweight cross-attention motion decoder.

A query is defined by a 3D coordinate $\mathbf{p}$ in the camera coordinate system of a source time, source, target, and camera indices, and a local visual descriptor $S$. The point coordinate is obtained by unprojecting a visible image query using the predicted depth at the relevant source time. Crucially, $S$ need not come from the source time: it is extracted from an arbitrary reference frame in which the point is visible. Consequently, the source coordinate and appearance descriptor need not correspond to the same image-plane observation.

The query embedding combines sinusoidal positional encoding of the 3D coordinate with embeddings of the temporal and camera indices and the visual descriptor. The decoder cross-attends independently to the encoded scene representation and predicts the queried point’s 3D position at the target time. Queries do not self-attend, which permits arbitrary query sets and flexible batching at inference.

Figure 2: The decoder combines a 3D coordinate query with a descriptor extracted from any frame where the point is visible, decoupling trajectory prediction from source-frame visibility.

Training uses a signed log-transformed L1 loss on 3D positions, reducing the influence of distant points. Confidence weighting modulates the point loss, while auxiliary losses enforce 2D reprojection consistency and target-time visibility prediction. The training mixture contains dynamic and static data, with static points treated as stationary trajectories. Dynamic training data include PointOdyssey, Dynamic Replica, BEDLAM2, Kubric, Waymo DriveTrack, and related datasets; static data include ScanNet, ScanNet++, BlendedMVS, Co3D, and WildRGBD. Each sample contains 16–64 frames, whereas the principal evaluation tests substantially longer sequences.

The descriptor design is not a cosmetic modification. The ablation isolates two factors: replacing 2D coordinates with 3D coordinates, and allowing the descriptor to originate from an arbitrary visible frame. On PointOdyssey long-video tracking, the full formulation reduces EPE from 0.891 for the 2D-query variant and 0.869 for the 3D-source-patch variant to 0.616, while increasing survival from 0.283 and 0.380 to 0.514. On Dynamic Replica, it reduces EPE to 0.155 and raises survival to 0.812, compared with 0.712/0.422 for 2D queries and 0.825/0.266 for 3D queries restricted to source-frame patches. Thus, both geometric query decoupling and visibility-agnostic descriptor selection are necessary for the reported long-range gains.

Autoregressive trajectory chaining

The long-video procedure partitions a sequence into overlapping chunks of 48 frames with an 8-frame overlap in the principal experiments. Each chunk is encoded independently. Adjacent chunk coordinate systems are aligned using a dense-depth-based Sim(3) transformation estimated over the shared frames. The predicted endpoint of each query in the current chunk is transformed into the coordinate system of the next chunk and used directly as its new 3D query. The appearance descriptor extracted at initialization is reused throughout the entire sequence.

This operation differs fundamentally from reproject-and-query schemes. No pixel location is required at the handoff, and no independent trajectories need to be matched. A point that is occluded in the overlap still has a valid 3D coordinate; a point outside the field of view can likewise be propagated geometrically. The approach therefore converts temporal correspondence across chunks into a coordinate-frame alignment problem plus decoder prediction.

The method nevertheless depends on the quality of the geometric subsystem. Errors in depth and adjacent-chunk Sim(3) alignment can compound over repeated handoffs. Moreover, the chunk encoder has no persistent feature memory, so a point that remains both occluded and unsupported by the current chunk’s observations must be inferred from the query coordinate and reused descriptor alone.

Long-video tracking results

The main evaluation uses 200-frame PointOdyssey and Dynamic Replica sequences and 150-frame TAPVid3D Panoptic Studio sequences. Performance is measured using EPE, APD at thresholds from 0.1 to 1.0 meters, and survival rate. Point4D is compared with feed-forward 4D reconstructors and iterative 3D trackers.

The strongest results are obtained on the long-video setting, where repeated chunk handoffs are required.

Dataset	Method	EPE $\downarrow$	APD $\uparrow$	Survival $\uparrow$
PointOdyssey	Point4D	0.616	0.585	0.514
PointOdyssey	SpatialTrackerV2	0.498	0.611	0.477
Dynamic Replica	Point4D	0.155	0.856	0.812
Dynamic Replica	TAPIP3D	0.185	0.806	0.748
Panoptic Studio	Point4D	0.236	0.731	0.664
Panoptic Studio	TAPIP3D	0.230	0.741	0.622

Point4D outperforms every feed-forward baseline on the reported long-video metrics and obtains the best average rank across the compared methods. It does not dominate every individual metric against iterative trackers: SpatialTrackerV2 has lower EPE and higher APD on PointOdyssey, while TAPIP3D has slightly lower EPE and higher APD on Panoptic Studio. However, Point4D has higher survival on all three datasets than those iterative trackers. This distinction matters because survival directly measures the ability to preserve a track over the sequence rather than merely minimizing error on frames before failure.

Figure 3: On 200-frame sequences, Point4D preserves dense trajectories across occlusion and field-of-view exits, while 2D-query chaining loses correspondence and SpatialTrackerV2 produces a sparser output.

The Dynamic Replica result is particularly strong for the feed-forward comparison: Point4D reaches EPE 0.155, APD 0.856, and survival 0.812, compared with 4RC’s 0.336, 0.733, and 0.654 and V-DPM’s 0.386, 0.666, and 0.587. The implication is that the proposed handoff mechanism, rather than merely a stronger short-window decoder, substantially improves trajectory persistence under repeated chaining.

The same pattern remains when evaluation is restricted to dynamic points. On PointOdyssey, Point4D achieves EPE 0.617, APD 0.566, and survival 0.474, compared with 4RC’s 0.879, 0.445, and 0.355. On Dynamic Replica, it reaches 0.200, 0.794, and 0.717, compared with 4RC’s 0.359, 0.698, and 0.598. These results rule out an explanation in which the advantage arises primarily from tracking static background points through camera motion.

The extended-horizon evaluation is more demanding. With the same 48-frame chunks and 8-frame overlaps, Point4D is evaluated on 500-frame PointOdyssey sequences and 300-frame Dynamic Replica sequences. It ranks first on every reported metric:

Dataset	EPE $\downarrow$	APD $\uparrow$	Survival $\uparrow$
PointOdyssey, 500 frames	0.972	0.482	0.387
Dynamic Replica, 300 frames	0.174	0.836	0.786

At these lengths, Point4D surpasses both feed-forward baselines and iterative trackers. This result supports the paper’s claim that the error accumulation induced by repeated 2D reprojection is more damaging than the per-chunk accuracy gap between methods. It should still be interpreted under the paper’s evaluation assumptions: trajectories and predictions are globally median-scale aligned, and chunk alignment relies on predicted depth.

Single-chunk accuracy and visibility-specific behavior

The single-chunk experiments isolate decoder quality from chaining. On sequences of up to 64 frames, Point4D performs competitively but is not uniformly superior. For example, on LSFOdyssey, its EPE is 0.27, compared with 0.14 for V-DPM and 0.16 for 4RC. On Dynamic Replica, Point4D obtains EPE 0.09 and APD 0.91, compared with 0.07/0.95 for 4RC and 0.07/0.93 for Any4D. On Panoptic Studio, it reaches 0.23/0.73, compared with V-DPM’s 0.17/0.79.

Figure 4: Point4D produces coherent 3D trajectories in single-chunk real-world sequences, although its short-range accuracy is generally comparable rather than uniformly superior.

These results are methodologically important. They indicate that the long-video advantage does not derive from a universally more accurate single-chunk motion predictor. Instead, it appears when the same decoder is used in a chaining regime where query representation determines whether point identity survives handoffs.

The visibility breakdown gives a more direct test of the proposed mechanism. On PointOdyssey, Point4D achieves EPE 0.548 for visible points and 0.597 for occluded points, compared with 4RC’s 0.672 and 0.915. On Dynamic Replica, the corresponding values are 0.140 and 0.202 for Point4D versus 0.290 and 0.574 for 4RC. On Panoptic Studio, Point4D obtains 0.228 and 0.265, compared with 0.354 and 0.443.

The largest improvements occur for occluded points, exactly where image-space re-querying is ill-defined. For out-of-frame points, the advantage is smaller: Point4D’s EPE is 1.147 on PointOdyssey, while 4RC obtains 1.097. This is a meaningful qualification. A 3D query remains syntactically valid outside the field of view, but if the current chunk contains no visual evidence for the point, the decoder cannot recover its motion reliably. In that regime, a 2D baseline’s clipped or incorrect image query can occasionally produce a smaller numerical error by chance.

Error propagation across chunks

Per-chunk analysis shows that Point4D’s advantage increases with the number of handoffs. The method degrades more slowly than 2D-query baselines on both Dynamic Replica and PointOdyssey. V-DPM can begin with better first-chunk accuracy, yet Point4D surpasses it after several chunks. This observation separates initialization quality from temporal stability: the relevant property for long-range reconstruction is not only the error of a single prediction but also how the handoff mechanism transforms that error into the next query.

Figure 5: Point4D’s chunk-wise APD and EPE degrade more slowly than competing methods as repeated trajectory chaining proceeds.

The result is consistent with the underlying error model. Reprojection-based methods repeatedly expose trajectories to camera-pose error, depth error, visibility ambiguity, and incorrect pixel association. Point4D removes the reprojection step, but it does not eliminate geometric drift: the query coordinate must still be transformed by estimated Sim(3) alignments, and the next chunk must infer motion from its own scene representation. The empirical finding is therefore not that 3D chaining is error-free, but that its accumulated error is lower under the tested conditions.

Efficiency and query scaling

Point4D combines a chunked encoder with a query-based decoder. Runtime therefore has two distinct scaling behaviors. Increasing the number of input frames increases encoder cost, while increasing the number of queries increases decoder cost. Dense DPT-style baselines pay for dense prediction even when only a small number of points are requested; Point4D can decode sparse queries in proportion to their number.

Figure 6: Point4D is efficient for sparse queries and remains executable for dense queries, whereas SpatialTrackerV2 exhausts memory at the largest tested query count.

On a single 48 GB A6000 at $294 \times 518$ resolution, Point4D is reported as the second-fastest method when the number of input frames is varied from 16 to 64 with 100 queries. Its runtime increases with the number of queries, whereas dense feed-forward decoders have approximately query-independent cost. This trade-off is favorable for sparse tracking, but it means that Point4D is not automatically the most efficient choice for dense reconstruction. At 5,000 queries, SpatialTrackerV2 runs out of memory, while Point4D completes inference, although its runtime has increased relative to sparse-query operation.

The efficiency claim is therefore conditional: Point4D provides flexibility and favorable sparse-query scaling, not uniformly lower computational cost than dense decoders. Its long-video feasibility comes principally from chunking and its correspondence formulation, while its query decoder determines the cost of the requested output density.

Limitations and open questions

The method’s most direct limitation is the absence of persistent scene or feature memory between chunks. Only the query’s transformed 3D coordinate and its visual descriptor are transferred. If a point remains occluded or outside the field of view for an entire chunk, the current encoder has no direct observation of it. The decoder must extrapolate from the query and the chunk-level scene representation, and the paper explicitly acknowledges that reliability can deteriorate in this situation.

A second limitation is geometric error accumulation. Point4D depends on predicted depth for query initialization and on Sim(3) alignment for coordinate propagation. Errors in either component can compound over multiple chunks. The method also uses global scale alignment during evaluation, which reduces sensitivity to scale ambiguity; deployment without such normalization may expose additional drift.

The evaluation is predominantly benchmark-based, with substantial synthetic or controlled data in training and testing. Although real-world sequences such as Panoptic Studio and DAVIS are included, the paper does not establish robustness across arbitrary monocular videos with severe illumination changes, nonrigid deformation, long-term disappearance, or substantial domain shift. A specific open question is whether a persistent memory mechanism can improve trajectories that receive no visual support for several consecutive chunks without sacrificing the feed-forward and query-flexible properties of Point4D.

Conclusion

Point4D reformulates long-range 4D motion reconstruction around 3D coordinate queries rather than image-plane queries. This representation allows predicted endpoints to be transferred directly across overlapping chunks and permits visual descriptors to be reused from frames where the point is visible. The resulting system achieves strong long-video performance: on 200-frame benchmarks it outperforms all feed-forward baselines, and on 500-frame PointOdyssey and 300-frame Dynamic Replica sequences it ranks first across EPE, APD, and survival. Its short-range decoder is competitive rather than universally dominant, which strengthens the interpretation that the principal contribution is stable trajectory chaining. The remaining challenge is maintaining reliable motion estimates when points lack visual support for entire chunks and when depth and Sim(3) alignment errors accumulate over extended sequences.