OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields

Abstract: Previous feed-forward 4D reconstruction methods either predict per-frame static point clouds, ignoring foreground motion, or estimate point cloud trajectories while being limited to small camera motions. This restricts their ability to aggregate observations over time and reconstruct complete dynamic scenes under large viewpoint changes. To address this limitation, we propose OmniX, a feed-forward 4D reconstruction framework that predicts dense 3D point trajectories for every pixel from videos with large camera motion. OmniX decouples dynamic motion modeling from static geometry prediction and represents motion using a compact set of dynamic tokens. By leveraging the sparse and low-rank structure of 3D motion, these tokens generate trajectory fields for all pixels across all images while efficiently preserving global interactions. To facilitate training, we further build an automatic UE5-based 4D data engine and introduce a large-scale dataset containing 80K scenes and 1.28M multi-view videos with full geometric annotations. OmniX achieves state-of-the-art performance on dense 3D point trajectory prediction and 3D point tracking, while also demonstrating competitive results on video depth estimation and camera pose estimation.

• The paper introduces a novel feed-forward framework that predicts dense per-pixel 3D trajectories by disentangling static geometry and dynamic motion.
• It employs a Sparse Spatiotemporal Attention module with a deformable sampling head to capture global context and fine motion details across time.
• Empirical evaluations show state-of-the-art performance with lower End-Point Error (EPE) and improved tracking on both synthetic and real-world datasets.

OmniX: Any-View and Any-Time 4D Reconstruction via Feed-Forward Trajectory Fields

Introduction and Motivation

OmniX addresses a critical challenge in dynamic 4D scene reconstruction: robust, temporally and geometrically consistent recovery of dynamic 3D scenes from arbitrary multi-view video input, including sequences with substantial camera motion or temporally discontinuous content. Existing feed-forward 4D reconstruction methods largely focus on per-frame static geometry or trajectory estimation in constrained camera motion regimes, frequently resulting in inefficient temporal aggregation and suboptimal handling of dynamic scenes. OmniX proposes a unified, feed-forward framework that predicts dense, per-pixel 3D point trajectories for all pixels across all images—fully disentangling dynamic foreground motion from static scene geometry—enabling robust any-view, any-time 4D dynamic scene reconstruction even under large viewpoint shifts.

Methodology

OmniX builds on three technical pillars: a unified decomposition of static and dynamic components, a Sparse Spatiotemporal Attention (SSA) scheme, and a large-scale, synthetic training corpus. The architectural pipeline predicts per-pixel depth, ray maps, and camera parameters as static geometry, from which 3D scene points are generated. A trajectory module then predicts continuous rigid transformations for each 3D point across all views and times, forming their complete 4D spatiotemporal trajectories.

Figure 1: High-level architecture of OmniX, illustrating depth and ray prediction, camera parameter estimation, sparse dynamic token selection, SSA-powered trajectory basis prediction, and deformable upsampling for dense 3D trajectory field recovery.

Sparse Spatiotemporal Attention (SSA) and Trajectory Field Representation

OmniX models foreground motion using a small set of dynamically selected "tokens," which exploit the empirical low-rank sparsity of dynamic 3D motion. These dynamic tokens are projected across time using a learnable temporal embedding and are expanded into queries that cross-attend to all image tokens via a multi-head attention block, capturing global geometric and motion context efficiently. The outcome is a sparse trajectory field: for each scene, a compact set of learnable transformation bases parameterizes possible per-pixel 3D motions.

To achieve pixel-level predictions, a Deformable Trajectory Sampling Head (DTSH) applies learned sampling weights and offsets to upsample the sparse trajectory field, generating a dense trajectory map per scene. This mechanism maintains differentiability for dynamic token selection and enables trajectory supervision to directly drive dynamic point identification.

Loss Functions and Training Paradigm

Supervision is multi-task and confidence-aware, with pixel-wise regression targets on depth, ray, and trajectory fields, each weighted by uncertainty estimates. Additional gradient, pose, and binary cross-entropy losses regularize geometry and dynamics predictions. Notably, the entire system is trained end-to-end on a custom synthetic dataset built using Unreal Engine 5, yielding 80,000 richly labeled scenes and 1.28 million multi-view, multi-timestep video instances.

Experimental Evaluation

Extensive empirical evaluation demonstrates that OmniX establishes new state-of-the-art results on dense 4D trajectory field prediction and sparse 3D point tracking across both synthetic and real-world benchmarks.

Figure 2: Visualization of dense 3D point trajectory prediction across complex camera motion episodes; OmniX maintains high accuracy for both rigid and non-rigid targets across synthetic and real datasets.

Figure 3: Qualitative comparison of 3D point tracking on the DAVIS dataset—OmniX outperforming existing approaches in dynamic foreground/background separation and accurate motion modeling in challenging multi-object scenes.

Across all input modalities—monocular video, disjoint video pairs, and hybrid image-video sets—OmniX significantly outperforms prior art in both End-Point Error (EPE) and Average Point Distance (APD3D). For instance, on disjoint video pairs, OmniX achieves APD3D (ALL) of 0.444 and EPE (ALL) of 0.094, surpassing both TraceAnything and VDPM by substantial margins, particularly in dynamic contexts with large viewpoint changes. In point tracking benchmarks such as TAPVid-3D, OmniX consistently delivers the lowest EPE and highest APD3D across all splits, validating robustness to occlusion, viewpoint shift, and deformable motion.

In addition, OmniX exhibits strong generalization to downstream tasks. On the KITTI dataset, its monocular depth estimation achieves Abs Rel 0.024 and $\delta < 1.25$ at 99.0%. For video pose estimation on Sintel and TUM-dynamic, OmniX reports the lowest ATE and among the best RPE (trans/rot) errors, indicating accurate pose reasoning in dynamic content.

Ablation Analysis

An extensive ablation study illustrates the centrality of the proposed SSA design. Temporal embedding via the customized MLP outperforms adaptive layer normalization schemes, while inclusion of cross-attention is shown to be crucial for dynamic motion modeling—removal leads to near-static predictions. Self-attention among temporal dynamic tokens brings negligible benefit, justifying its omission. Performance gains scale with training data volume and dynamic token percentiles, affirming the design's scalability.

Figure 4: Qualitative ablation comparison for SSA components, highlighting degradation in dynamic motion fidelity when cross-attention or proper embedding are omitted.

Practical and Theoretical Implications

OmniX introduces a stepwise advance in scalable, efficient 4D scene reconstruction:

• Efficiency without Iterative Tracking: The SSA-driven, feed-forward design enables recovery of dense 3D trajectories in a single pass, vastly reducing computational cost compared to iterative trackers and supporting long input sequences.
• Disentangled Motion/Geometry Reasoning: By structurally separating dynamic and static inference streams, OmniX mitigates the deleterious interactions prevalent in prior art, enhancing accuracy and foreground-background discrimination in highly dynamic scenes.
• Synthetic Data Engine: The open framework for procedural 4D scene synthesis at scale, coupled with dense annotations, is poised to accelerate progress and standardization in dynamic scene understanding tasks.

Looking forward, OmniX paves the way for more universal 4D perception engines. Possible future directions include integration with multi-modal inputs (e.g., events or language), open-world generalization, and tighter coupling with downstream embodied or AR/VR applications requiring real-time, dynamic 4D representations.

Conclusion

OmniX presents a differentiated, feed-forward 4D reconstruction paradigm, combining sparse dynamic tokenization, spatiotemporal attention, and scalable synthetic data to realize robust, efficient, and highly accurate dynamic scene reconstructions under unconstrained viewing conditions. The results demonstrate strong empirical performance across multiple axes of dense and sparse trajectory estimation, reinforcing the method's practical impact and relevance for the next generation of dynamic geometry learning systems.