Any4D: Unified Feed-Forward Metric 4D Reconstruction (2512.10935v1)

Abstract: We present Any4D, a scalable multi-view transformer for metric-scale, dense feed-forward 4D reconstruction. Any4D directly generates per-pixel motion and geometry predictions for N frames, in contrast to prior work that typically focuses on either 2-view dense scene flow or sparse 3D point tracking. Moreover, unlike other recent methods for 4D reconstruction from monocular RGB videos, Any4D can process additional modalities and sensors such as RGB-D frames, IMU-based egomotion, and Radar Doppler measurements, when available. One of the key innovations that allows for such a flexible framework is a modular representation of a 4D scene; specifically, per-view 4D predictions are encoded using a variety of egocentric factors (depthmaps and camera intrinsics) represented in local camera coordinates, and allocentric factors (camera extrinsics and scene flow) represented in global world coordinates. We achieve superior performance across diverse setups - both in terms of accuracy (2-3X lower error) and compute efficiency (15X faster), opening avenues for multiple downstream applications.

• The paper introduces a unified feed-forward transformer that achieves dense, metric 4D reconstruction from heterogeneous multi-sensor inputs.
• It employs a factored representation that separates egocentric depth and allocentric scene flow to effectively model dynamic geometry and motion.
• Empirical results demonstrate state-of-the-art 3D tracking and scene flow performance with significantly faster inference compared to iterative methods.

Any4D: Unified Feed-Forward Metric 4D Reconstruction

Introduction

Any4D addresses the longstanding challenge of 4D ($3$D+$t$) reconstruction in computer vision: dense, metric-scale recovery of dynamic geometry and motion from potentially heterogeneous multi-sensor inputs. While prior work has achieved partial progress via sub-task specialization (depth estimation, scene flow, tracking, pose), and often with limitations such as requiring iterative optimization or being restricted to 2-frame or sparse settings, Any4D introduces a unified, feed-forward transformer that exploits multi-view and multi-modal input for robust, real-time, and dense metric 4D scene reconstruction.

Figure 1: Any4D’s unified architecture addresses key limitations of prior 4D reconstruction models regarding efficiency, scalability, and sensor flexibility.

Model Architecture and Factored 4D Representation

The core innovation in Any4D is a factored representation of the 4D scene which separates per-view egocentric information (depth maps, ray directions) and per-view allocentric information (camera extrinsics and forward scene flow), with a global metric scaling factor. This decomposition allows the model to leverage heterogeneous training datasets with differing annotation availability (e.g., datasets with only geometry, or only motion, or both, at various scales).

During inference, the model operates on $N$ frames, processing RGB images and optionally auxiliary modalities such as depth, IMU-based poses, and radar-measured Doppler velocities. The architecture comprises modality-specific encoders, a multi-view transformer backbone (alternating attention, 12×12-MHA/MLP blocks), and dedicated DPT-based heads for geometry and motion.

Figure 2: Any4D simultaneously predicts metric scale, egocentric depth and ray directions, and allocentric scene flow and camera trajectory for all input views.

The output is the set $$(\tilde{s}, \{\tilde{R}_i, \tilde{D}_i, \tilde{T}_i, \tilde{F}_i\}_{i=1}^N)$$ where $\tilde{s}$ is global scale, $\tilde{R}_i$ ray direction maps, $\tilde{D}_i$ depth per view, $\tilde{T}_i$ camera poses, and $\tilde{F}_i$ scene flow. This design enables direct composition of allocentric metric 3D geometry and motion across arbitrary view counts.

Training Methodology

The training regime leverages both large-scale (static and dynamic) datasets with mixed annotation completeness and enforces scale-normalized, multi-objective supervision (geometry, motion, pose, etc.). Multi-modal conditioning is simulated by probabilistic dropping of input channels and adding simulated Doppler observations (projection of egocentric scene flow to radial direction). Key practical considerations include partial supervision, scale-invariant loss formulation (addressing dataset heterogeneity), and initialization from pretrained map-based transformer models.

4-view random sampling during training is essential for generalization to arbitrary input counts, as 2-view training significantly degrades multi-frame inference performance.

Figure 4: 4-view training yields generalization to many frames; 2-view training results in significantly increased error as the number of inference views grows.

Empirical Results

3D Tracking

Any4D achieves state-of-the-art performance across dynamic 3D tracking benchmarks, including DriveTrack, Dynamic Replica, LSFOdyssey, and Parallel-Studio. Any4D yields the lowest end-point error (EPE), highest density (APD), and highest inlier ratios for both dynamic points and scene flow, outperforming both optimization-heavy trackers and other feed-forward multi-view architectures. Notably, inference is an order of magnitude faster (0.5s for 50-frame inference vs. 11s–146s for competitors), supporting real-time operation.

Figure 5: Any4D provides dense and precise motion estimation, in contrast to sparse or noisy outputs from state-of-the-art baselines.

Dense Scene Flow and Video Depth

On standardized scene flow tasks (e.g., Kubric-4D with static/dynamic camera, VKITTI-2), Any4D improves the average points-within-delta and EPE metrics by $2\!-\!3\times$ or more compared to the next best method. Additionally, on video depth estimation, Any4D matches or surpasses single-step feed-forward baselines while remaining competitive against iterative and task-specific models.

Multi-Modal Input and Ablations

The flexible input pipeline allows integrating auxiliary modalities (depth, pose, Doppler). The inclusion of geometry and Doppler consistently enhances performance for both motion and 3D tracking metrics. Direct prediction of allocentric scene flow is clearly optimal: regressing "points after motion" (the approach in some baselines) produces substantial errors and artifacts, especially at object boundaries and in background handling.

Figure 6: Parameterizing motion as allocentric scene flow leads to artifacts-free 4D reconstructions; alternative parameterizations (e.g., points after motion) suffer significant noise.

Limitations

The model requires the object of interest to be present in the reference (first) view, does not model sensor noise in auxiliary channels, and its generalization is bounded by the diversity of existing 4D training datasets. Scenarios with extreme camera motion or limited visual overlap may present failure modes, and signal the need for real-world sensor integration and richer data.

Practical and Theoretical Implications

Any4D’s design answers critical bottlenecks for downstream applications: its unified, real-time, and dense 4D reconstructions are directly useful for open-world video understanding, dynamic asset creation, predictive control in robotics, and integration as foundational models in AR/VR pipelines. The model's factored output parameterization establishes a blueprint for compositionality in 4D inference architectures, enabling exploitation of partial data and interoperability across hardware platforms.

Theoretically, the empirical superiority of allocentric scene flow as a regression target sets a new direction for future architectures. The single-pass, N-view transformer backbone and flexible integration of arbitrary sensor streams are well-aligned with anticipated requirements in embodied AI and heterogeneous-agent systems.

Conclusion

Any4D advances the state of 4D vision by combining efficiency, scalability, and flexibility into a single architecture for metric-dense feed-forward reconstruction. Empirical results confirm its dominance across dynamic tracking, scene flow, and depth estimation tasks, supported by substantial acceleration over prior art. The model’s design—factored output representations, multi-modal conditioning, and direct allocentric motion prediction—point toward robust generalization and immediate utility in emerging application domains. As such, Any4D provides a credible foundation for future real-time, large-scale 4D dynamic scene understanding and manipulation tasks (2512.10935).