UFO-4D: Unposed Feedforward 4D Reconstruction from Two Images

Abstract: Dense 4D reconstruction from unposed images remains a critical challenge, with current methods relying on slow test-time optimization or fragmented, task-specific feedforward models. We introduce UFO-4D, a unified feedforward framework to reconstruct a dense, explicit 4D representation from just a pair of unposed images. UFO-4D directly estimates dynamic 3D Gaussian Splats, enabling the joint and consistent estimation of 3D geometry, 3D motion, and camera pose in a feedforward manner. Our core insight is that differentiably rendering multiple signals from a single Dynamic 3D Gaussian representation offers major training advantages. This approach enables a self-supervised image synthesis loss while tightly coupling appearance, depth, and motion. Since all modalities share the same geometric primitives, supervising one inherently regularizes and improves the others. This synergy overcomes data scarcity, allowing UFO-4D to outperform prior work by up to 3 times in joint geometry, motion, and camera pose estimation. Our representation also enables high-fidelity 4D interpolation across novel views and time. Please visit our project page for visual results: https://ufo-4d.github.io/

• The paper introduces a unified feedforward model that employs dynamic 3D Gaussian splatting to jointly predict dense geometry, scene flow, and relative camera pose from two unposed images.
• It leverages a differentiable rasterization process for 4D spatiotemporal interpolation, ensuring robust performance and effective cross-task regularization under sparse, noisy supervision.
• The method achieves significant improvements, including more than threefold lower scene flow errors and enhanced pose regression, demonstrating its practical applicability in dynamic scene understanding.

Unposed Feedforward 4D Reconstruction from Two Images

Introduction and Motivation

UFO-4D introduces a unified feedforward model for dense 4D dynamic scene reconstruction, leveraging only two unposed images as input. Unlike previous methods that typically require multi-stage processing pipelines, test-time optimization, or rely on specialized heads for disparate tasks, UFO-4D adopts a single explicit dynamic 3D Gaussian Splatting (D-3DGS) representation in a fully feedforward architecture. This joint representation directly yields consistent 3D geometry, 3D motion (scene flow), and relative camera pose, while also supporting view and temporal interpolation without requiring extrinsic pose priors or post-processing. The framework is motivated by the observation that rendering dense signals from a unified set of 3D geometric primitives enables cross-task regularization, crucial in the context of sparse and noisy 4D ground-truth supervision prevalent in real-world datasets.

Figure 1: Given a pair of unposed images, the UFO-4D method outputs dynamic 3D Gaussians and relative camera pose in a feedforward manner, supporting geometry/motion estimation and 4D interpolation.

Methodology

UFO-4D employs a ViT-based encoder-decoder architecture with shared weights for image tokenization, incorporating learnable pose and intrinsic tokens. The network predicts, in a single forward pass, both: (1) a set of parameterized dynamic 3D Gaussians (per-pixel, per-frame); and (2) the relative pose between the input views. Each Gaussian is described by a 3D center $\boldsymbol\mu$, 3D velocity vector $\mathbf{v}$ modeling local scene flow, covariance (size and orientation), view-dependent color encoded with spherical harmonics, and opacity as a confidence factor. The entire set of Gaussians is defined in the canonical coordinate frame of the first image.

Figure 2: Network architecture—two input images and camera intrinsics are encoded, and the output heads predict Gaussians and camera pose; the representation is fully explicit and differentiable.

The core innovation is a differentiable rasterization procedure: Gaussians are interpolated linearly in time for $t' \in [t, t+1]$, allowing for rendering novel frames between the two inputs. Rasterization supports not only color images but also dense 3D points and scene flow, facilitating direct photometric and geometric supervision on both rendered images and structure/motion maps. The rendering engine appropriately handles occlusion/disocclusion via depth-ordered alpha blending and the dynamic translation of Gaussian centers.

Figure 3: Dynamic rasterization; Gaussians are translated according to their motion vectors, enabling synthesis of 3D geometry and motion at arbitrary intermediate timestamps.

Losses comprise a supervised component (matching ground-truth points, flows, and pose) and self-supervised terms (photometric/Lpips error on rendered views, and spatial smoothness on point and motion maps, modulated by image gradients). The tight coupling between appearance, geometry, and motion via the shared Gaussians allows each supervisory signal to act as regularizer for the others, mitigating annotation sparsity and noise.

Analysis of Representation and Emergent Properties

The opacity parameter of each Gaussian is shown to emerge as a learnable confidence metric. In regions of occlusion/disocclusion, or where appearance changes are localized, the network learns to boost the corresponding Gaussians' opacity, essentially focusing reconstruction capacity on critical scene elements. Conversely, redundant or ambiguous Gaussians are suppressed, leading to efficient scene representation.

Figure 4: Opacity as learnable confidence—the model autonomously modulates contribution of Gaussians based on visibility and consistency across views.

The approach also exhibits robustness to dataset domain shift and noisy labels, validated empirically across heterogeneous datasets (Stereo4D, KITTI, Bonn, Sintel). Notably, on benchmarks with significant annotation noise, such as Stereo4D, UFO-4D's reliance on dense image-based self-supervision yields superior 3D motion/background separation relative to methods limited by sparse ground-truth.

Strong Empirical Results and Novel Insights

UFO-4D achieves substantial improvements over direct competitors (DynaDUSt3R, ZeroMSF, St4RTrack, MonST3R) across all major geometry and motion metrics. On Stereo4D and KITTI, the method reports more than threefold lower end-point errors (EPE) in scene flow estimation and achieves state-of-the-art absolute relative and depth inlier scores (see main Table 1 and Table 2). Pose regression accuracy is also enhanced compared to iterative PnP-solvers, with the direct head greatly outperforming RANSAC-based alternatives.

The unified model supports simultaneous estimation and decomposition of geometry/motion, outperforming per-point and per-frame specialized representations in spatial consistency, especially in the presence of ambiguous motion boundaries and occlusions. The representation supports consistent 4D spatiotemporal interpolation; given two frames, the model renders novel images, pointmaps, and motion at any intermediate timestamp and arbitrary camera viewpoint—capabilities not matched by task-specific or unidirectional motion estimation methods.

Figure 5: 4D interpolation—the model synthesizes realistic images, geometry, and motion between two frames, with accurate object segmentation and motion attribute interpolation.

Ablation studies highlight the necessity of differentiable rasterization and dense photometric loss for overall performance. Architectural comparisons (per-pixel vs. dynamic 3DGS) demonstrate the advantage of the unified dynamic primitives, particularly with respect to geometry estimation in challenging domains.

Failure Modes and Limitations

In scenes dominated by large textureless areas (e.g., blank walls in the Bonn dataset), the method's reliance on large and overlapping Gaussians can degrade point accuracy due to excessive blending, in contrast to per-pixel approaches which are less susceptible to such error accumulation. Future enhancements are suggested, such as integrating compact (sparser) intermediate representations, non-linear Gaussian motion attributes, and time-varying splats to support longer sequences and more complex scene dynamics.

Figure 6: Comparison with DynaDUSt3R on Bonn—dynamic 3DGS may induce minor geometric inaccuracies in textureless regions due to blending of large-scale Gaussians; brighter indicates higher error.

Implications and Future Directions

UFO-4D provides a solid technical advancement toward unified, dense 4D reconstruction under minimal input supervision. Its explicit, differentiable 4D representation elegantly couples geometry, motion, and pose, which significantly aids robustness against label scarcity and error. Practically, this method is immediately useful for tasks requiring joint geometry and motion understanding from uncalibrated image pairs—e.g., robotics, AR/VR, autonomous navigation—without requiring heavy annotation, explicit multi-frame synchronization, or camera extrinsics.

On a theoretical level, the work demonstrates the effectiveness of shared geometric primitives as a representational bottleneck, yielding emergent cross-task regularization and generalizing favorably even with noisy real-world data. As computational and memory efficiency of dynamic Gaussian frameworks increases, this paradigm is poised to underlie end-to-end perception and scene understanding modules in generative and analytical AI systems, especially if adapted for longer-term and non-linear dynamics.

Conclusion

UFO-4D represents a significant step toward practical, unified 4D dynamic scene understanding from unposed image pairs. By leveraging dynamic 3D Gaussian splats in a feedforward, differentiable model, it achieves high-fidelity, interpretable geometry and motion estimation, as well as 4D spatiotemporal interpolation and robust pose regression. Its architectural design and strong empirical performance suggest that explicit, cross-modal scene representations will be increasingly central to general-purpose, data-efficient 4D perception systems in both research and applications.