Fast 3D Foundation Model Initialized Gaussian Splatting

Abstract: This paper introduces a fast method for high-quality 3D Gaussian Splatting (3DGS) reconstruction without traditional Structure-from-Motion (SfM). The proposed approach leverages 3D Foundation Models (3DFMs) for camera pose and point-cloud initialization, then jointly optimizes both camera poses and Gaussian primitives using a depth-guided loss function. This enables fast convergence even from rough initialization with as few as 50-60 input views. To further improve reconstruction quality in sparse-view scenarios, an MLP-based pose refinement module is introduced alongside depth-guided supervision from the foundation model. Extensive experiments on Mip-NeRF 360, Tanks and Temples, and RobustNeRF demonstrate that the proposed method achieves competitive reconstruction quality (23.61 dB PSNR, 0.19 LPIPS) while reducing training time to approximately three minutes per scene. The proposed method produces ready-to-use 3DGS models at a fraction of the time required by existing pipelines, making it suitable for near real-time applications in robotics, VR, and autonomous navigation.

• The paper introduces a rapid pipeline employing a 3D foundation model (VGGT-X) that eliminates traditional SfM for camera pose initialization.
• It leverages depth-guided losses and MCMC-based optimization to jointly refine Gaussian primitives and camera parameters for high fidelity.
• Empirical results demonstrate competitive PSNR and pose accuracy with drastically reduced training time, highlighting its potential for real-time applications.

Fast 3D Foundation Model Initialized Gaussian Splatting

Introduction

The paper "Fast 3D Foundation Model Initialized Gaussian Splatting" (2607.03209) proposes a highly accelerated pipeline for 3D Gaussian Splatting (3DGS), leveraging large-scale 3D Foundation Models (3DFMs) to eliminate reliance on traditional Structure-from-Motion (SfM) for camera parameter initialization. The methodology enables joint optimization of camera poses and 3D Gaussian primitives through novel depth-guided losses and global pose refinement. The contribution is particularly notable for its significantly reduced training time (approximately three minutes per scene) while attaining competitive fidelity and localization metrics, making it suitable for real-time or near-real-time applications.

Background and Motivation

Recent advances in 3D scene representation, especially the use of 3DGS, have provided efficient and high-fidelity radiance field reconstructions through spatially adaptive anisotropic Gaussian primitives. However, state-of-the-art pipelines have been bottlenecked by the need for accurate and robust camera poses, typically provided by computationally expensive SfM systems like COLMAP. This reliance introduces latency, scales poorly with large scene sizes, and inherits robustness issues in texture-sparse scenes or those with few available viewpoints.

As 3DFMs such as VGGT and its extension VGGT-X have achieved high-accuracy pose and depth inference directly from multi-view images, they provide an opportunity to overhaul 3DGS pipelines for accelerated end-to-end optimization, decoupled from classical multi-stage geometry pipelines.

Figure 1: The VGGT architecture, a unified multi-view transformer model for camera parameter, depth, and geometry inference.

Methodology

The proposed pipeline integrates advanced 3DFMs with modern 3DGS optimization strategies, specifically:

1. Pose and Depth Initialization: Multi-view images are processed by the VGGT-X transformer, which outputs camera intrinsics, extrinsics, and dense depth maps. The VGGT-X model relies on multi-view cross-frame attention to achieve implicit epipolar consistency and geometric reasoning.

   Figure 2: The VGGT-X pipeline diagram, illustrating the scalable architecture for dense pose and depth inference.

2. Gaussian Primitives Seeding: Initial point-clouds are constructed by back-projecting VGGT-X depth maps. These points seed Gaussian primitives, providing means, covariance, and confidence for downstream optimization. Confidence maps produced by VGGT-X are used to reject unreliable input points.
3. Joint Optimization with Depth Guidance: The 3DGS and camera poses are jointly optimized, with a custom loss function comprising a weighted sum of photometric L1+SSIM error and a depth-guided term comparing rendered alpha-based depth to VGGT-X depth outputs. The photometric supervision alone is prone to ambiguities, especially in sparse view regimes; the additional geometric regularization via depth losses significantly improves convergence and geometric fidelity.
4. MLP-Based Pose Refinement: The pose optimization leverages a globally shared MLP network (per-camera embedding), refining both translation and rotation offsets for each camera. This allows learning global pose correlations and correcting for drift, outperforming per-camera optimization schemes in terms of global consistency.

   Figure 3: Pose MLP module architecture for learned camera pose refinement across all views.

5. MCMC-based 3DGS Optimization: Following recent advances, the optimization step is interpreted as a form of SGLD-based Markov Chain Monte Carlo (MCMC) sampling; noise injection enables robust escape from local minima caused by imperfect initialization, and a regularizer reduces over-parameterization.

   Figure 4: Overall pipeline schematic—VGGT-X provides pose and depth initialization, followed by joint optimization of Gaussians and poses with depth guidance.

Empirical Results

Quantitative and qualitative evaluations are carried out on Mip-NeRF 360, Tanks and Temples, and RobustNeRF datasets. The pipeline is validated on metrics such as PSNR, SSIM, LPIPS, Absolute Trajectory Error (ATE), and Rotation Error (RTE):

• Novel View Synthesis: The pipeline achieves PSNR values around 23.61 dB and LPIPS as low as 0.19 on validation sets. In sparse view regimes (50–60 views), these results are competitive with (or outperform, on perceptual LPIPS) traditional 3DGS or pose-free pipelines, despite training for only 7000 steps (vs. ~30000 as typical in 3DGS baselines).
• Pose Registration: The pipeline achieves an ATE of ~0.016 m and rotational errors (RTE) competitive with state-of-the-art, with moderate degradation compared to COLMAP only in high-barrel-distortion or high field-of-view scenarios.

  Figure 5: Qualitative rendering results from multiple datasets, showcasing high-fidelity geometry and photorealistic appearance.

Ablation studies indicate that omitting MCMC-based Gaussian optimization or the depth-guided loss significantly degrades PSNR, SSIM, and perceptual metrics as well as pose accuracy, confirming the necessity of each component. The pipeline is robust to input view sparsity and maintains competitive accuracy with reduced training time and computational resources.

Practical and Theoretical Implications

The accelerated and SfM-free nature of this pipeline unlocks multiple practical applications: robotics (real-time map-centric SLAM and navigation), AR/VR (fast scene digitization for head-mounted displays), and 3D content creation. The theoretical implications include demonstrating that foundation model-derived geometric signals are sufficient for initializing and jointly optimizing geometric and photometric radiance field representations, circumventing classical geometry pipelines. The use of depth-guided losses illustrates the benefit of explicit geometry signals as inductive priors, especially in ill-posed regimes.

Limitations and Future Directions

While the framework attains state-of-the-art speed and comparable accuracy, there are failure cases for high-distortion camera models where VGGT-X underperforms. Intrinsic estimation in extreme wide-FOV or barrel distortion scenarios remains challenging. The pipeline so far focuses on static scenes with a single camera; extending the framework to handle dynamic content and arbitrary camera models is a natural avenue.

Potential future work could also consider:

• Integrating semantic segmentation or open-world weak supervision directly into the 3DFM backbone.
• Exploring large-scale outdoor navigation scenarios and optimizations for embedded hardware.
• Applying the pipeline in an online/SLAM context with real-time incremental reconstruction and updating.

Conclusion

This work presents a practical and theoretically sound method for rapid, high-quality 3DGS scene reconstruction, initialized and guided by 3D foundation model outputs and augmented by robust MCMC and multi-task optimization. The method dramatically reduces model preparation time, delivers competitive visual and geometric accuracy, and is readily extensible to new domains where classical SfM is infeasible or a bottleneck. The combination of foundation model geometry and neural rendering via 3DGS represents a significant step forward in scalable 3D scene understanding.