Glob3R: Global Structure-from-Motion with 3D Foundation Models

Abstract: Recent 3D geometric foundation models, such as VGGT, provide robust feed-forward 3D reconstruction by directly predicting camera poses and 3D scene points from input images. However, their results remain inaccurate, and scaling them to long sequences or large unordered image sets typically requires chunk-wise processing, which can introduce drift and inconsistency. We present Glob3R, a global SfM-style reconstruction built on 3D foundation models. Our key idea is to explicitly optimize feed-forward geometric predictions. To this end, we augment a frozen Pi3X backbone with a lightweight dense matching head that predicts image warps between selected reference frames and neighboring views. These dense warps are converted into sparse but reliable multi-view feature tracks, which provide correspondence constraints for global optimization. We further introduce a keyframe-based sliding-window association strategy that propagates tracks and relative poses across overlapping windows, enabling scalable reconstruction. Finally, we perform global motion averaging and bundle adjustment to refine camera poses, reduce scale inconsistencies, and recover dense scene geometry. Extensive experiments on indoor, outdoor, large-scale driving, and unordered SfM benchmarks demonstrate that Glob3R achieves robust and accurate reconstruction. It consistently improves over feed-forward foundation-model baselines and recent scalable reconstruction methods, while being more robust than classical SfM pipelines. The refined poses also lead to higher-quality neural rendering, validating the benefit of combining foundation-model priors with global geometric optimization. Project page: https://junyuandeng.github.io/Glob3r

• The paper introduces a novel dense matching head that integrates 3D foundation models with global optimization, significantly improving SfM accuracy and scalability.
• It employs a keyframe-based sliding window and pose graph construction to maintain robust global alignment over both ordered and unordered image sets.
• Empirical evaluations demonstrate notable improvements, with higher PSNR and reduced RMSE across benchmarks, underlining its practical impact in 3D reconstruction.

Glob3R: Global Structure-from-Motion with 3D Foundation Models

Introduction

Glob3R introduces a new paradigm for large-scale, high-fidelity structure-from-motion (SfM) by leveraging geometric priors from 3D foundation models and integrating them into a global, optimization-driven reconstruction pipeline. The method addresses accuracy, scalability, and robustness deficiencies inherent in both feed-forward geometric models (such as Pi3X, VGGT) and classical SfM systems. It achieves this by augmenting frozen model backbones with a novel dense matching head, transforming feed-forward geometric predictions into explicit, optimizable constraints that span long sequences or unordered image sets.

Methodology

Glob3R unifies feed-forward foundation models with classical global SfM through several innovations:

1. Dense Feature Warping Head: On top of a frozen Pi3X backbone, a lightweight matching head predicts dense image warps between reference and neighboring frames. These warps are refined through a multi-stage process and converted into sparse, high-confidence multi-view feature tracks, providing explicit correspondence constraints for geometric optimization.
2. Keyframe-Based Sliding Window Association: Images are processed in overlapping windows via a keyframe selection strategy based on reprojection coverage. This approach enables the propagation of feature tracks and relative poses across the entire sequence or image set, scaling to long trajectories and unordered collections without suffering chunk-weighted drift or loss of geometric consistency.
3. Pose Graph Construction and Global Optimization: The system merges local tracks into a global pose graph, initializing camera poses via robust motion averaging (decoupling rotation and translation) and then refines all variables—including scale factors and sparse/dense geometry—via bundle adjustment. This yields consistency at scale and highly accurate camera registration.
4. Dense Reconstruction: Refined camera poses and depths are leveraged to back-project and fuse globally aligned dense point clouds.

The pipeline explicitly avoids backbone retraining, works with both ordered and unordered datasets, and remains efficient thanks to GPU-accelerated optimization modules.

Empirical Evaluation

The framework is extensively evaluated on a diverse suite of benchmarks: Tanks and Temples (multi-object/object-centric), TUM RGB-D (indoor), ETH3D (unordered scenes), and KITTI Odometry (large-scale driving). Quantitative results demonstrate that:

• On novel view synthesis tasks (Tanks and Temples), Glob3R achieves the highest PSNR across most sequences (e.g., 2–3 dB over feed-forward baselines, ≈1 dB over COLMAP poses), validating its geometric consistency in downstream neural rendering.
• On KITTI, Glob3R outperforms recent feed-forward, streaming, and chunk-based models by 10–50% in trajectory RMSE, with superior global trajectory preservation and substantially reduced drift.

  Figure 1: Camera trajectory comparison on KITTI Odometry. Glob3R maintains global shape and mitigates drift on long, forward-motion sequences.

• On ETH3D, Glob3R surpasses contemporary global SfM systems and learning-based methods in both rotation and translation accuracy at strict ($1^\circ$) and loose ($5^\circ$) thresholds, nearly doubling translation accuracy over the latest foundation model baselines.
• High-fidelity, dense reconstructions align closely with ground truth, supporting robust use in high-precision applications such as neural rendering.

  Figure 2: Dense point cloud reconstructions utilizing optimized camera poses and recovered depths.

Glob3R's dense warping module also achieves sharp, faithful view synthesis with minimal pose-induced artifacts, outperforming both hand-crafted and other learning-based approaches in qualitative and PSNR-based evaluations.

Figure 3: Qualitative comparison on Tanks and Temples. Glob3R yields sharper, more faithful novel view renders with suppressed geometric artifacts.

Discussion and Implications

A salient advance is the demonstrated synergy of global geometric optimization and powerful feed-forward priors. Glob3R resolves the limited-accuracy and scalability bottlenecks of current 3D foundation models, which, trained primarily with noisy SfM-derived supervision and limited by hardware constraints, often fail to extend beyond moderate sequence lengths or unordered image collections. By refining predictions with correspondence-driven motion averaging and bundle adjustment, the method achieves both robustness and metric accuracy.

Equally important is the design of its dense matching head, which, unlike two-view methods (e.g., RoMa v2) or single-reference track-and-repeat strategies (e.g., VGGT), provides efficient, multi-view, spatially connected correspondences. This brings superior matching quality, essential for reliable global optimization.

The sliding-window and pseudo-sequencing strategies afford unmatched flexibility on unordered datasets, which are challenging for sequence-dependent or recurrent-memory models. The explicit global pose graph, constructed from dense learned tracks rather than classic keypoint-based matchings, increases resilience to weak textures, repetitive structures, and low-parallax degeneracies.

The method makes no assumption of perfect foundation-model geometry—empirically, it is robust to small errors in scale and pose predictions, cleaning up substantial inconsistencies via global optimization. Nevertheless, cases of severe geometric ambiguity in the backbone can propagate errors into the correspondence and optimization stages, indicating the importance of ongoing advances in foundation model architectures and more robust matching strategies.

Limitations and Future Directions

Failure analyses highlight that the accuracy and robustness of the dense matching head remain contingent on the quality of the underlying backbone’s geometric predictions. Future research should focus on decoupling this dependence—potentially via alternative cross-view feature normalization or geometry-agnostic matching modules—and on adaptive window sizing and confidence-based track weighting for improved long-range scalability and efficiency.

Open directions also include tighter integration with neural rendering frameworks (e.g., NeRF), development of foundation models with robust, ground-truth supervision, and the design of direct dense matching networks with learned oracles for ambiguous geometry.

Conclusion

Glob3R establishes a globally optimized, foundation-model-driven pipeline for robust, scalable, and accurate 3D reconstruction. It delivers substantial empirical improvements over both classical and learning-based baselines, providing state-of-the-art results on challenging benchmarks spanning indoor, outdoor, and unordered scenarios. The hybridization of explicit geometry optimization with dense, learned correspondence priors sets a new direction for both practical deployment and theoretical advancement in 3D vision, paving the way for scalable, high-fidelity scene reconstruction and rendering in downstream applications.