InstantSfM: Fully Sparse and Parallel Structure-from-Motion (2510.13310v1)

Abstract: Structure-from-Motion (SfM), a method that recovers camera poses and scene geometry from uncalibrated images, is a central component in robotic reconstruction and simulation. Despite the state-of-the-art performance of traditional SfM methods such as COLMAP and its follow-up work, GLOMAP, naive CPU-specialized implementations of bundle adjustment (BA) or global positioning (GP) introduce significant computational overhead when handling large-scale scenarios, leading to a trade-off between accuracy and speed in SfM. Moreover, the blessing of efficient C++-based implementations in COLMAP and GLOMAP comes with the curse of limited flexibility, as they lack support for various external optimization options. On the other hand, while deep learning based SfM pipelines like VGGSfM and VGGT enable feed-forward 3D reconstruction, they are unable to scale to thousands of input views at once as GPU memory consumption increases sharply as the number of input views grows. In this paper, we unleash the full potential of GPU parallel computation to accelerate each critical stage of the standard SfM pipeline. Building upon recent advances in sparse-aware bundle adjustment optimization, our design extends these techniques to accelerate both BA and GP within a unified global SfM framework. Through extensive experiments on datasets of varying scales (e.g. 5000 images where VGGSfM and VGGT run out of memory), our method demonstrates up to about 40 times speedup over COLMAP while achieving consistently comparable or even improved reconstruction accuracy. Our project page can be found at https://cre185.github.io/InstantSfM/.

• The paper presents InstantSfM, which unifies bundle adjustment and global positioning using sparse Jacobian matrices to reduce computational complexity from O(n²) to O(n).
• It demonstrates a robust, scalable SfM pipeline that exploits GPU parallelism in PyTorch, achieving up to 40x speedup compared to traditional methods like COLMAP.
• The integration of monocular depth priors and robust outlier rejection enhances metric accuracy, making the approach highly effective for large-scale 3D reconstruction tasks.

InstantSfM: Fully Sparse and Parallel Structure-from-Motion

Introduction and Motivation

The paper introduces InstantSfM, a global Structure-from-Motion (SfM) pipeline that leverages fully sparse and parallel optimization on GPUs, implemented natively in PyTorch. The motivation is to address the computational bottlenecks of traditional SfM systems—such as COLMAP and GLOMAP—which rely on CPU-centric, dense matrix operations for bundle adjustment (BA) and global positioning (GP). These approaches scale poorly with increasing image counts, limiting their applicability to large-scale datasets. Deep learning-based SfM pipelines, while flexible, suffer from prohibitive GPU memory consumption and lack scalability. InstantSfM aims to combine the flexibility of Python-based frameworks with the efficiency of sparse, parallel GPU computation, enabling robust and scalable SfM for thousands of images.

Sparse-Aware Optimization in SfM

InstantSfM extends sparse bundle adjustment techniques to both BA and GP, unifying them in a single global SfM framework. The key technical innovation is the exploitation of the inherent sparsity in the Jacobian matrices arising from the reprojection error minimization in BA and GP. Each observation in the scene graph only affects a small subset of parameters (camera intrinsics, extrinsics, and 3D points), resulting in a highly sparse block-diagonal Jacobian structure.

Figure 1: Visualization of the sparse Jacobian $J^S$ and the corresponding sparse-aware operations for efficient LM optimization, including matrix-matrix and matrix-vector multiplications, and diagonal element fetching.

InstantSfM stores the Jacobian in a sparse format, retaining only non-zero blocks and their indices, which reduces both spatial and computational complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(n)$, where $n$ is the number of 3D points. This enables efficient parallelization across GPU compute units, with each non-zero block processed independently. The pipeline implements custom routines for differentiable quaternion addition and large sparse matrix multiplications, not natively supported in PyTorch, and supports seamless integration with other PyTorch/Python packages.

Levenberg-Marquardt with Sparse Jacobians

The optimization in InstantSfM is performed using the Levenberg-Marquardt (LM) algorithm, which solves the following linear system at each iteration:

$$(\mathbf{J}^\top \mathbf{J} + \lambda \operatorname{diag}(\mathbf{J}^\top \mathbf{J})) \Delta \boldsymbol{\theta} = -\mathbf{J}^\top \mathbf{r}$$

where $\mathbf{J}$ is the sparse Jacobian, $\lambda$ is the damping factor, and $\mathbf{r}$ is the reprojection error. The sparse representation allows for efficient computation of $\mathbf{J}^\top \mathbf{J}$, $\mathbf{J}^\top \mathbf{r}$, and diagonal extraction, all of which are critical for the LM update step. Robust loss functions are integrated directly into the sparse matrix operations, enabling automatic outlier rejection without dense intermediate computations.

Depth Prior Integration

InstantSfM supports the injection of monocular depth priors into the optimization, enabling metric-scale recovery of camera parameters and 3D points. The scale term in the GP objective is replaced with the inverse of the depth sampled from estimated or ground-truth depth maps, enforcing metric consistency in the reconstructed scene. This is particularly relevant for robotics and simulation applications where physical scale is critical.

Implementation Details

The pipeline is implemented entirely in PyTorch, with specialized routines for sparse block assembly, cross-stage memory management, and robust outlier integration. Memory pools are used to reuse sparse matrix storage across GP and BA stages, minimizing costly reallocations. The system is designed for single-node execution, but the architecture is amenable to future distributed extensions.

Experimental Results

InstantSfM is evaluated on multiple benchmarks, including MipNeRF360, DTU, Tanks and Temples, ScanNet, and ScanNet++. The method demonstrates up to $\sim$40$\times$ speedup over COLMAP and substantial improvements over GLOMAP, while maintaining comparable or superior reconstruction accuracy. On large-scale datasets (e.g., 5000 images), InstantSfM remains robust and efficient, whereas deep learning-based baselines such as VGGSfM run out of memory.

Figure 2: Qualitative results on various datasets, showing estimated camera poses and sampled input images for each reconstruction.

On ScanNet and ScanNet++, InstantSfM exhibits significantly better robustness, with COLMAP and GLOMAP frequently failing due to solver or reconstruction issues. The integration of depth priors further improves metric accuracy, as evidenced by lower Chamfer distances in reconstructed point clouds.

Runtime Analysis

InstantSfM achieves dramatic reductions in total SfM running time compared to COLMAP and GLOMAP, especially as the number of images increases. The speedup is attributed to the fully sparse and parallel GPU implementation, which scales efficiently with scene size.

Figure 3: Total SfM running time comparison among COLMAP, GLOMAP, and InstantSfM across varying image counts (log scale).

Figure 4: BA and GP running time comparison between GLOMAP and InstantSfM for different image counts (log scale).

Limitations and Future Directions

The current implementation does not accelerate 3D point triangulation with CUDA/Triton operators, which could yield further performance gains. The pipeline is optimized for single-node execution and does not yet support distributed computation, which may limit scalability for extremely large datasets. Future work should address these limitations and explore deeper integration of learning-based features for improved generalization and reconstruction quality.

Implications and Outlook

InstantSfM demonstrates that fully sparse, parallel optimization in Python-based frameworks can match or exceed the performance of traditional C++-based SfM systems, while offering greater flexibility and extensibility. The approach is particularly well-suited for large-scale 3D reconstruction tasks in robotics, simulation, and digital heritage, where scalability and metric accuracy are paramount. The open-source release of InstantSfM will facilitate further research and practical deployment in diverse application domains. Future developments may include distributed sparse optimization, tighter integration with neural implicit representations, and hybrid learning-geometric pipelines for robust, scalable 3D scene understanding.

Conclusion

InstantSfM provides a robust, efficient, and flexible solution for global SfM, leveraging sparse-aware GPU computation in PyTorch. The method achieves substantial speedups and maintains high reconstruction accuracy across a range of benchmarks, with demonstrated scalability to thousands of images. The integration of depth priors and robust outlier handling further enhances its applicability to real-world scenarios. The work sets a new standard for scalable SfM pipelines and opens avenues for future research in distributed optimization and hybrid learning-based 3D reconstruction.