Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

COLMAP-Free 3D Gaussian Splatting (2312.07504v2)

Published 12 Dec 2023 in cs.CV

Abstract: While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses. To relax this constraint, multiple efforts have been made to train Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the implicit representations of NeRFs provide extra challenges to optimize the 3D structure and camera poses at the same time. On the other hand, the recently proposed 3D Gaussian Splatting provides new opportunities given its explicit point cloud representations. This paper leverages both the explicit geometric representation and the continuity of the input video stream to perform novel view synthesis without any SfM preprocessing. We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time, without the need to pre-compute the camera poses. Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes. Our project page is https://oasisyang.github.io/colmap-free-3dgs

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16610–16620, 2023.
  2. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  3. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  4. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706, 2023.
  5. Nope-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4160–4169, 2023.
  6. Lu-nerf: Scene and pose estimation by synchronizing local unposed nerfs. arXiv preprint arXiv:2306.05410, 2023.
  7. Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation. arXiv e-prints, 2022.
  8. Mononerf: Learning generalizable nerfs from monocular videos without camera pose. In ICML, 2023.
  9. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV, 2021.
  10. Multiple view geometry in computer vision. 2003.
  11. Automatic photo pop-up. In ACM SIGGRAPH 2005 Papers, pages 577–584, 2005.
  12. Tour into the picture: using a spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 225–232, 1997.
  13. Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image. In ICCV, 2020.
  14. Self-calibrating neural radiance fields. In ICCV, 2021.
  15. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.
  16. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
  17. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 2017.
  18. Neural point catacaustics for novel-view synthesis of reflections. ACM Transactions on Graphics (TOG), 41(6):1–15, 2022.
  19. Video autoencoder: self-supervised disentanglement of 3d structure and motion. In ICCV, 2021.
  20. Mine: Towards continuous depth mpi with nerf for novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12578–12588, 2021.
  21. Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
  22. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651–15663, 2020.
  23. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713, 2023.
  24. Progressively optimized local radiance fields for robust view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16539–16548, 2023.
  25. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
  26. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph.
  27. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 2015.
  28. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
  29. Automatic differentiation in pytorch. 2017.
  30. Vision transformers for dense prediction. In ICCV, 2021.
  31. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335–14345, 2021.
  32. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901–10911, 2021.
  33. Free view synthesis. In ECCV, 2020.
  34. Stable view synthesis. In CVPR, 2021.
  35. Rust: Latent neural scene representations from unposed imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17297–17306, 2023.
  36. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
  37. Structure-from-motion revisited. In CVPR, 2016.
  38. Implicit neural representations with periodic activation functions. NeurIPS, 2020.
  39. Flowcam: Training generalizable 3d radiance fields without camera poses via pixel-aligned scene flow. arXiv preprint arXiv:2306.00180, 2023.
  40. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  41. Visual slam algorithms: A survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications, 9(1):1–11, 2017.
  42. Single-view view synthesis with multiplane images. In CVPR, 2020.
  43. Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04):376–380, 1991.
  44. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE, 2022.
  45. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004.
  46. NeRF−⁣−--- -: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021.
  47. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
  48. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. 2022.
  49. Sinnerf: Training neural radiance fields on complex scenes from a single image. In European Conference on Computer Vision, pages 736–753. Springer, 2022a.
  50. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438–5448, 2022b.
  51. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8254–8263, 2023.
  52. inerf: Inverting neural radiance fields for pose estimation. In IROS, 2021.
  53. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG), 38(6):1–14, 2019.
  54. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5761, 2021.
  55. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
  56. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia 2022 Conference Papers, pages 1–12, 2022.
  57. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  58. Stereo magnification: Learning view synthesis using multiplane images. 2018.
Citations (65)
View on Semantic Scholar

Summary

  • The paper introduces CF-3DGS, a method that eliminates reliance on pre-computed camera poses for on-the-fly 3D reconstruction.
  • It employs a dual-local-global optimization strategy by sequentially updating 3D Gaussian sets from video frames to enhance synthesis accuracy.
  • The approach achieves robust performance under wide-ranging camera movements, markedly improving efficiency and quality in 360° video scenes.

Introduction to Novel View Synthesis

The evolving domain of photo-realistic scene reconstruction and novel view synthesis has seen remarkable progress, particularly with the advent of Neural Radiance Fields (NeRF). These developments hinge on upfront computation of camera poses, traditionally derived from Structure-from-Motion (SfM) techniques, such as those provided by the COLMAP library. However, pre-computed camera poses can create bottlenecks and limitations.

Challenges and Innovations

NeRF's implicit scene representations create inherent constraints when simultaneously determining 3D scene structure and camera posing. For instance, methods like 'Nope-NeRF' may face difficulties when camera poses alter substantially, a common case in 360-degree video recordings. Integrating camera pose estimation within the NeRF framework has long been a complex task, akin to the proverbial chicken-and-egg problem.

Enter the method of 3D Gaussian Splatting. It offers a new perspective with its explicit point cloud representations, opening a window for bypassing pre-processed camera pose estimations. Recognizing this potential, a new approach has been proposed—COLMAP-Free 3D Gaussian Splatting (CF-3DGS). This method harnesses the explicit geometric nature of video sequences and the continuity of video frame passages to perform innovative view synthesis without any SfM pre-processing.

CF-3DGS Methodology

CF-3DGS operates by processing input video frames sequentially, allowing the set of 3D Gaussian representations to 'grow' as the camera navigates through space. Each incoming frame leads to an updated local 3D Gaussian set, which is then correlated to a global representation of the scene. This dual-local-global optimization, working with both current and previous frames, contributes to significantly better scene reconstruction and camera pose predictions when benchmarked against other non-SfM methods.

Advantages and Results

The proposed CF-3DGS method sets itself apart by achieving robustness in pose estimation and higher quality in novel view synthesis across various scenes. Not confined to small shifts in camera motion, its performance excels notably when challenged with wide-ranging camera movements, particularly in scenarios of 360-degree video captures. Moreover, it brings efficiency to the process, achieving results on par with state-of-the-art methods like 'Nope-NeRF' with notably more expedient training times.

Future Implications

In the pursuit of mirroring and manipulating reality through technology, methods like CF-3DGS edge us closer to seamless and realistic virtual experiences. Whether for entertainment, simulation, or education, the impacts of such advances open unexplored doors to how we might interact with and visualize our surroundings through the lens of artificial intelligence.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. COLMAP-Free 3D Gaussian Splatting
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube