Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting (2402.15870v2)

Published 24 Feb 2024 in cs.CV

Abstract: The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces. Project page is https://ingra14m.github.io/Spec-Gaussian-website/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
  2. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  3. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
  4. High-quality surface splatting on today’s gpus. In Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, 2005., pages 17–141. IEEE, 2005.
  5. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022a.
  6. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pages 333–350. Springer, 2022b.
  7. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, 2023a.
  8. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.00277, 2022c.
  9. Neurbf: A neural fields representation with adaptive radial basis functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4182–4194, 2023b.
  10. Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.16585, 2023c.
  11. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023.
  12. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  13. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346–14355, 2021.
  14. Point-based graphics. Elsevier, 2011.
  15. Multiscale tensor decomposition and rendering equation encoding for view synthesis. In The IEEE / CVF Computer Vision and Pattern Recognition Conference, pages 4232–4241, 2023.
  16. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875–5884, 2021.
  17. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774–19783, 2023.
  18. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. pages 1–11, 2023.
  19. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. arXiv preprint arXiv:2312.03461, 2023a.
  20. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. arXiv preprint arXiv:2311.17977, 2023b.
  21. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
  22. Approximate differentiable rendering with algebraic surfaces. In European Conference on Computer Vision, pages 596–614. Springer, 2022.
  23. Flexible techniques for differentiable rendering with 3d gaussians. arXiv preprint arXiv:2308.14737, 2023.
  24. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812, 2023a.
  25. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
  26. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching, 2023.
  27. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
  28. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651–15663, 2020.
  29. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In SIGGRAPH, 2023.
  30. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00109, 2023.
  31. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
  32. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  33. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022.
  34. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280–8290, 2022.
  35. Ash: Animatable gaussian splats for efficient and photoreal human rendering. 2023.
  36. Camp: Camera preconditioning for neural radiance fields. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023.
  37. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
  38. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335–14345, 2021.
  39. Relightable gaussian codec avatars. 2023.
  40. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 4104–4113, 2016.
  41. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05133, 2023.
  42. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
  43. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  44. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023.
  45. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
  46. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021a.
  47. BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4170–4179, 2023.
  48. NeRF−⁣−--- -: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021b.
  49. Nex: Real-time view synthesis with neural basis expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  50. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023a.
  51. Reconfusion: 3d reconstruction with diffusion priors. arXiv, 2023b.
  52. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In International Conference on Learning Representations (ICLR), 2023c.
  53. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198, 2023.
  54. Anisotropic spherical gaussians. ACM Transactions on Graphics, 32(6):209:1–209:11, 2013.
  55. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
  56. Sire-ir: Inverse rendering for brdf reconstruction with shadow and illumination removal in high-illuminance scenes. arXiv preprint arXiv:2310.13030, 2023b.
  57. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101, 2023c.
  58. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv 2310.10642, 2023d.
  59. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. arXiv preprint arXiv:2310.08529, 2023.
  60. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG), 38(6):1–14, 2019.
  61. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5761, 2021.
  62. Differentiable point-based radiance fields for efficient view synthesis. arXiv preprint arXiv:2205.14330, 2022.
  63. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG), 40(6):1–18, 2021.
  64. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv, 2023.
  65. Drivable 3d gaussian avatars. 2023.
  66. Ewa volume splatting. In Proceedings Visualization, 2001. VIS’01., pages 29–538. IEEE, 2001.
Citations (24)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now