Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion (2404.04687v2)

Published 6 Apr 2024 in cs.CV, cs.GR, and cs.LG

Abstract: Differentiable 3D-Gaussian splatting (GS) is emerging as a prominent technique in computer vision and graphics for reconstructing 3D scenes. GS represents a scene as a set of 3D Gaussians with varying opacities and employs a computationally efficient splatting operation along with analytical derivatives to compute the 3D Gaussian parameters given scene images captured from various viewpoints. Unfortunately, capturing surround view ($360{\circ}$ viewpoint) images is impossible or impractical in many real-world imaging scenarios, including underwater imaging, rooms inside a building, and autonomous navigation. In these restricted baseline imaging scenarios, the GS algorithm suffers from a well-known 'missing cone' problem, which results in poor reconstruction along the depth axis. In this manuscript, we demonstrate that using transient data (from sonars) allows us to address the missing cone problem by sampling high-frequency data along the depth axis. We extend the Gaussian splatting algorithms for two commonly used sonars and propose fusion algorithms that simultaneously utilize RGB camera data and sonar data. Through simulations, emulations, and hardware experiments across various imaging scenarios, we show that the proposed fusion algorithms lead to significantly better novel view synthesis (5 dB improvement in PSNR) and 3D geometry reconstruction (60% lower Chamfer distance).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, no. 4, pp. 1–14, 2023.
  2. M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Ewa volume splatting,” in Proceedings Visualization, 2001. VIS’01.   IEEE, 2001, pp. 29–538.
  3. T. Lu, M. Yu, L. Xu, Y. Xiangli, L. Wang, D. Lin, and B. Dai, “Scaffold-gs: Structured 3d gaussians for view-adaptive rendering,” arXiv preprint arXiv:2312.00109, 2023.
  4. Z. Yu, A. Chen, B. Huang, T. Sattler, and A. Geiger, “Mip-splatting: Alias-free 3d gaussian splatting,” arXiv preprint arXiv:2311.16493, 2023.
  5. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  6. J. Gao, C. Gu, Y. Lin, H. Zhu, X. Cao, L. Zhang, and Y. Yao, “Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing,” arXiv preprint arXiv:2311.16043, 2023.
  7. Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma, “Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces,” arXiv preprint arXiv:2311.17977, 2023.
  8. Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang, “Fsgs: Real-time few-shot view synthesis using gaussian splatting,” arXiv preprint arXiv:2312.00451, 2023.
  9. J. Fang, J. Wang, X. Zhang, L. Xie, and Q. Tian, “Gaussianeditor: Editing 3d gaussians delicately with text instructions,” arXiv preprint arXiv:2311.16037, 2023.
  10. D. Das, C. Wewer, R. Yunus, E. Ilg, and J. E. Lenssen, “Neural parametric gaussians for monocular non-rigid object reconstruction,” arXiv preprint arXiv:2312.01196, 2023.
  11. Z. Yang, H. Yang, Z. Pan, X. Zhu, and L. Zhang, “Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting,” arXiv preprint arXiv:2310.10642, 2023.
  12. A. Kratimenos, J. Lei, and K. Daniilidis, “Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting,” arXiv preprint arXiv:2312.00112, 2023.
  13. R. Shao, J. Sun, C. Peng, Z. Zheng, B. Zhou, H. Zhang, and Y. Liu, “Control4d: Dynamic portrait editing by learning 4d gan from 2d diffusion-based editor,” arXiv preprint arXiv:2305.20082, 2023.
  14. Y.-H. Huang, Y.-T. Sun, Z. Yang, X. Lyu, Y.-P. Cao, and X. Qi, “Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes,” arXiv preprint arXiv:2312.14937, 2023.
  15. H. Yu, J. Julin, Z. Á. Milacski, K. Niinuma, and L. A. Jeni, “Cogs: Controllable gaussian splatting,” arXiv preprint arXiv:2312.05664, 2023.
  16. J. Huang and H. Yu, “Point’n move: Interactive scene object manipulation on gaussian splatting radiance fields,” arXiv preprint arXiv:2311.16737, 2023.
  17. F. Macias-Garza, K. R. Diller, and A. C. Bovik, “Missing cone of frequencies and low-pass distortion in three-dimensional microscopic images,” Optical Engineering, vol. 27, no. 6, pp. 461–465, 1988.
  18. X. Liu, S. Bauer, and A. Velten, “Analysis of feature visibility in non-line-of-sight measurements,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 140–10 148.
  19. J. Chung, J. Oh, and K. M. Lee, “Depth-regularized optimization for 3d gaussian splatting in few-shot images,” 2024.
  20. H. Liu, M. Roznere, and A. Q. Li, “Deep underwater monocular depth estimation with single-beam echosounder,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 1090–1097.
  21. M. Qadri, K. Zhang, A. Hinduja, M. Kaess, A. Pediredla, and C. A. Metzler, “Aoneus: A neural rendering framework for acoustic-optical sensor fusion,” 2024.
  22. A. Reed, J. Kim, T. Blanford, A. Pediredla, D. Brown, and S. Jayasuriya, “Neural volumetric reconstruction for coherent synthetic aperture sonar,” ACM Transactions on Graphics (TOG), vol. 42, no. 4, pp. 1–20, 2023.
  23. M. F. Fallon, J. Folkesson, H. McClelland, and J. J. Leonard, “Relocating underwater features autonomously using sonar-based slam,” IEEE Journal of Oceanic Engineering, vol. 38, no. 3, pp. 500–513, 2013.
  24. P. Yang, H. Liu, M. Roznere, and A. Q. Li, “Monocular camera and single-beam sonar-based underwater collision-free navigation with domain randomization,” in The International Symposium of Robotics Research.   Springer, 2022, pp. 85–101.
  25. M. Roznere and A. Q. Li, “Underwater monocular image depth estimation using single-beam echosounder,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 1785–1790.
  26. T. Zhang, S. Liu, X. He, H. Huang, and K. Hao, “Underwater target tracking using forward-looking sonar for autonomous underwater vehicles,” Sensors, vol. 20, no. 1, p. 102, 2019.
  27. P. S. Heckbert, “Fundamentals of texture mapping and image warping,” 1989.
  28. M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Surface splatting,” in Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001, pp. 371–378.
  29. ——, “Ewa splatting,” IEEE Transactions on Visualization and Computer Graphics, vol. 8, no. 3, pp. 223–238, 2002.
  30. B. Fei, J. Xu, R. Zhang, Q. Zhou, W. Yang, and Y. He, “3d gaussian as a new vision era: A survey,” arXiv preprint arXiv:2402.07181, 2024.
  31. H. Matsuki, R. Murai, P. H. Kelly, and A. J. Davison, “Gaussian splatting slam,” arXiv preprint arXiv:2312.06741, 2023.
  32. N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, “Splatam: Splat, track & map 3d gaussians for dense rgb-d slam,” arXiv preprint arXiv:2312.02126, 2023.
  33. C. Yan, D. Qu, D. Wang, D. Xu, Z. Wang, B. Zhao, and X. Li, “Gs-slam: Dense visual slam with 3d gaussian splatting,” arXiv preprint arXiv:2311.11700, 2023.
  34. S. Sun, M. Mielle, A. J. Lilienthal, and M. Magnusson, “High-fidelity slam using gaussian splatting with rendering-guided densification and regularized optimization,” arXiv preprint arXiv:2403.12535, 2024.
  35. F. Ferreira, D. Machado, G. Ferri, S. Dugelay, and J. Potter, “Underwater optical and acoustic imaging: A time for fusion? a brief overview of the state-of-the-art,” OCEANS 2016 MTS/IEEE Monterey, pp. 1–6, 2016.
  36. Y. Raaj, A. John, and T. Jin, “3d object localization using forward looking sonar (fls) and optical camera via particle filter based calibration and fusion,” OCEANS 2016 MTS/IEEE Monterey, pp. 1–10, 2016.
  37. S. Williams and I. Mahon, “Simultaneous localisation and mapping on the great barrier reef,” IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004, vol. 2, pp. 1771–1776, 2004.
  38. M. Babaee and S. Negahdaripour, “3-d object modeling from 2-d occluding contour correspondences by opti-acoustic stereo imaging,” Computer Vision and Image Understanding, vol. 132, pp. 56–74, 2015.
  39. R. Nabati and H. Qi, “Centerfusion: Center-based radar and camera fusion for 3d object detection,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 1527–1536.
  40. R. Grover, G. Brooker, and H. F. Durrant-Whyte, “A low level fusion of millimeter wave radar and night-vision imaging for enhanced characterization of a cluttered environment,” in Proceedings 2001 Australian Conference on Robotics and Automation, 2001.
  41. M. Bijelic, T. Gruber, F. Mannan, F. Kraus, W. Ritter, K. Dietmayer, and F. Heide, “Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 682–11 692.
  42. Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Miscusik, and S. Thrun, “Multi-view image and tof sensor fusion for dense 3d reconstruction,” in 2009 IEEE 12th international conference on computer vision workshops, ICCV workshops.   IEEE, 2009, pp. 1542–1549.
  43. D. B. Lindell, M. O’Toole, and G. Wetzstein, “Single-photon 3d imaging with deep sensor fusion.” ACM Trans. Graph., vol. 37, no. 4, p. 113, 2018.
  44. M. Nishimura, D. B. Lindell, C. Metzler, and G. Wetzstein, “Disambiguating monocular depth estimation with a single transient,” in European Conference on Computer Vision.   Springer, 2020, pp. 139–155.
  45. M. Salvi and K. Vaidyanathan, “Multi-layer alpha blending,” in Proceedings of the 18th Meeting of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2014, pp. 151–158.
  46. A. H. Delaney and Y. Bresler, “Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography,” IEEE Transactions on image processing, vol. 7, no. 2, pp. 204–221, 1998.
  47. M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob, “Mitsuba 2: A retargetable forward and inverse renderer,” ACM Transactions on Graphics (TOG), vol. 38, no. 6, pp. 1–17, 2019.
  48. T. E. Blanford, J. D. McKay, D. C. Brown, J. D. Park, and S. F. Johnson, “Development of an in-air circular synthetic aperture sonar system as an educational tool,” vol. 36.   ASA, 8 2019, p. 070002.
  49. J. D. Park, T. E. Blanford, D. C. Brown, and D. Plotnick, “Alternative representations and object classification of circular synthetic aperture in-air acoustic data,” The Journal of the Acoustical Society of America, vol. 148, no. 4_Supplement, pp. 2661–2661, 2020.
  50. A. Reed, T. Blanford, D. C. Brown, and S. Jayasuriya, “Sinr: Deconvolving circular sas images using implicit neural representations,” IEEE Journal of Selected Topics in Signal Processing, 2022.
  51. M. Qadri, M. Kaess, and I. Gkioulekas, “Neural implicit surface reconstruction using imaging sonar,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 1040–1047.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube