Papers
Topics
Authors
Recent
2000 character limit reached

Aquatic-GS: A Hybrid 3D Representation for Underwater Scenes (2411.00239v2)

Published 31 Oct 2024 in cs.CV

Abstract: Representing underwater 3D scenes is a valuable yet complex task, as attenuation and scattering effects during underwater imaging significantly couple the information of the objects and the water. This coupling presents a significant challenge for existing methods in effectively representing both the objects and the water medium simultaneously. To address this challenge, we propose Aquatic-GS, a hybrid 3D representation approach for underwater scenes that effectively represents both the objects and the water medium. Specifically, we construct a Neural Water Field (NWF) to implicitly model the water parameters, while extending the latest 3D Gaussian Splatting (3DGS) to model the objects explicitly. Both components are integrated through a physics-based underwater image formation model to represent complex underwater scenes. Moreover, to construct more precise scene geometry and details, we design a Depth-Guided Optimization (DGO) mechanism that uses a pseudo-depth map as auxiliary guidance. After optimization, Aquatic-GS enables the rendering of novel underwater viewpoints and supports restoring the true appearance of underwater scenes, as if the water medium were absent. Extensive experiments on both simulated and real-world datasets demonstrate that Aquatic-GS surpasses state-of-the-art underwater 3D representation methods, achieving better rendering quality and real-time rendering performance with a 410x increase in speed. Furthermore, regarding underwater image restoration, Aquatic-GS outperforms representative dewatering methods in color correction, detail recovery, and stability. Our models, code, and datasets can be accessed at https://aquaticgs.github.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. W. Wang, B. Joshi, N. Burgdorfer, K. Batsosc, A. Q. Lid, P. Mordohaia, and I. Rekleitisb, “Real-time dense 3d mapping of underwater environments,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 5184–5191.
  2. M. Yuval and T. Treibitz, “Releasing a dataset of 3d models of artificial reefs from the northern red-sea for 3d printing and virtual reality applications,” Remote Sensing Applications: Society and Environment, vol. 36, p. 101305, 2024.
  3. M. Johnson-Roberson, M. Bryson, A. Friedman, O. Pizarro, G. Troni, P. Ozog, and J. C. Henderson, “High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology,” Journal of Field Robotics, vol. 34, no. 4, pp. 625–643, 2017.
  4. G. Llorach-Tó, E. Martínez, J. D. R. Fernández, and E. García-Ladona, “Experience obsea: a web-based 3d virtual environment of a seafloor observatory,” in OCEANS 2023-Limerick.   IEEE, 2023, pp. 1–6.
  5. C. Boittiaux, R. Marxer, C. Dune, A. Arnaubec, M. Ferrera, and V. Hugel, “Sucre: Leveraging scene structure for underwater color restoration,” in 2024 International Conference on 3D Vision (3DV).   IEEE, 2024, pp. 1488–1497.
  6. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  7. B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023.
  8. T. Zhang and M. Johnson-Roberson, “Beyond nerf underwater: Learning neural reflectance fields for true color correction of marine imagery,” IEEE Robotics and Automation Letters, 2023.
  9. D. Levy, A. Peleg, N. Pearl, D. Rosenbaum, D. Akkaynak, S. Korman, and T. Treibitz, “Seathru-nerf: Neural radiance fields in scattering media,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 56–65.
  10. Y. Tang, C. Zhu, R. Wan, C. Xu, and B. Shi, “Neural underwater scene representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 11 780–11 789.
  11. B. Fei, J. Xu, R. Zhang, Q. Zhou, W. Yang, and Y. He, “3d gaussian splatting as new era: A survey,” IEEE Transactions on Visualization and Computer Graphics, 2024.
  12. Y. Bekerman, S. Avidan, and T. Treibitz, “Unveiling optical properties in underwater images,” in 2020 IEEE International Conference on Computational Photography (ICCP).   IEEE, 2020, pp. 1–12.
  13. D. Akkaynak and T. Treibitz, “A revised underwater image formation model,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6723–6732.
  14. D. Nakath, M. She, Y. Song, and K. Köser, “In-situ joint light and medium estimation for underwater color restoration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3731–3740.
  15. L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10 371–10 381.
  16. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5470–5479.
  17. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM transactions on graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022.
  18. Y. Yang, S. Zhang, Z. Huang, Y. Zhang, and M. Tan, “Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 901–15 911.
  19. L. Xu, Y. Xiangli, S. Peng, X. Pan, N. Zhao, C. Theobalt, B. Dai, and D. Lin, “Grid-guided neural radiance fields for large urban scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8296–8306.
  20. A. Ramazzina, M. Bijelic, S. Walz, A. Sanvito, D. Scheuble, and F. Heide, “Scatternerf: Seeing through fog with physically-based inverse neural rendering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17 957–17 968.
  21. H. Dahmani, M. Bennehar, N. Piasco, L. Roldao, and D. Tsishkou, “Swag: Splatting in the wild images with appearance-conditioned gaussians,” arXiv preprint arXiv:2403.10427, 2024.
  22. Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang, “Fsgs: Real-time few-shot view synthesis using gaussian splatting,” in European Conference on Computer Vision.   Springer, 2025, pp. 145–163.
  23. H. Xiong, S. Muttukuru, R. Upadhyay, P. Chari, and A. Kadambi, “Sparsegs: Real-time 360 {{\{{\\\backslash\deg}}\}} sparse view synthesis using gaussian splatting,” arXiv preprint arXiv:2312.00206, 2023.
  24. J. Li, J. Zhang, X. Bai, J. Zheng, X. Ning, J. Zhou, and L. Gu, “Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20 775–20 785.
  25. A. Paliwal, W. Ye, J. Xiong, D. Kotovenko, R. Ranjan, V. Chandra, and N. K. Kalantari, “Coherentgs: Sparse novel view synthesis with coherent 3d gaussians,” arXiv preprint arXiv:2403.19495, 2024.
  26. W. Chen and L. Liu, “Deblur-gs: 3d gaussian splatting from camera motion blurred images,” Proceedings of the ACM on Computer Graphics and Interactive Techniques, vol. 7, no. 1, pp. 1–15, 2024.
  27. B. Lee, H. Lee, X. Sun, U. Ali, and E. Park, “Deblurring 3d gaussian splatting,” arXiv preprint arXiv:2401.00834, 2024.
  28. J. Lin, Z. Li, X. Tang, J. Liu, S. Liu, J. Liu, Y. Lu, X. Wu, S. Xu, Y. Yan et al., “Vastgaussian: Vast 3d gaussians for large scene reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5166–5175.
  29. G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4d gaussian splatting for real-time dynamic scene rendering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20 310–20 320.
  30. Y. Gong, “Eggs: Edge guided gaussian splatting for radiance fields,” in Proceedings of the 29th International ACM Conference on 3D Web Technology, 2024, pp. 1–5.
  31. J. Zhang, F. Zhan, M. Xu, S. Lu, and E. Xing, “Fregs: 3d gaussian splatting with progressive frequency regularization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 21 424–21 433.
  32. J. Hyung, S. Hong, S. Hwang, J. Lee, J. Choo, and J.-H. Kim, “Effective rank analysis and regularization for enhanced 3d gaussian splatting,” arXiv preprint arXiv:2406.11672, 2024.
  33. Z. Zhang, W. Hu, Y. Lao, T. He, and H. Zhao, “Pixel-gs: Density control with pixel-aware gradient for 3d gaussian splatting,” arXiv preprint arXiv:2403.15530, 2024.
  34. Z. Ye, W. Li, S. Liu, P. Qiao, and Y. Dou, “Absgs: Recovering fine details in 3d gaussian splatting,” in ACM Multimedia 2024, 2024.
  35. G. Fang and B. Wang, “Mini-splatting: Representing scenes with a constrained number of gaussians,” arXiv preprint arXiv:2403.14166, 2024.
  36. M. Turkulainen, X. Ren, I. Melekhov, O. Seiskari, E. Rahtu, and J. Kannala, “Dn-splatter: Depth and normal priors for gaussian splatting and meshing,” arXiv preprint arXiv:2403.17822, 2024.
  37. H. Li, W. Song, T. Xu, A. Elsig, and J. Kulhanek, “Watersplatting: Fast underwater 3d scene reconstruction using gaussian splatting,” arXiv preprint arXiv:2408.08206, 2024.
  38. T. Zhang, W. Zhi, K. Huang, J. Mangelson, C. Barbalata, and M. Johnson-Roberson, “Recgs: Removing water caustic with recurrent gaussian splatting,” arXiv preprint arXiv:2407.10318, 2024.
  39. C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” in 2012 IEEE conference on computer vision and pattern recognition.   IEEE, 2012, pp. 81–88.
  40. W. Zhang, P. Zhuang, H.-H. Sun, G. Li, S. Kwong, and C. Li, “Underwater image enhancement via minimal color loss and locally adaptive contrast enhancement,” IEEE Transactions on Image Processing, vol. 31, pp. 3997–4010, 2022.
  41. J. Zhou, S. Wang, Z. Lin, Q. Jiang, and F. Sohel, “A pixel distribution remapping and multi-prior retinex variational model for underwater image enhancement,” IEEE Transactions on Multimedia, 2024.
  42. M. Badran and M. Torki, “Daut: Underwater image enhancement using depth aware u-shape transformer,” in 2023 IEEE International Conference on Image Processing (ICIP).   IEEE, 2023, pp. 1830–1834.
  43. C. Li, C. Guo, W. Ren, R. Cong, J. Hou, S. Kwong, and D. Tao, “An underwater image enhancement benchmark dataset and beyond,” IEEE transactions on image processing, vol. 29, pp. 4376–4389, 2019.
  44. Y. Xie, L. Kong, K. Chen, Z. Zheng, X. Yu, Z. Yu, and B. Zheng, “Uveb: A large-scale benchmark and baseline towards real-world underwater video enhancement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 358–22 367.
  45. Z. Wang, L. Shen, M. Xu, M. Yu, K. Wang, and Y. Lin, “Domain adaptation for underwater image enhancement,” IEEE Transactions on Image Processing, vol. 32, pp. 1442–1457, 2023.
  46. R. Liu, Z. Jiang, S. Yang, and X. Fan, “Twin adversarial contrastive learning for underwater image enhancement and beyond,” IEEE Transactions on Image Processing, vol. 31, pp. 4922–4936, 2022.
  47. P. L. Drews, E. R. Nascimento, S. S. Botelho, and M. F. M. Campos, “Underwater depth estimation and image restoration based on single images,” IEEE computer graphics and applications, vol. 36, no. 2, pp. 24–35, 2016.
  48. Y.-T. Peng and P. C. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE transactions on image processing, vol. 26, no. 4, pp. 1579–1594, 2017.
  49. D. Akkaynak and T. Treibitz, “Sea-thru: A method for removing water from underwater images,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1682–1691.
  50. D. Berman, D. Levy, S. Avidan, and T. Treibitz, “Underwater single image color restoration using haze-lines and a new quantitative dataset,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 8, pp. 2822–2837, 2020.
  51. Z. Fu, H. Lin, Y. Yang, S. Chai, L. Sun, Y. Huang, and X. Ding, “Unsupervised underwater image restoration: From a homology perspective,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, 2022, pp. 643–651.
  52. A. V. Sethuraman, M. S. Ramanagopal, and K. A. Skinner, “Waternerf: Neural radiance fields for underwater scenes,” in OCEANS 2023-MTS/IEEE US Gulf Coast.   IEEE, 2023, pp. 1–7.
  53. R. Habel, B. Mustata, and M. Wimmer, “Efficient spherical harmonics lighting with the preetham skylight model.” in Eurographics (short papers), 2008, pp. 119–122.
  54. S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5501–5510.
  55. M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Ewa volume splatting,” in Proceedings Visualization, 2001. VIS’01.   IEEE, 2001, pp. 29–538.
  56. C. Reiser, S. Garbin, P. Srinivasan, D. Verbin, R. Szeliski, B. Mildenhall, J. Barron, P. Hedman, and A. Geiger, “Binary opacity grids: Capturing fine geometric detail for mesh-based view synthesis,” ACM Transactions on Graphics (TOG), vol. 43, no. 4, pp. 1–14, 2024.
  57. D. Rebain, M. Matthews, K. M. Yi, D. Lagun, and A. Tagliasacchi, “Lolnerf: Learn from one look,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1558–1567.
  58. L. Jiang, B. Dai, W. Wu, and C. C. Loy, “Focal frequency loss for image reconstruction and synthesis,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 13 919–13 929.
  59. C. Korkmaz, A. M. Tekalp, and Z. Dogan, “Training generative image super-resolution models by wavelet-domain losses enables better control of artifacts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5926–5936.
  60. Y. Liu, C. Yu, J. Cheng, Z. J. Wang, and X. Chen, “Mm-net: A mixformer-based multi-scale network for anatomical and functional image fusion,” IEEE Transactions on Image Processing, vol. 33, pp. 2197–2212, 2024.
  61. J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4104–4113.
  62. Z. Murez, T. Treibitz, R. Ramamoorthi, and D. Kriegman, “Photometric stereo in a scattering medium,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 3415–3423.
  63. L. Peng, C. Zhu, and L. Bian, “U-shape transformer for underwater image enhancement,” IEEE Transactions on Image Processing, vol. 32, pp. 3066–3079, 2023.
  64. G. Sharma, W. Wu, and E. N. Dalal, “The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations,” COLOR research & application, vol. 30, no. 1, pp. 21–30, 2005.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 4 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube