Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting (2405.15125v4)

Published 24 May 2024 in cs.CV

Abstract: High dynamic range (HDR) novel view synthesis (NVS) aims to create photorealistic images from novel viewpoints using HDR imaging techniques. The rendered HDR images capture a wider range of brightness levels containing more details of the scene than normal low dynamic range (LDR) images. Existing HDR NVS methods are mainly based on NeRF. They suffer from long training time and slow inference speed. In this paper, we propose a new framework, High Dynamic Range Gaussian Splatting (HDR-GS), which can efficiently render novel HDR views and reconstruct LDR images with a user input exposure time. Specifically, we design a Dual Dynamic Range (DDR) Gaussian point cloud model that uses spherical harmonics to fit HDR color and employs an MLP-based tone-mapper to render LDR color. The HDR and LDR colors are then fed into two Parallel Differentiable Rasterization (PDR) processes to reconstruct HDR and LDR views. To establish the data foundation for the research of 3D Gaussian splatting-based methods in HDR NVS, we recalibrate the camera parameters and compute the initial positions for Gaussian point clouds. Experiments demonstrate that our HDR-GS surpasses the state-of-the-art NeRF-based method by 3.84 and 1.91 dB on LDR and HDR NVS while enjoying 1000x inference speed and only requiring 6.3% training time. Code and recalibrated data will be publicly available at https://github.com/caiyuanhao1998/HDR-GS . A brief video introduction of our work is available at https://youtu.be/wtU7Kcwe7ck

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. X. Huang, Q. Zhang, Y. Feng, H. Li, X. Wang, and Q. Wang, “Hdr-nerf: High dynamic range neural radiance fields,” in CVPR, 2022.
  2. Z. Yang, Y. Chai, D. Anguelov, Y. Zhou, P. Sun, D. Erhan, S. Rafferty, and H. Kretzschmar, “Surfelgan: Synthesizing realistic sensor data for autonomous driving,” in CVPR, 2020.
  3. S. Huang, Z. Gojcic, Z. Wang, F. Williams, Y. Kasten, S. Fidler, K. Schindler, and O. Litany, “Neural lidar fields for novel view synthesis,” in ICCV, 2023.
  4. G. Wang, Z. Chen, C. C. Loy, and Z. Liu, “Sparsenerf: Distilling depth ranking for few-shot novel view synthesis,” in ICCV, 2023.
  5. M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in CVPR, 2022.
  6. S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell, “Editing conditional radiance fields,” in ICCV, 2021.
  7. J. Sun, X. Wang, Y. Shi, L. Wang, J. Wang, and Y. Liu, “Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,” ACM ToG, 2022.
  8. Y.-J. Yuan, Y.-T. Sun, Y.-K. Lai, Y. Ma, R. Jia, and L. Gao, “Nerf-editing: geometry editing of neural radiance fields,” in CVPR, 2022.
  9. S. Kobayashi, E. Matsumoto, and V. Sitzmann, “Decomposing nerf for editing via feature field distillation,” in NeurIPS, 2022.
  10. L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt, “Neural actor: Neural free-view synthesis of human actors with pose control,” ACM TOG, 2021.
  11. T. Hu, K. Sarkar, L. Liu, M. Zwicker, and C. Theobalt, “Egorenderer: Rendering human avatars from egocentric camera images,” in ICCV, 2021.
  12. J. Zheng, Y. Jang, A. Papaioannou, C. Kampouris, R. A. Potamias, F. P. Papantoniou, E. Galanakis, A. Leonardis, and S. Zafeiriou, “Ilsh: The imperial light-stage head dataset for human head view synthesis,” in ICCV, 2023.
  13. Z. Zheng, H. Huang, T. Yu, H. Zhang, Y. Guo, and Y. Liu, “Structured local radiance fields for human avatar modeling,” in CVPR, 2022.
  14. B. Mildenhall, P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in ECCV, 2020.
  15. B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, 2023.
  16. P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in SIGGRAPH, 1997.
  17. J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in CVPR, 2016.
  18. R. Szeliski, Computer vision: algorithms and applications. Springer Nature, 2022.
  19. T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion,” in Pacific Conference on Computer Graphics and Applications, 2007.
  20. G. Ward, E. Reinhard, and P. Debevec, “High dynamic range imaging & image-based lighting,” in SIGGRAPH, 2008.
  21. T. Grosch et al., “Fast and robust high dynamic range image generation with camera and object movement,” Vision, Modeling and Visualization, RWTH Aachen, 2006.
  22. K. Jacobs, C. Loscos, and G. Ward, “Automatic high-dynamic range image generation for dynamic scenes,” IEEE Computer Graphics and Applications, 2008.
  23. N. K. Kalantari, R. Ramamoorthi, et al., “Deep high dynamic range imaging of dynamic scenes.,” ACM ToG, 2017.
  24. O. T. Tursun, A. O. Akyüz, A. Erdem, and E. Erdem, “The state of the art in hdr deghosting: A survey and evaluation,” in Computer Graphics Forum, 2015.
  25. Q. Yan, Y. Zhu, and Y. Zhang, “Robust artifact-free high dynamic range imaging of dynamic scenes,” Multimedia Tools and Applications, 2019.
  26. G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “Hdr image reconstruction from a single exposure using deep cnns,” ACM TOG, 2017.
  27. Z. Khan, M. Khanna, and S. Raman, “Fhdr: Hdr image reconstruction from a single ldr image using feedback network,” in IEEE Global Conference on Signal and Information Processing, 2019.
  28. J. Kim, S. Lee, and S.-J. Kang, “End-to-end differentiable learning to hdr image synthesis for multi-exposure images,” in AAAI, 2021.
  29. Z. Liu, W. Lin, X. Li, Q. Rao, T. Jiang, M. Han, H. Fan, J. Sun, and S. Liu, “Adnet: Attention-guided deformable convolutional network for high dynamic range imaging,” in CVPRW, 2021.
  30. R. Chen, B. Zheng, H. Zhang, Q. Chen, C. Yan, G. Slabaugh, and S. Yuan, “Improving dynamic hdr imaging with fusion transformer,” in AAAI, 2023.
  31. Z. Liu, Y. Wang, B. Zeng, and S. Liu, “Ghost-free high dynamic range imaging with context-aware transformer,” in ECCV, 2022.
  32. Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, “Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction,” in CVPR, 2022.
  33. J. W. Song, Y.-I. Park, K. Kong, J. Kwak, and S.-J. Kang, “Selective transhdr: Transformer-based selective hdr imaging using ghost region mask,” in ECCV, 2022.
  34. Y. Cai, H. Bian, J. Lin, H. Wang, R. Timofte, and Y. Zhang, “Retinexformer: One-stage retinex-based transformer for low-light image enhancement,” in ICCV, 2023.
  35. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in ICCV, 2021.
  36. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in CVPR, 2022.
  37. D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” in CVPR, 2022.
  38. W. Hu, Y. Wang, L. Ma, B. Yang, L. Gao, X. Liu, and Y. Ma, “Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields,” in ICCV, 2023.
  39. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Zip-nerf: Anti-aliased grid-based neural radiance fields,” in ICCV, 2023.
  40. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM ToG, 2022.
  41. C. Reiser, R. Szeliski, D. Verbin, P. Srinivasan, B. Mildenhall, A. Geiger, J. Barron, and P. Hedman, “Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes,” TOG, 2023.
  42. A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in ECCV, 2022.
  43. R. Li, H. Gao, M. Tancik, and A. Kanazawa, “Nerfacc: Efficient sampling accelerates nerfs,” in ICCV, 2023.
  44. L. Yariv, P. Hedman, C. Reiser, D. Verbin, P. P. Srinivasan, R. Szeliski, J. T. Barron, and B. Mildenhall, “Bakedsdf: Meshing neural sdfs for real-time view synthesis,” in SIGGRAPH, 2023.
  45. Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi, “Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures,” in CVPR, 2023.
  46. T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia, “Efficientnerf efficient neural radiance fields,” in CVPR, 2023.
  47. Z. Cui, L. Gu, X. Sun, X. Ma, Y. Qiao, and T. Harada, “Aleth-nerf: Illumination adaptive nerf with concealing field assumption,” in AAAI, 2024.
  48. Y. Cai, J. Wang, A. Yuille, Z. Zhou, and A. Wang, “Structure-aware sparse-view x-ray 3d reconstruction,” in CVPR, 2024.
  49. L. Ma, X. Li, J. Liao, Q. Zhang, X. Wang, J. Wang, and P. V. Sander, “Deblur-nerf: Neural radiance fields from blurry images,” in CVPR, 2022.
  50. N. Pearl, T. Treibitz, and S. Korman, “Nan: Noise-aware nerfs for burst-denoising,” in CVPR, 2022.
  51. Z. Yang, H. Yang, Z. Pan, X. Zhu, and L. Zhang, “Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting,” arXiv preprint arXiv:2310.10642, 2023.
  52. G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4d gaussian splatting for real-time dynamic scene rendering,” arXiv preprint arXiv:2310.08528, 2023.
  53. J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan, “Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis,” arXiv preprint arXiv:2308.09713, 2023.
  54. N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, “Splatam: Splat, track & map 3d gaussians for dense rgb-d slam,” arXiv preprint arXiv:2312.02126, 2023.
  55. V. Yugay, Y. Li, T. Gevers, and M. R. Oswald, “Gaussian-slam: Photo-realistic dense slam with gaussian splatting,” arXiv preprint arXiv:2312.10070, 2023.
  56. H. Matsuki, R. Murai, P. H. Kelly, and A. J. Davison, “Gaussian splatting slam,” arXiv preprint arXiv:2312.06741, 2023.
  57. C. Yan, D. Qu, D. Wang, D. Xu, Z. Wang, B. Zhao, and X. Li, “Gs-slam: Dense visual slam with 3d gaussian splatting,” arXiv preprint arXiv:2311.11700, 2023.
  58. Z. Liang, Q. Zhang, Y. Feng, Y. Shan, and K. Jia, “Gs-ir: 3d gaussian splatting for inverse rendering,” arXiv preprint arXiv:2311.16473, 2023.
  59. T. Xie, Z. Zong, Y. Qiu, X. Li, Y. Feng, Y. Yang, and C. Jiang, “Physgaussian: Physics-integrated 3d gaussians for generative dynamics,” arXiv preprint arXiv:2311.12198, 2023.
  60. Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma, “Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces,” arXiv preprint arXiv:2311.17977, 2023.
  61. X. Liu, X. Zhan, J. Tang, Y. Shan, G. Zeng, D. Lin, X. Liu, and Z. Liu, “Humangaussian: Text-driven 3d human generation with gaussian splatting,” arXiv preprint arXiv:2311.17061, 2023.
  62. M. Kocabas, J.-H. R. Chang, J. Gabriel, O. Tuzel, and A. Ranjan, “Hugs: Human gaussian splats,” arXiv preprint arXiv:2311.17910, 2023.
  63. S. Hu and Z. Liu, “Gauhuman: Articulated gaussian splatting from monocular human videos,” arXiv preprint arXiv:, 2023.
  64. J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” arXiv preprint arXiv:2309.16653, 2023.
  65. T. Yi, J. Fang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang, “Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors,” arXiv preprint arXiv:2310.08529, 2023.
  66. Y. Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y. Chen, “Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching,” arXiv preprint arXiv:2311.11284, 2023.
  67. M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Ewa volume splatting,” in Proceedings Visualization, 2001. VIS’01., IEEE, 2001.
  68. G. Kopanas, J. Philip, T. Leimkühler, and G. Drettakis, “Point-based neural rendering with per-view optimization,” in Computer Graphics Forum, 2021.
  69. Y. Cai, Y. Liang, J. Wang, A. Wang, Y. Zhang, X. Yang, Z. Zhou, and A. Yuille, “Radiative gaussian splatting for efficient x-ray novel view synthesis,” arXiv preprint arXiv:2403.04116, 2024.
  70. K. R. Prabhakar, S. Agrawal, D. K. Singh, B. Ashwath, and R. V. Babu, “Towards practical and efficient high-resolution hdr deghosting with cnn,” in ECCV, 2020.
  71. Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in CVPR, 2019.
  72. “Blender.” https://www.blender.org/.
  73. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” in NeurIPS, 2019.
  74. D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
  75. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncell, “Image quality assessment: from error visibility to structural similarity,” TIP, 2004.
  76. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018.
  77. “Photomatix Pro 6.” https://www.hdrsoft.com/.
  78. R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in CVPR, 2021.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces HDR-GS, which leverages a dual dynamic range Gaussian point cloud model to render both HDR and LDR images with exceptional speed.
  • It employs parallel differentiable rasterization to achieve high-quality image synthesis, surpassing existing NeRF methods with improvements of 3.84 dB in LDR PSNR and 1.91 dB in HDR PSNR.
  • The framework recalibrates camera parameters using structure-from-motion, enabling efficient and versatile HDR novel view synthesis for real-time applications.

High Dynamic Range Novel View Synthesis Using Gaussian Splatting

The paper addresses the challenges inherent in high dynamic range (HDR) novel view synthesis (NVS) and introduces a novel framework named High Dynamic Range Gaussian Splatting (HDR-GS). HDR NVS aims to render photorealistic images from new viewpoints using HDR imaging techniques, which provide a higher range of luminance levels than traditional low dynamic range (LDR) imaging. The primary limitations of existing HDR NVS methods, primarily based on Neural Radiance Fields (NeRF), are their extensive training times and slow inference speeds. This paper proposes HDR-GS, which leverages a Dual Dynamic Range (DDR) Gaussian point cloud model to significantly enhance efficiency while maintaining or improving the quality of the synthesized images.

Methodological Innovations

The HDR-GS framework introduces several innovative components that collectively address the deficiencies of current HDR NVS methods:

  1. Dual Dynamic Range (DDR) Gaussian Point Clouds: The DDR model utilizes spherical harmonics (SH) to represent HDR colors and employs a multi-layer perceptron (MLP)-based tone-mapper to render LDR colors based on user-defined exposure times. This contrastive modeling allows HDR-GS to efficiently reconstruct both HDR and LDR views.
  2. Parallel Differentiable Rasterization (PDR): The framework implements two parallel PDR processes to individually reconstruct HDR and LDR images from the Gaussian point clouds. This dual-path approach ensures that HDR-GS can generate images with controllable exposure levels while maintaining high fidelity.
  3. Recalibration of Camera Parameters: To adapt the 3D Gaussian splatting technique for HDR imaging, the authors recalibrate camera parameters and employ structure-from-motion (SfM) techniques to compute initial positions for Gaussian point clouds. This recalibration is pivotal in overcoming the limitations of the normalized device coordinate (NDC) system when applying 3D Gaussian splatting techniques.

Experimental Results

The empirical evaluation shows that HDR-GS significantly outperforms state-of-the-art NeRF-based methods in both LDR and HDR novel view synthesis tasks:

  • Performance Metrics: HDR-GS achieves notable improvements in peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS) scores. Specifically, HDR-GS surpasses the best-performing NeRF-based method by 3.84 dB in PSNR for LDR rendering and 1.91 dB for HDR rendering.
  • Efficiency: HDR-GS demonstrates a dramatic enhancement in efficiency—achieving approximately 1000 times faster inference speed and requiring only 6.3% of the training time compared to current best-performing NeRF methods.

Implications and Future Directions

The proposed HDR-GS framework offers several significant implications for the broader field of computer vision and generative AI:

  1. Practical Applicability: By significantly reducing both the training and inference times, HDR-GS enables real-time applications in augmented reality (AR), virtual reality (VR), and other interactive systems requiring dynamic scene rendering with HDR imagery.
  2. Advancement in HDR Imaging: The dual dynamic range approach provides a new paradigm for HDR image synthesis, which could inspire further research into more efficient and versatile HDR reconstruction methods.
  3. Adaptability: The recalibration of camera parameters for initial Gaussian point cloud positions establishes a robust foundation for extending Gaussian splatting to various imaging scenarios.

Future research could build upon the HDR-GS framework by exploring more advanced model architectures for DDR and improving the efficiency of the PDR processes. Additionally, further studies could investigate the applicability of this framework in other domains such as 3D scene reconstruction, video synthesis, and beyond.

In conclusion, HDR-GS represents a significant advancement in the efficient rendering of HDR images from novel viewpoints, enhancing both the quality and speed of HDR NVS. This framework not only addresses the limitations of existing NeRF-based methods but also sets the stage for further innovations in the field of HDR imaging.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. [2405.15125] HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting (3 points, 0 comments)