Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images (2312.15942v2)

Published 26 Dec 2023 in cs.CV and eess.IV

Abstract: Panoramic imaging research on geometry recovery and High Dynamic Range (HDR) reconstruction becomes a trend with the development of Extended Reality (XR). Neural Radiance Fields (NeRF) provide a promising scene representation for both tasks without requiring extensive prior data. However, in the case of inputting sparse Low Dynamic Range (LDR) panoramic images, NeRF often degrades with under-constrained geometry and is unable to reconstruct HDR radiance from LDR inputs. We observe that the radiance from each pixel in panoramic images can be modeled as both a signal to convey scene lighting information and a light source to illuminate other pixels. Hence, we propose the irradiance fields from sparse LDR panoramic images, which increases the observation counts for faithful geometry recovery and leverages the irradiance-radiance attenuation for HDR reconstruction. Extensive experiments demonstrate that the irradiance fields outperform state-of-the-art methods on both geometry recovery and HDR reconstruction and validate their effectiveness. Furthermore, we show a promising byproduct of spatially-varying lighting estimation. The code is available at https://github.com/Lu-Zhan/Pano-NeRF.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. HRDFuse: Monocular 360deg depth estimation by collaboratively learning holistic-with-regional depth distributions. In CVPR, 13273–13282.
  2. Arrighetti, W. 2017. The academy color encoding system (ACES): A professional color-management framework for production, post-production and archival of still and motion pictures. Journal of Imaging, 3(4): 40.
  3. Azimi, M.; et al. 2021. PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR. In 2021 Picture Coding Symposium (PCS), 1–5. IEEE.
  4. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 5855–5864.
  5. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 5470–5479.
  6. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824.
  7. Nerd: Neural reflectance decomposition from image collections. In ICCV, 12684–12694.
  8. Depth estimation from indoor panoramas with neural scene representation. In CVPR, 899–908.
  9. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 14124–14133.
  10. HDR video reconstruction: A coarse-to-fine network and a real-world benchmark dataset. In ICCV, 2502–2511.
  11. Text2Light: Zero-shot text-driven HDR panorama generation. ACM TOG, 41(6).
  12. Stereo radiance fields (SRF): Learning view synthesis for sparse views of novel scenes. In CVPR, 7911–7920.
  13. 3D scene geometry estimation from 360 imagery: A survey. ACM Computing Surveys, 55(4): 1–39.
  14. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In SIGGRAPH, 11–20.
  15. Depth-supervised NeRF: Fewer views and faster training for free. In CVPR, 12882–12891.
  16. Casual indoor HDR radiance capture from omnidirectional images. In BMVC.
  17. HDR-NeRF: High dynamic range neural radiance fields. In CVPR, 18398–18408.
  18. All-around depth from small motion with a spherical panoramic camera. In ECCV, 156–172. Springer.
  19. Putting NeRF on a diet: Semantically consistent few-shot view synthesis. In ICCV, 5885–5894.
  20. HDR-Plenoxels: Self-calibrating high dynamic range radiance fields. In ECCV, 384–401. Springer.
  21. Kajiya, J. T. 1986. The rendering equation. In SIGGRAPH, 143–150.
  22. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3): 165–174.
  23. Adam: A method for stochastic optimization. In ICLR.
  24. 360FusionNeRF: Panoramic neural radiance fields with joint guidance. In IROS, 7202–7209.
  25. HDRNET: Single-image-based HDR reconstruction using channel attention cnn. In ICMSSP, 119–124.
  26. Lighting, reflectance and geometry estimation from 360 panoramic stereo. In CVPR, 10586–10595.
  27. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In CVPR, 2801–2810.
  28. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In CVPR, 2475–2484.
  29. Multi-view inverse rendering for large-scale real-world indoor scenes. In CVPR, 12499–12509.
  30. Single-image HDR reconstruction by learning to reverse the camera pipeline. In CVPR, 1651–1660.
  31. HDR-VDP-3: A multi-metric for predicting image differences, quality and contrast distortions in high dynamic range and regular content. arXiv preprint arXiv:2304.13625.
  32. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, 16190–16199.
  33. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99–106.
  34. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 5480–5490.
  35. Ldr2Hdr: on-the-fly reverse tone mapping of legacy video and photographs. ACM TOG, 26(3).
  36. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 12892–12901.
  37. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM TOG, 39(4): 80–1.
  38. Generalized random walks for fusion of multi-exposure images. IEEE TIP, 20(12): 3634–3646.
  39. ViP-NeRF: Visibility prior for sparse input neural radiance fields. In ACM SIGGRAPH.
  40. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 7495–7504.
  41. Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In CVPR, 8080–8089.
  42. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797.
  43. Volume rendering digest (for NeRF). arXiv preprint arXiv:2209.02417.
  44. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In CVPR, 5491–5500.
  45. Bifuse: Monocular 360 depth estimation via bi-projection fusion. In CVPR, 462–471.
  46. 360SD-Net: 360 stereo depth estimation with learnable cost volume. In 2020 IEEE ICRA, 582–588.
  47. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4): 600–612.
  48. Learning indoor inverse rendering with 3D spatially-varying lighting. In ICCV, 12538–12547.
  49. Rendering with radiance: The art and science of lighting visualization.
  50. DiffusioNeRF: Regularizing neural radiance fields with denoising diffusion models. In CVPR.
  51. Automatic spatially varying illumination recovery of indoor scenes based on a single RGB-D image. IEEE TVCG, 26(4): 1672–1685.
  52. FreeNeRF: Improving few-shot neural rendering with free frequency regularization. In CVPR, 8254–8263.
  53. NeILF: Neural incident light field for physically-based material estimation. In ECCV, 700–716. Springer.
  54. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 4578–4587.
  55. Luminance attentive networks for HDR image and panorama reconstruction. In Computer Graphics Forum, volume 40, 181–192. Wiley Online Library.
  56. Improving 360 monocular depth estimation via non-local dense prediction transformer and joint supervised and self-supervised learning. In AAAI, volume 36, 3224–3233.
  57. Unsupervised depth estimation from monocular videos with hybrid geometric-refined loss and contextual attention. Neurocomputing, 379: 250–261.
  58. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.
  59. NeRFactor: Neural factorization of shape and reflectance under an unknown illumination. ACM TOG, 40(6): 1–18.
  60. Modeling indirect illumination for inverse rendering. In CVPR, 18643–18652.
  61. Glosh: Global-local spherical harmonics for intrinsic image decomposition. In ICCV, 7820–7829.
  62. ACDNet: Adaptively combined dilated convolution for monocular panorama depth estimation. In AAAI, volume 36, 3653–3661.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Lu-Zhan/Pano-NeRF (4 stars)