Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural Spectro-polarimetric Fields (2306.12562v2)

Published 21 Jun 2023 in cs.CV and eess.IV

Abstract: Modeling the spatial radiance distribution of light rays in a scene has been extensively explored for applications, including view synthesis. Spectrum and polarization, the wave properties of light, are often neglected due to their integration into three RGB spectral bands and their non-perceptibility to human vision. However, these properties are known to encompass substantial material and geometric information about a scene. Here, we propose to model spectro-polarimetric fields, the spatial Stokes-vector distribution of any light ray at an arbitrary wavelength. We present Neural Spectro-polarimetric Fields (NeSpoF), a neural representation that models the physically-valid Stokes vector at given continuous variables of position, direction, and wavelength. NeSpoF manages inherently noisy raw measurements, showcases memory efficiency, and preserves physically vital signals - factors that are crucial for representing the high-dimensional signal of a spectro-polarimetric field. To validate NeSpoF, we introduce the first multi-view hyperspectral-polarimetric image dataset, comprised of both synthetic and real-world scenes. These were captured using our compact hyperspectral-polarimetric imaging system, which has been calibrated for robustness against system imperfections. We demonstrate the capabilities of NeSpoF on diverse scenes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Edward H Adelson and John YA Wang. 1992. Single lens stereo with a plenoptic camera. IEEE transactions on pattern analysis and machine intelligence 14, 2 (1992), 99–106.
  2. Hyperspectral imaging for tumor segmentation on pigmented skin lesions. Journal of Biomedical Optics 27, 10 (2022), 106007.
  3. Mantis shrimp–inspired organic photodetector for simultaneous hyperspectral and polarimetric imaging. Science Advances 7, 10 (2021), eabe3196.
  4. Amrit Ambirajan and Dwight C Look Jr. 1995. Optimum angles for a polarimeter: part I. Optical Engineering 34, 6 (1995), 1651–1655.
  5. Illumination estimation from specular highlight in a multi-spectral image. Optics Express 23, 13 (2015), 17008–17023.
  6. Deep shape from polarization. In European Conference on Computer Vision. Springer, 554–571.
  7. Seung-Hwan Baek and Felix Heide. 2021. Polarimetric spatio-temporal light transport probing. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–18.
  8. Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Trans. Graph. 37, 6 (2018), 268–1.
  9. Image-based acquisition and modeling of polarimetric reflectance. ACM Trans. Graph. 39, 4 (2020), 139.
  10. Static full-Stokes Fourier transform imaging spectropolarimeter capturing spectral, polarization, and spatial characteristics. Opt. Express 29, 23 (Nov 2021), 38623–38645. https://doi.org/10.1364/OE.443350
  11. James R Bergen and Edward H Adelson. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1 (1991), 8.
  12. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
  13. SPICES: spectro-polarimetric imaging and characterization of exoplanetary systems. Experimental Astronomy 34, 2 (2012), 355–384.
  14. Tensorial multi-view subspace clustering for polarimetric hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing (2022).
  15. Li-Jen Cheng and George F Reyes. 1995. AOTF polarimetric hyperspectral imaging for mine detection. In Detection Technologies for Mines and Minelike Targets, Vol. 2496. SPIE, 305–311.
  16. Edward Collett. 2005. Field guide to polarization. Spie Bellingham, WA.
  17. Snapshot hyperspectral light field tomography. Optica 8, 12 (2021), 1552–1558.
  18. PANDORA: Polarization-Aided Neural Decomposition Of Radiance. arXiv preprint arXiv:2203.13458 (2022).
  19. Unstructured light fields. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 305–314.
  20. Fernando del Molino and Adolfo Muñoz. 2019. Polarization mapping. Computers & Graphics 83 (2019), 42–50.
  21. Spectropolarimetric imaging for object recognition. In 26th AIPR Workshop: Exploiting New Image Sources and Sensors, Vol. 3240. SPIE, 8–18.
  22. Deep polarization imaging for 3D shape and SVBRDF acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15567–15576.
  23. Polarimetric helmholtz stereopsis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5037–5046.
  24. Building an orthonormal basis, revisited. JCGT 6, 1 (2017).
  25. Compressive full-Stokes polarization and flexible hyperspectral imaging with efficient reconstruction. Optics and Lasers in Engineering 160 (2023), 107256.
  26. Four-dimensional compressed spectropolarimetric imaging. Signal Processing (2022).
  27. Hyperspectral polarization-compressed imaging and reconstruction with sparse basis optimized by particle swarm optimization. Chemometrics and Intelligent Laboratory Systems 206 (2020), 104163.
  28. Polarimetric normal stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 682–690.
  29. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–10.
  30. Joseph W Goodman and P Sutton. 1996. Introduction to Fourier optics. Quantum and Semiclassical Optics-Journal of the European Optical Society Part B 8, 5 (1996), 1095.
  31. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 43–54.
  32. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3791–3800.
  33. Ultra-compact snapshot spectral light-field imaging. Nature communications 13, 1 (2022), 2732.
  34. Sparse ellipsometry: portable acquisition of polarimetric SVBRDF and shape with unstructured flash photography. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–14.
  35. Hyperspectral image dataset for benchmarking on salient object detection. In 2018 Tenth international conference on quality of multimedia experience (qoMEX). IEEE, 1–3.
  36. Mitsuba 3 renderer. https://mitsuba-renderer.org.
  37. Multi-view stereo reconstruction of dense shape and complex appearance. International Journal of Computer Vision 63, 3 (2005), 175–189.
  38. Polarized 3d: High-quality depth sensing with polarization cues. In Proceedings of the IEEE International Conference on Computer Vision. 3370–3378.
  39. James T Kajiya and Brian P Von Herzen. 1984. Ray tracing volume densities. ACM SIGGRAPH computer graphics 18, 3 (1984), 165–174.
  40. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–10.
  41. Deep Polarization Cues for Transparent Object Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  42. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–14.
  43. Illuminant estimation in multispectral imaging. JOSA A 34, 7 (2017), 1085–1098.
  44. 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–11.
  45. Accurate polarimetric BRDF for real polarization scene rendering. In European Conference on Computer Vision. Springer, 220–236.
  46. Shape from polarization for complex scenes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12632–12641.
  47. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 31–42.
  48. Light field microscopy. In ACM SIGGRAPH 2006 Papers. 924–934.
  49. Multispectral illumination estimation using deep unrolling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2672–2681.
  50. Polarimetric dehazing utilizing spatial frequency segregation of images. Applied Optics 54, 27 (2015), 8116–8122.
  51. PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8665–8675.
  52. Snapshot spectral polarimetric light field imaging using a single detector. Optics Letters 45, 23 (2020), 6522–6525.
  53. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Transactions on Graphics 32, 4 (2013), 47–1.
  54. Glass Segmentation Using Intensity and Spectral Polarization Cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12622–12631.
  55. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16190–16199.
  56. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.
  57. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision. Springer, 405–421.
  58. Separation of reflection components using color and polarization. International Journal of Computer Vision 21, 3 (1997), 163–186.
  59. Eric Penner and Li Zhang. 2017. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–11.
  60. pCON: Polarimetric Coordinate Networks for Neural Scene Representations. In Proc. CVPR.
  61. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–14.
  62. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).
  63. DeepVoxels: Learning Persistent 3D Feature Embeddings. In Proc. CVPR.
  64. Light Field Neural Rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8269–8279.
  65. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33 (2020), 7537–7547.
  66. Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE transactions on biomedical engineering 68, 4 (2020), 1330–1340.
  67. Hyperspectral imaging for underwater object detection. Sensor Review 41, 2 (2021), 176–191.
  68. Polarization guided specular reflection separation. IEEE Transactions on Image Processing 30 (2021), 7280–7291.
  69. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. (2012).
  70. High-speed videography using a dense camera array. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 2. IEEE, II–II.
  71. High performance imaging using large camera arrays. In ACM SIGGRAPH 2005 Papers. 765–776.
  72. Alexander Wilkie and Andrea Weidlich. 2010. A standardised polarisation visualisation for images. In Proceedings of the 26th Spring Conference on Computer Graphics. 43–50.
  73. Pu Xia and Xuebin Liu. 2016. Image dehazing technique based on polarimetric spectral analysis. Optik 127, 18 (2016), 7350–7358.
  74. A method of removing reflected highlight on images based on polarimetric imaging. Journal of Sensors 2016 (2016).
  75. A real-time distributed light field camera. Rendering Techniques 2002 (2002), 77–86.
  76. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia 2022 Conference Papers. 1–12.
  77. Spectropolarimetric imaging for pathological analysis of skin. Applied optics 48, 10 (2009), D236–D246.
  78. Learning to dehaze with polarization. Advances in Neural Information Processing Systems 34 (2021), 11487–11500.
  79. 3d human shape reconstruction from a polarization image. In European Conference on Computer Vision. Springer, 351–368.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now