Papers
Topics
Authors
Recent
Search
2000 character limit reached

Computational Spectral Imaging with Unified Encoding Model: A Comparative Study and Beyond

Published 20 Dec 2023 in eess.IV and cs.CV | (2312.13310v1)

Abstract: Computational spectral imaging is drawing increasing attention owing to the snapshot advantage, and amplitude, phase, and wavelength encoding systems are three types of representative implementations. Fairly comparing and understanding the performance of these systems is essential, but challenging due to the heterogeneity in encoding design. To overcome this limitation, we propose the unified encoding model (UEM) that covers all physical systems using the three encoding types. Specifically, the UEM comprises physical amplitude, physical phase, and physical wavelength encoding models that can be combined with a digital decoding model in a joint encoder-decoder optimization framework to compare the three systems under a unified experimental setup fairly. Furthermore, we extend the UEMs to ideal versions, namely, ideal amplitude, ideal phase, and ideal wavelength encoding models, which are free from physical constraints, to explore the full potential of the three types of computational spectral imaging systems. Finally, we conduct a holistic comparison of the three types of computational spectral imaging systems and provide valuable insights for designing and exploiting these systems in the future.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Sparse recovery of hyperspectral signal from natural rgb images. In European Conference on Computer Vision (ECCV), pages 19–34. Springer, 2016.
  2. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Processing Magazine (SPM), 31(1):105–115, 2013.
  3. Colored coded aperture design by concentration of measure in compressive spectral imaging. IEEE Transactions on Image Processing (TIP), 23(4):1896–1908, 2014.
  4. Shift-variant color-coded diffractive spectral imaging system. Optica, 8(11):1424–1434, 2021.
  5. Compressive hyperspectral imaging by random separable projections in both the spatial and the spectral domains. Applied Optics (AO), 52(10):D46–D54, 2013.
  6. Single-shot hyperspectral-depth imaging with learned diffractive optics. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2651–2660, 2021.
  7. Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 745–755, 2022a.
  8. Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging. Advances in Neural Information Processing Systems (NIPS), 35:37749–37761, 2022b.
  9. An introduction to compressive sampling. IEEE Signal Processing Magazine (SPM), 25(2):21–30, 2008.
  10. Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world. IEEE Signal Processing Magazine (SPM), 33(5):95–108, 2016.
  11. Statistics of Real-World Hyperspectral Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 193–200, 2011.
  12. High-quality hyperspectral reconstruction using a spectral prior. ACM Transactions on Graphics (TOG), 36(6):218:1–13, 2017.
  13. Compressive spectral imaging with colored-patterned detectors. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7789–7793. IEEE, 2014.
  14. Snapshot colored compressive spectral imager. JOSA A, 32(10):1754–1763, 2015.
  15. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems (NIPS), 28, 2015.
  16. Denoising prior driven deep neural network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 41(10):2305–2318, 2018.
  17. David L Donoho. Compressed sensing. IEEE Transactions on information theory (TIT), 52(4):1289–1306, 2006.
  18. Spectral compressive sensing. Applied and Computational Harmonic Analysis (ACHA), 35(1):111–129, 2013.
  19. Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine (SPM), 25(2):83–91, 2008.
  20. Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. Optica, 7(8):913–922, 2020.
  21. Joint camera spectral response selection and hyperspectral image recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), PP(99):1–1, 2020a.
  22. Joint camera spectral response selection and hyperspectral image recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 44(1):256–272, 2020b.
  23. Learned spectral super-resolution. arXiv preprint arXiv:1703.09470, 2017.
  24. Single-shot compressive spectral imaging with a dual-disperser architecture. Optics Express (OE), 15(21):14013–14027, 2007.
  25. Andrew S Glassner. How to derive a spectrum from an rgb triplet. IEEE Computer Graphics and Applications (CG&A), 9(4):95–99, 1989.
  26. Compressed sensing snapshot spectral imaging by a regular digital camera with an added optical diffuser. Applied Optics (AO), 55(3):432–443, 2016.
  27. A residual dense u-net neural network for image denoising. IEEE Access, 9:31742–31754, 2021.
  28. A hyperspectral and rgb dataset for building facade segmentation. In European Conference on Computer Vision Workshops (ECCVW), pages 258–267. Springer, 2023.
  29. Dd-net: spectral imaging from a monochromatic dispersed and diffused snapshot. Applied Optics (AO), 59(36):11196–11208, 2020.
  30. Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In IEEE International Conference on Computational Photography (ICCP), pages 1–12. IEEE, 2021.
  31. D 2 uf: Deep coded aperture design and unrolling algorithm for compressive spectral image fusion. IEEE Journal of Selected Topics in Signal Processing (J-STSP), 2022.
  32. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Transactions on Graphics (TOG), 38(4):1–13, 2019a.
  33. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Transactions on Graphics (TOG), 2019b.
  34. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  35. Going deeper with contextual cnn for hyperspectral image classification. IEEE Transactions on Image Processing (TIP), 26(10):4843–4855, 2017.
  36. Quantization-aware deep optics for diffractive snapshot hyperspectral imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19780–19789, 2022.
  37. Review of spectral imaging technology in biomedical engineering: achievements and challenges. Journal of Biomedical Optics, 18(10):100901–100901, 2013.
  38. Spatial-spectral encoded compressive hyperspectral imaging. ACM Transactions on Graphics (TOG), 33(6):1–11, 2014a.
  39. Dual-coded compressive hyperspectral imaging. Optics Letters, 39(7):2044–2047, 2014b.
  40. Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity. Optics Express (OE), 30(20):36973–36984, 2022.
  41. Coded aperture compressive temporal imaging. Optics Express (OE), 21(9):10526–10545, 2013.
  42. A single model cnn for hyperspectral image denoising. IEEE Transactions on Geoscience and Remote Sensing (TGRS), 58(4):2516–2529, 2019.
  43. Real-time hyperspectral imaging in hardware via trained metasurface encoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12692–12702, 2022.
  44. Challenges of multi/hyper spectral images in precision agriculture applications. In IOP Conference Series: Earth and Environmental Science (EES), page 012001, 2019.
  45. Hyca: A new technique for hyperspectral compressive sensing. IEEE Transactions on Geoscience and Remote Sensing (TGRS), 53(5):2819–2831, 2014.
  46. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1375–1385, 2020.
  47. l-net: Reconstruct hyperspectral images from a snapshot measurement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4059–4069, 2019.
  48. Enabling smart vision with metasurfaces. Nature Photonics, 17(1):26–35, 2023.
  49. Training-based spectral reconstruction from a single rgb image. In European Conference on Computer Vision (ECCV), pages 186–201. Springer, 2014.
  50. Deeply learned filter response functions for hyperspectral reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4767–4776, 2018.
  51. Compressive sensing hyperspectral imaging by spectral multiplexing with liquid crystal. Journal of imaging, 5(1):3, 2018.
  52. The diffractive achromat full spectrum computational imaging with diffractive optics. In SIGGRAPH ASIA 2016 Virtual Reality meets Physical Reality: Modelling and Simulating Virtual Humans and Environments, pages 1–2. 2016.
  53. Antonio Robles-Kelly. Single image spectral reconstruction for multimedia applications. In Proceedings of the ACM International Conference on Multimedia (ICM), pages 251–260, 2015.
  54. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  55. Compressive spectral imaging based on colored coded apertures. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7799–7803. IEEE, 2014.
  56. Coded aperture optimization in spatial spectral compressive spectral imagers. IEEE Transactions on Computational Imaging (TCI), 6:764–777, 2020.
  57. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
  58. Brian Smits. An rgb-to-spectrum conversion for reflectances. Journal of Graphics tools, 4(4):11–22, 1999.
  59. Deep maximum a posterior estimator for video denoising. International Journal of Computer Vision (IJCV), 129:2827–2845, 2021a.
  60. Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1386–1396, 2020.
  61. Deriving spectra from colors and rendering light interference. IEEE Computer Graphics and Applications (CG&A), 19(4):61–67, 1999.
  62. Unsupervised spatial-spectral network learning for hyperspectral compressive snapshot reconstruction. IEEE Transactions on Geoscience and Remote Sensing (TGRS), 60:1–14, 2021b.
  63. Design and fabrication of diffractive optical elements. Microelectronic Engineering, 34(3-4):219–242, 1997.
  64. Neural nano-optics for high-quality thin lens imaging. Nature Communications, 12(1):6493, 2021.
  65. Attention is all you need. Advances in Neural Information Processing Systems (NIPS), 30, 2017.
  66. Single disperser design for coded aperture snapshot spectral imaging. Applied Optics (AO), 47(10):B44–B51, 2008.
  67. An architecture for compressive imaging. In IEEE International Conference on Image Processing (ICIP), pages 1273–1276. IEEE, 2006.
  68. Hyperreconnet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging. IEEE Transactions on Image Processing (TIP), 28(5):2257–2270, 2018.
  69. Hscnn: Cnn-based hyperspectral image recovery from spectrally undersampled projections. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pages 518–525, 2017.
  70. Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum. Technical report, 2008.
  71. Research progress on the application of spectral imaging technology in pharmaceutical tablet analysis. International Journal of Pharmaceutics, page 122100, 2022.
  72. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3217–3226, 2020a.
  73. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 44(10):6360–6376, 2021.
  74. Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17532–17541, 2022.
  75. Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters(GRSL), 15(5):749–753, 2018.
  76. Amp-net: Denoising-based deep unfolding for compressive image sensing. IEEE Transactions on Image Processing (TIP), 30:1487–1500, 2020b.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up