Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Latent Diffusion Prior Enhanced Deep Unfolding for Snapshot Spectral Compressive Imaging (2311.14280v2)

Published 24 Nov 2023 in eess.IV and cs.CV

Abstract: Snapshot compressive spectral imaging reconstruction aims to reconstruct three-dimensional spatial-spectral images from a single-shot two-dimensional compressed measurement. Existing state-of-the-art methods are mostly based on deep unfolding structures but have intrinsic performance bottlenecks: $i$) the ill-posed problem of dealing with heavily degraded measurement, and $ii$) the regression loss-based reconstruction models being prone to recover images with few details. In this paper, we introduce a generative model, namely the latent diffusion model (LDM), to generate degradation-free prior to enhance the regression-based deep unfolding method. Furthermore, to overcome the large computational cost challenge in LDM, we propose a lightweight model to generate knowledge priors in deep unfolding denoiser, and integrate these priors to guide the reconstruction process for compensating high-quality spectral signal details. Numeric and visual comparisons on synthetic and real-world datasets illustrate the superiority of our proposed method in both reconstruction quality and computational efficiency. Code will be released.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Img. Sci., 2(1):183–202, 2009.
  2. A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Transactions on Image Processing, 16(12):2992–3004, 2007.
  3. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  4. Coarse-to-fine sparse transformer for hyperspectral image reconstruction. In European Conference on Computer Vision, pages 686–704. Springer, 2022a.
  5. Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17502–17511, 2022b.
  6. Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging. Advances in Neural Information Processing Systems, 35:37749–37761, 2022c.
  7. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3:84–98, 2017.
  8. Learning sparse codes for hyperspectral imagery. IEEE Journal of Selected Topics in Signal Processing, 5(5):963–978, 2011.
  9. Hierarchical integration diffusion model for realistic image deblurring. arXiv preprint arXiv:2305.12966, 2023.
  10. Recurrent neural networks for snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2264–2281, 2022.
  11. High-quality hyperspectral reconstruction using a spectral prior. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017), 36(6):218:1–13, 2017.
  12. Residual degradation learning unfolding framework with mixing priors across spectral and spatial for compressive spectral imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22262–22271, 2023.
  13. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10021–10030, 2023.
  14. Single-shot compressive spectral imaging with a dual-disperser architecture. Optics Express, 15(21):14013–14027, 2007.
  15. Imaging spectrometry for earth remote sensing. science, 228(4704):1147–1153, 1985.
  16. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35:27953–27965, 2022.
  17. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  18. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  19. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  20. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.07696, 2022.
  21. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  22. Hdnet: High-resolution dual-domain learning for spectral compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17542–17551, 2022.
  23. Deep gaussian scale mixture prior for spectral compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16216–16225, 2021.
  24. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  25. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
  26. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics, 49(36):6824–6833, 2010.
  27. Prior-based tensor approximation for anomaly detection in hyperspectral imagery. IEEE Transactions on Neural Networks and Learning Systems, 33(3):1037–1050, 2020.
  28. Pixel adaptive deep unfolding transformer for hyperspectral image reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959–12968, 2023.
  29. Deep learning for hyperspectral image classification: An overview. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6690–6709, 2019.
  30. Generalized alternating projection for weighted-2,1 minimization with applications to model-based compressive sensing. SIAM Journal on Imaging Sciences, 7(2):797–823, 2014.
  31. Rank minimization for snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):2990–3006, 2019.
  32. Medical hyperspectral imaging: a review. Journal of biomedical optics, 19(1):010901–010901, 2014.
  33. Rafnet: Recurrent attention fusion network of hyperspectral and multispectral images. Signal Processing, 177:107737, 2020.
  34. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  35. Deep tensor admm-net for snapshot compressive imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10223–10232, 2019.
  36. Gap-net for snapshot compressive imaging. arXiv preprint arXiv:2012.08364, 2020a.
  37. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In European Conference on Computer Vision (ECCV), 2020b.
  38. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In European conference on computer vision, pages 187–204. Springer, 2020c.
  39. Deep unfolding for snapshot compressive imaging. International Journal of Computer Vision, pages 1–26, 2023.
  40. λ𝜆\lambdaitalic_λ-net: Reconstruct hyperspectral images from a snapshot measurement. In IEEE/CVF Conference on Computer Vision (ICCV), 2019.
  41. Multispectral imaging using multiplexed illumination. In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  42. Siamese transformer network for hyperspectral image target detection. IEEE Transactions on Geoscience and Remote Sensing, 60:1–19, 2022.
  43. Multiscale structure guided diffusion for image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10721–10733, 2023.
  44. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  45. Plug-and-play methods provably converge with properly trained denoisers. In International Conference on Machine Learning, pages 5546–5557. PMLR, 2019.
  46. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022.
  47. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  48. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  49. Aziz ul Rehman and Shahzad Ahmad Qureshi. A review of the medical hyperspectral imaging systems and unmixing algorithms’ in biological tissues. Photodiagnosis and Photodynamic Therapy, 33:102165, 2021.
  50. Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 39–48, 2017.
  51. Tracking via object reflectance using a hyperspectral video camera. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pages 44–51. IEEE, 2010.
  52. Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47(10):B44–B51, 2008a.
  53. Single disperser design for coded aperture snapshot spectral imaging. Applied optics, 47(10):B44–B51, 2008b.
  54. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(10):2104–2111, 2017.
  55. Dnu: Deep non-local unrolling for computational spectral imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1661–1671, 2020.
  56. Snapshot spectral compressive imaging reconstruction using convolution and contextual transformer. Photonics Research, 10(8):1848–1858, 2022.
  57. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  58. Diffir: Efficient diffusion model for image restoration. arXiv preprint arXiv:2303.09472, 2023.
  59. Degradation-aware dynamic fourier-based network for spectral compressive imaging. IEEE Transactions on Multimedia, 2023.
  60. Xin Yuan. Generalized alternating projection based total variation minimization for compressive sensing. In 2016 IEEE International Conference on Image Processing (ICIP), pages 2539–2543, 2016.
  61. Plug-and-play algorithms for large-scale snapshot compressive imaging. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  62. Plug-and-play algorithms for video snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  63. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
  64. Computational hyperspectral imaging based on dimension-discriminative low-rank tensor recovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10183–10192, 2019.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now