Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging (2312.13319v2)

Published 20 Dec 2023 in eess.IV and cs.CV

Abstract: Dual-Camera Compressed Hyperspectral Imaging (DCCHI) offers the capability to reconstruct 3D Hyperspectral Image (HSI) by fusing compressive and Panchromatic (PAN) image, which has shown great potential for snapshot hyperspectral imaging in practice. In this paper, we introduce a novel DCCHI reconstruction network, the Intra-Inter Similarity Exploiting Transformer (In2SET). Our key insight is to make full use of the PAN image to assist the reconstruction. To this end, we propose using the intra-similarity within the PAN image as a proxy for approximating the intra-similarity in the original HSI, thereby offering an enhanced content prior for more accurate HSI reconstruction. Furthermore, we aim to align the features from the underlying HSI with those of the PAN image, maintaining semantic consistency and introducing new contextual information for the reconstruction process. By integrating In2SET into a PAN-guided unrolling framework, our method substantially enhances the spatial-spectral fidelity and detail of the reconstructed images, providing a more comprehensive and accurate depiction of the scene. Extensive experiments conducted on both real and simulated datasets demonstrate that our approach consistently outperforms existing state-of-the-art methods in terms of reconstruction quality and computational complexity. Code will be released.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Processing Magazine, 31(1):105–115, 2013.
  2. Higher-order computational model for coded aperture spectral imaging. Applied Optics, 52(10):D12–D21, 2013.
  3. A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Transactions on Image Processing, 16(12):2992–3004, 2007.
  4. Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and Remote Sensing Magazine, 1(2):6–36, 2013.
  5. Coarse-to-fine sparse transformer for hyperspectral image reconstruction. In European Conference on Computer Vision, pages 686–704, 2022a.
  6. Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pages 17502–17511, 2022b.
  7. Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging. In Advances in Neural Information Processing Systems, pages 37749–37761, 2022c.
  8. Hyperspectral imaging in the medical field: Present and future. Applied Spectroscopy Reviews, 49(6):435–447, 2014.
  9. Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world. IEEE Signal Processing Magazine, 33(5):95–108, 2016.
  10. Prior image guided snapshot compressive spectral imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  11. High-quality hyperspectral reconstruction using a spectral prior. ACM Transactions on Graphics, 36(6):1–13, 2017.
  12. Deep spatial–spectral representation learning for hyperspectral image denoising. IEEE Transactions on Computational Imaging, 5(4):635–648, 2019.
  13. Residual degradation learning unfolding framework with mixing priors across spectral and spatial for compressive spectral imaging. In IEEE Conference on Computer Vision and Pattern Recognition, pages 22262–22271, 2023.
  14. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE Journal of Selected Topics in Signal Processing, 1(4):586–597, 2007.
  15. Exploiting spectral-spatial correlation for coded hyperspectral image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3727–3736, 2016.
  16. Single-shot compressive spectral imaging with a dual-disperser architecture. Optics Express, 15(21):14013–14027, 2007.
  17. Non-local meets global: An integrated paradigm for hyperspectral denoising. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6868–6877, 2019.
  18. Fast hyperspectral image recovery of dual-camera compressive hyperspectral imaging via non-iterative subspace-based fusion. IEEE Transactions on Image Processing, 30:7170–7183, 2021.
  19. Hdnet: High-resolution dual-domain learning for spectral compressive imaging. In IEEE Conference on Computer Vision and Pattern Recognition, pages 17542–17551, 2022.
  20. Deep gaussian scale mixture prior for spectral compressive imaging. In IEEE Conference on Computer Vision and Pattern Recognition, pages 16216–16225, 2021.
  21. Adam: A method for stochastic optimization. arXiv preprint, 2014.
  22. Edmund Y Lam. Computational photography with plenoptic camera and light field capture: Tutorial. JOSA A, 32(11):2021–2032, 2015.
  23. Hyperspectral image segmentation using a new bayesian approach with active learning. IEEE Transactions on Geoscience and Remote Sensing, 49(10):3947–3960, 2011.
  24. Pixel adaptive deep unfolding transformer for hyperspectral image reconstruction. In IEEE International Conference on Computer Vision, pages 12959–12968, 2023.
  25. The reuven ramaty high-energy solar spectroscopic imager (rhessi). In The Reuven Ramaty High-Energy Solar Spectroscopic Imager (RHESSI): Mission Description and Early Results, pages 3–32, 2003.
  26. Spatial-spectral encoded compressive hyperspectral imaging. ACM Transactions on Graphics, 33(6):1–11, 2014.
  27. Rank minimization for snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):2990–3006, 2018.
  28. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In European Conference on Computer Vision, pages 187–204, 2020.
  29. Self-supervised neural networks for spectral snapshot compressive imaging. In IEEE International Conference on Computer Vision, pages 2622–2631, 2021.
  30. Lambda-net: Reconstruct hyperspectral images from a snapshot measurement. In IEEE International Conference on Computer Vision, pages 4059–4069, 2019.
  31. John L Nazareth. Conjugate gradient method. Wiley Interdisciplinary Reviews: Computational Statistics, 1(3):348–353, 2009.
  32. Face recognition in hyperspectral images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12):1552–1560, 2003.
  33. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 234–241, 2015.
  34. Hypersectral imaging for military and security applications: Combining myriad processing and sensing techniques. IEEE Geoscience and Remote Sensing Magazine, 7(2):101–117, 2019.
  35. Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47(10):B44–B51, 2008.
  36. Dual-camera design for coded aperture snapshot spectral imaging. Applied Optics, 54(4):848–858, 2015a.
  37. High-speed hyperspectral video acquisition with a dual-camera architecture. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4942–4950, 2015b.
  38. Simultaneous depth and spectral imaging with a cross-modal stereo system. IEEE Transactions on Circuits and Systems for Video Technology, 28(3):812–817, 2016.
  39. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(10):2104–2111, 2017.
  40. High-speed hyperspectral video acquisition by combining nyquist and compressive sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4):857–870, 2018a.
  41. Hyperreconnet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging. IEEE Transactions on Image Processing, 28(5):2257–2270, 2018b.
  42. Hyperspectral image reconstruction using a deep spatial-spectral prior. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8032–8041, 2019.
  43. Dnu: deep non-local unrolling for computational spectral imaging. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1661–1671, 2020.
  44. Adaptive dimension-discriminative low-rank tensor recovery for computational hyperspectral imaging. International Journal of Computer Vision, 129(10):2907–2926, 2021.
  45. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  46. Attention-based pyramid network for segmentation and classification of high-resolution and hyperspectral remote sensing images. Remote Sensing, 12(21):3501, 2020.
  47. Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum. IEEE Transactions on Image Processing, 19(9):2241–2253, 2010.
  48. Xin Yuan. Generalized alternating projection based total variation minimization for compressive sensing. In IEEE International Conference on Image Processing, pages 2539–2543, 2016.
  49. Compressive hyperspectral imaging with side information. IEEE Journal of Selected Topics in Signal Processing, 9(6):964–976, 2015.
  50. Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 38(2):65–88, 2021.
  51. Compressive hyperspectral imaging with non-zero mean noise. Optics Express, 27(13):17449–17462, 2019a.
  52. Fast parallel implementation of dual-camera compressive hyperspectral imaging system. IEEE Transactions on Circuits and Systems for Video Technology, 29(11):3404–3414, 2018.
  53. Computational hyperspectral imaging based on dimension-discriminative low-rank tensor recovery. In IEEE International Conference on Computer Vision, pages 10183–10192, 2019b.
  54. Handheld snapshot multi-spectral camera at tens-of-megapixel resolution. Nature Communications, 14(1):5043, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now