Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion (2404.09293v2)

Published 14 Apr 2024 in cs.CV

Abstract: In image fusion tasks, images from different sources possess distinct characteristics. This has driven the development of numerous methods to explore better ways of fusing them while preserving their respective characteristics.Mamba, as a state space model, has emerged in the field of natural language processing. Recently, many studies have attempted to extend Mamba to vision tasks. However, due to the nature of images different from causal language sequences, the limited state capacity of Mamba weakens its ability to model image information. Additionally, the sequence modeling ability of Mamba is only capable of spatial information and cannot effectively capture the rich spectral information in images. Motivated by these challenges, we customize and improve the vision Mamba network designed for the image fusion task. Specifically, we propose the local-enhanced vision Mamba block, dubbed as LEVM. The LEVM block can improve local information perception of the network and simultaneously learn local and global spatial information. Furthermore, we propose the state sharing technique to enhance spatial details and integrate spatial and spectral information. Finally, the overall network is a multi-scale structure based on vision Mamba, called LE-Mamba. Extensive experiments show the proposed methods achieve state-of-the-art results on multispectral pansharpening and multispectral and hyperspectral image fusion datasets, and demonstrate the effectiveness of the proposed approach. Codes can be accessed at \url{https://github.com/294coder/Efficient-MIF}.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Reversible Column Networks. In ICLR.
  2. Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images. Inf. Fus. 104 (2024), 102158.
  3. Simple baselines for image restoration. In ECCV. Springer, 17–33.
  4. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Trans. Geosci. Remote Sens. 59, 8 (2020), 6995–7010.
  5. Machine Learning in Pansharpening: A benchmark, from shallow to deep networks. IEEE Geosci. Remote Sens. Mag. 10, 3 (2022), 279–315.
  6. PSRT: Pyramid shuffle-and-reshuffle transformer for multispectral and hyperspectral image fusion. IEEE Trans. Geosci. Remote Sens. 61 (2023), 1–15.
  7. Renwei Dian and Shutao Li. 2019. Hyperspectral Image Super-Resolution via Subspace-Based Low Tensor Multi-Rank Regularization. IEEE Trans. Image Process. 28, 10 (2019), 5135–5146.
  8. Learning a Low Tensor-Train Rank Representation for Hyperspectral Image Super-Resolution. IEEE Trans. Neural Netw. Learn. Syst. 30, 9 (2019), 2672–2683.
  9. Model-guided deep hyperspectral image super-resolution. IEEE Trans. Image Process. 30 (2021), 5754–5768.
  10. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.
  11. MIMO-SST: Multi-Input Multi-Output Spatial-Spectral Transformer for Hyperspectral and Multispectral Image Fusion. IEEE Trans. Geosci. Remote Sens. 62 (2024), 1–20.
  12. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023).
  13. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 (2021).
  14. Flatten transformer: Vision transformer using focused linear attention. In CVPR. 5961–5971.
  15. Pansharpening via detail injection based convolutional neural networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 12, 4 (2019), 1188–1204.
  16. Pan-Mamba: Effective pan-sharpening with State Space Model. arXiv preprint arXiv:2402.12192 (2024).
  17. Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geoscience and Remote Sensing Letters 19 (2022), 1–5.
  18. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks. IEEE Trans. Neural Netw. Learn. Syst. 33, 12 (2021), 7251–7265.
  19. Deep hyperspectral image fusion network with iterative spatio-spectral regularization. IEEE Transactions on Computational Imaging 8 (2022), 201–214.
  20. LAGConv: Local-Context Adaptive Convolution Kernels with Global Harmonic Bias for Pansharpening. AAAI 36, 1 (Jun. 2022), 1113–1121.
  21. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Transactions on Image Processing 27, 8 (2018), 4118–4130.
  22. Swin-umamba: Mamba-based unet with imagenet-based pretraining. arXiv preprint arXiv:2402.03302 (2024).
  23. Remote sensing image fusion based on two-stream fusion network. Inf. Fus. 55 (2020), 1–15.
  24. VMamba: Visual State Space Model. arXiv preprint arXiv:2401.10166 (2024).
  25. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV. 10012–10022.
  26. Haze correction for contrast-based multispectral pansharpening. IEEE Geosci. Remote Sens. Lett. 14, 12 (2017), 2255–2259.
  27. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
  28. Structured State Space Models for In-Context Reinforcement Learning. ArXiv abs/2303.03982 (2023).
  29. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 (2024).
  30. Learning a 3D-CNN and transformer prior for hyperspectral image super-resolution. Inf. Fus. 100 (2023), 101907.
  31. PanDiff: A Novel Pansharpening Method Based on Denoising Diffusion Probabilistic Model. IEEE Trans. Geosci. Remote Sens. 61 (2023), 1–17.
  32. S4nd: Modeling images and videos as multidimensional signals with state spaces. Neurips 35 (2022), 2846–2861.
  33. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 234–241.
  34. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 234–241.
  35. Jiacheng Ruan and Suncheng Xiang. 2024. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402.02491 (2024).
  36. Dual Spatial-spectral Pyramid Network with Transformer for Hyperspectral Image Fusion. IEEE Trans. Geosci. Remote Sens. (2023).
  37. Interpretable Model-Driven Deep Network for Hyperspectral, Multispectral, and Panchromatic Image Fusion. IEEE Trans. Neural Netw. Learn. Syst. (2023), 1–14.
  38. Gemine Vivone. 2023. Multispectral and hyperspectral image fusion in remote sensing: A survey. Information Fusion 89 (2023), 405–417.
  39. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Trans. Image Process. 27, 7 (2018), 3418–3431.
  40. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV. 568–578.
  41. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
  42. Dynamic cross feature fusion for remote sensing pansharpening. In ICCV. 14687–14696.
  43. LRTCFPan: Low-Rank Tensor Completion Based Framework for Pansharpening. IEEE Transactions on Image Processing 32 (2023), 1640–1655. https://doi.org/10.1109/TIP.2023.3247165
  44. An Iterative Regularization Method Based on Tensor Subspace Representation for Hyperspectral Image Super-Resolution. IEEE Trans. Geosci. Remote Sens. 60 (2022), 1–16.
  45. PanNet: A Deep Network Architecture for Pan-Sharpening. In ICCV. 1753–1761.
  46. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10819–10829.
  47. SSR-NET: Spatial–spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Trans. Geosci. Remote Sens. 59, 7 (2020), 5953–5965.
  48. Effective Pan-Sharpening With Transformer and Invertible Neural Network. IEEE Trans. Geosci. Remote Sens. 60 (2022), 1–15.
  49. Pan-sharpening with Customized Transformer and Invertible Neural Network. In AAAI.
  50. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv preprint arXiv:2401.09417 (2024).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Xiao Wu (55 papers)
  2. Yu Zhong (27 papers)
  3. Liang-jian Deng (32 papers)
  4. ZiHan Cao (10 papers)
Citations (6)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets