CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening (2404.01121v1)
Abstract: Pansharpening aims to enhance remote sensing image (RSI) quality by merging high-resolution panchromatic (PAN) with multispectral (MS) images. However, prior techniques struggled to optimally fuse PAN and MS images for enhanced spatial and spectral information, due to a lack of a systematic framework capable of effectively coordinating their individual strengths. In response, we present the Cross Modulation Transformer (CMT), a pioneering method that modifies the attention mechanism. This approach utilizes a robust modulation technique from signal processing, integrating it into the attention mechanism's calculations. It dynamically tunes the weights of the carrier's value (V) matrix according to the modulator's features, thus resolving historical challenges and achieving a seamless integration of spatial and spectral attributes. Furthermore, considering that RSI exhibits large-scale features and edge details along with local textures, we crafted a hybrid loss function that combines Fourier and wavelet transforms to effectively capture these characteristics, thereby enhancing both spatial and spectral accuracy in pansharpening. Extensive experiments demonstrate our framework's superior performance over existing state-of-the-art methods. The code will be publicly available to encourage further research.
- Wele Gedara Chaminda Bandara and Vishal M Patel. Hypertransformer: A textural and spectral feature fusion transformer for pansharpening. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1767–1777, 2022.
- Joseph W Boardman. Automating spectral unmixing of aviris data using convex geometry concepts. 1993.
- Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17502–17511, 2022.
- Spanconv: A new convolution via spanning kernel space for lightweight pansharpening. International Joint Conference on Artificial Intelligence, pages 841–847, 2022.
- Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, pages 6995–7010, 2020.
- Machine learning in pansharpening: A benchmark, from shallow to deep networks. IEEE Geoscience and Remote Sensing Magazine, pages 279–315, 2022.
- Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, pages 2672–2683, 2019.
- Ralph VL Hartley. Transmission of information 1. Bell System technical journal, pages 535–563, 1928.
- Pansharpening via detail injection based convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 1188–1204, 2019.
- Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geoscience and Remote Sensing Letters, pages 1–5, 2022.
- Bam: Bilateral activation mechanism for image fusion. Proceedings of the 29th ACM international conference on multimedia, pages 4315–4323, 2021.
- Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. AAAI Conference on Artificial Intelligence, 2022.
- An efficient method for detection of copy-move forgery using discrete wavelet transform. International Journal on Computer Science and Engineering, page 2010, 1801.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Pansharpening by convolutional neural networks. Remote Sensing, page 594, 2016.
- Vision transformer for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, pages 1–11, 2022.
- Raymond Edward Alan Christopher Paley and Norbert Wiener. Fourier transforms in the complex domain. American Mathematical Soc., 1934.
- Hans Roder. Amplitude, phase, and frequency modulation. Proceedings of the Institute of Radio Engineers, pages 2145–2176, 1931.
- Interpretable model-driven deep network for hyperspectral, multispectral, and panchromatic image fusion. IEEE Transactions on Neural Networks and Learning Systems, 2023.
- Attention is all you need. Advances in neural information processing systems, 2017.
- A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, pages 2565–2586, 2015.
- Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geoscience and Remote Sensing Letters, pages 930–934, 2014.
- Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric engineering and remote sensing, pages 691–699, 1997.
- Dynamic cross feature fusion for remote sensing pansharpening. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14687–14696, October 2021.
- Memory-augmented model-driven network for pansharpening. European Conference on Computer Vision, pages 306–322, 2022.
- Pannet: A deep network architecture for pan-sharpening. In Proceedings of the IEEE international conference on computer vision, pages 5449–5457, 2017.
- P2sharpen: A progressive pansharpening network with deep spectral transformation. Information Fusion, pages 103–122, 2023.
- A triple-double convolutional neural network for panchromatic sharpening. IEEE Transactions on Neural Networks and Learning Systems, 2022.
- Panformer: A transformer based model for pan-sharpening. IEEE International Conference on Multimedia and Expo, pages 1–6, 2022.