HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening (2203.02503v3)

Abstract: Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyperspectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. In this paper, we present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively. HyperTransformer consists of three main modules, namely two separate feature extractors for PAN and HSI, a multi-head feature soft attention module, and a spatial-spectral feature fusion module. Such a network improves both spatial and spectral quality measures of the pansharpened HSI by learning cross-feature space dependencies and long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be utilized across multiple spatial scales at the backbone for obtaining improved performance. Extensive experiments conducted on three widely used datasets demonstrate that HyperTransformer achieves significant improvement over the state-of-the-art methods on both spatial and spectral quality measures. Implementation code and pre-trained weights can be accessed at https://github.com/wgcban/HyperTransformer.

• The paper introduces HyperTransformer, a transformer-based architecture that fuses low-resolution hyperspectral and high-resolution panchromatic images to reduce spatial and spectral distortions.
• It employs multi-scale feature fusion and novel loss functions to capture long-range dependencies, enhancing both textural and spectral details.
• Empirical evaluations on datasets like Pavia Center show that HyperTransformer outperforms state-of-the-art methods in pansharpening accuracy and image quality.

HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening

The paper "HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening" addresses the problem of pansharpening by proposing an innovative transformer-based approach. The task of pansharpening involves fusing a high-resolution panchromatic (PAN) image with a low-resolution hyperspectral image (LR-HSI) to generate a high-resolution hyperspectral image. Traditional methods often suffer from spatial and spectral distortions due to inadequacies in feature fusion processes and a reliance on traditional convolutional neural networks (ConvNets). This paper introduces the HyperTransformer, a novel architecture employing attention mechanisms to overcome these limitations.

Technical Contributions

1. HyperTransformer Architecture: The proposed method includes a novel transformer-based framework, which integrates features from LR-HSI and PAN images using queries, keys, and values within the transformer architecture. The HyperTransformer comprises separate feature extractors for PAN and HSI, a multi-head feature soft-attention module, and a spatial-spectral feature fusion module. This design allows the network to effectively capture and integrate long-range dependencies and cross-feature space information, thus enhancing both spatial and spectral quality in the resulting pansharpened images.
2. Multi-Scale Feature Fusion: Unlike traditional techniques, which operate at a single spatial scale, HyperTransformer applies its fusion strategy across multiple spatial scales. This enables the network to capture multi-scale long-range details and dependencies, further improving performance metrics.
3. Novel Loss Functions: The paper introduces two new loss functions—synthesized perceptual loss and transfer perceptual loss—in addition to the standard L1 loss. These losses are designed to enhance the ability of the HyperTransformer to learn nuanced features from both PAN and LR-HSI datasets, leading to higher-quality pansharpened outputs.

Empirical Evaluation

The effectiveness of the HyperTransformer is substantiated through experiments on three publicly available hyperspectral datasets: Pavia Center, Botswana, and Chikusei. The results consistently demonstrate that the proposed method outperforms existing state-of-the-art (SOTA) methods. Noteworthy improvements are reported in standard evaluation metrics such as CC, SAM, RMSE, ERGAS, and PSNR.

Quantitatively, the HyperTransformer shows a significant reduction in spatial and spectral distortions when compared to both classical methods (such as PCA, GFPCA) and ConvNet-based approaches (such as HyperPNN, PanNet, and DHP-DARN). Qualitatively, it demonstrates a marked reduction in mean absolute error across the spectral bands, particularly evident in challenging spectral regions.

Broader Implications and Future Directions

The introduction of the HyperTransformer architecture has significant implications for remote sensing and image enhancement domains. By utilizing attention mechanisms adeptly, it becomes possible to achieve higher accuracy and effectiveness in generating pansharpened hyperspectral images. This has potential applications in remote sensing tasks such as object recognition, change detection, and scene interpretation.

Future development could focus on addressing limitations observed in the ultraviolet (UV) and infrared (IR) bands due to insufficient spectral features in PAN images. Additionally, exploring the applicability of this architecture to other image fusion tasks, such as thermal-visible or MRI image fusion, may yield promising results in diverse domains beyond hyperspectral imaging.

In summary, the HyperTransformer represents a significant advancement in pansharpening methodologies by leveraging transformer architectures and novel loss functions, offering substantial improvements over SOTA methods in both theoretical framework and empirical results.