Multimodal Fusion Transformer for Remote Sensing Image Classification (2203.16952v2)

Abstract: Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a {distinctive representation} in a reduced and hierarchical feature space. Extensive experiments are carried out on {widely used benchmark} datasets {i.e.,} the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at \url{https://github.com/AnkurDeria/MFT}.}

Overview of Multimodal Fusion Transformer for Remote Sensing Image Classification

The research paper presents a sophisticated approach for remote sensing image classification by leveraging the capabilities of the Multimodal Fusion Transformer (MFT) network. This paper targets the integration of multimodal data, such as hyperspectral imaging (HSI), multispectral imaging (MSI), synthetic-aperture radar (SAR), digital surface models (DSM), and light detection and ranging (LiDAR), to exploit their complementary information. The proposed architecture extends the application of transformers in the domain of remote sensing, traditionally dominated by convolutional neural networks (CNNs).

The authors introduce a novel transformer-based learning model that incorporates a multihead cross patch attention (mCrossPA) mechanism, enabling effective data fusion at the feature level. The proposed framework overcomes the limitations of conventional feature fusion techniques, which are often computationally intensive and less efficient for handling large-scale remote sensing datasets with varying data modalities.

Key Components of the MFT Network

1. Multimodal Data Utilization: The network exploits the diverse spectral and spatial features captured by different remote sensing modalities. LiDAR data are specifically utilized as an external classification (CLS) token, enabling improved generalization and performance.
2. Transformer Encoder with mCrossPA: Central to the MFT network is the transformer encoder module, which leverages the mCrossPA to fuse the CLS token with HSI patch tokens. This mechanism enhances the model's ability to characterize long-range dependencies and interactions between data modalities.
3. Tokenization: An innovative tokenization strategy is employed to generate both CLS and HSI patch tokens, which facilitates the learning of distinctive representations in a reduced hierarchical feature space.
4. Fusion of Complementary Information: By embedding information from external sources as tokens, the MFT model successfully merges multimodal datasets, enhancing the classification accuracy without a significant increase in computational overhead.

Experimental Evaluation and Implications

The experiments conducted on various benchmark datasets—namely, University of Houston, Trento, University of Southern Mississippi Gulfpark, and Augsburg—demonstrate the effectiveness of the proposed model. The MFT network consistently outperformed conventional methods like KNN, SVM, classical CNN models, and even other transformer-based models such as Vision Transformers (ViTs) and SpectralFormers. The results reveal substantial improvements in terms of overall accuracy (OA), average accuracy (AA), and kappa coefficients across all datasets evaluated.

Implications of the Research:

• Enhanced Model Generalization: The integration of complementary information from multimodal datasets via the CLS token significantly contributes to the generalization capabilities of the model.
• Scalability and Robustness: The ability of the MFT model to adapt to various types of remote sensing data enhances its utility across diverse applications, such as environmental monitoring, urban planning, and disaster management.
• Efficient Data Processing: By minimizing the need for extensive parameter tuning and computational resources associated with traditional CNNs, transformers provide a scalable solution for processing large-scale remote sensing data.

Future Prospects in Artificial Intelligence

This research sets the stage for future endeavors in artificial intelligence by illustrating the potential of transformer-based architectures in multisource data fusion in remote sensing. Moving forward, there are several directions worth exploring:

• Hyperparameter Optimization: Investigation into automated hyperparameter tuning mechanisms could further enhance the performance and adaptability of the MFT model across different datasets and tasks.
• Integration with Emerging Data Types: Extending the architecture to accommodate emerging data modalities and addressing challenges related to variable data quality and scale are promising avenues.
• Real-time Processing: The development of real-time processing capabilities using transformer networks could facilitate the timely extraction of actionable insights in critical applications such as emergency response and environmental monitoring.

In conclusion, the introduction of the Multimodal Fusion Transformer network represents a significant advancement in remote sensing image classification, effectively bridging the gap between diverse data sources and providing a robust framework for future research and application in geoscience and remote sensing.