Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model (2208.03987v4)

Abstract: Large-scale vision foundation models have made significant progress in visual tasks on natural images, with vision transformers being the primary choice due to their good scalability and representation ability. However, large-scale models in remote sensing (RS) have not yet been sufficiently explored. In this paper, we resort to plain vision transformers with about 100 million parameters and make the first attempt to propose large vision models tailored to RS tasks and investigate how such large models perform. To handle the large sizes and objects of arbitrary orientations in RS images, we propose a new rotated varied-size window attention to replace the original full attention in transformers, which can significantly reduce the computational cost and memory footprint while learning better object representation by extracting rich context from the generated diverse windows. Experiments on detection tasks show the superiority of our model over all state-of-the-art models, achieving 81.24% mAP on the DOTA-V1.0 dataset. The results of our models on downstream classification and segmentation tasks also show competitive performance compared to existing advanced methods. Further experiments show the advantages of our models in terms of computational complexity and data efficiency in transferring.

• The paper adapts plain Vision Transformers for remote sensing, introducing Rotated Varied-Size Window Attention (RVSA) to efficiently process the diverse and arbitrarily oriented objects in RS imagery.
• The proposed models achieve state-of-the-art performance across RS tasks like object detection (81.24% mAP on DOTA), classification, and segmentation, while offering improved computational efficiency.
• Utilizing Masked Image Modeling (MIM) for pretraining on the large MillionAID dataset helps overcome labeled data scarcity and significantly advances the development of RS foundation models.

Overview of Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model

The paper "Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model" explores the application of large-scale vision transformers in remote sensing (RS) tasks. Vision transformers (ViTs), known for their scalability and robust representation capabilities, have been predominantly used in natural image processing. However, their potential in RS applications has not been sufficiently explored. This paper aims to bridge this gap by developing a plain vision transformer model with approximately 100 million parameters, specifically tailored for RS tasks such as object detection, scene classification, and semantic segmentation.

Key Contributions

1. Rotated Varied-Size Window Attention (RVSA): The paper introduces RVSA, an innovative approach for adapting window attention mechanisms to RS datasets. Traditional window-based attention limits context extraction by maintaining fixed-size, non-rotated windows. RVSA dynamically adjusts window size, shape, and orientation to better capture the diverse and arbitrarily oriented objects found in RS imagery. This adaptation significantly reduces computational costs and enhances the model’s ability to process high-resolution RS images effectively.
2. Model Performance and Computational Efficiency: The models developed in this paper surpass several state-of-the-art benchmarks. For instance, the proposed models achieve a mean average precision (mAP) of 81.24% on the DOTA-V1.0 dataset, indicative of their efficacy in object detection tasks. Additionally, they demonstrate enhanced performance in classification and segmentation tasks, endorsing their broad applicability and efficiency in varied RS contexts. The RVSA-equipped models also offer advantages in terms of lower memory consumption and faster training processes, positioning them as computationally efficient alternatives to existing methodologies.
3. Pretraining Strategy: To accommodate the nature of RS imagery where labeled data can be scarce, the paper employs Masked Image Modeling (MIM) for pretraining. Utilizing the MillionAID dataset—a comprehensive RS dataset—the models undergo unsupervised pretraining, thereby ameliorating the limitations posed by labeled data scarcity in RS contexts.

Implications and Future Directions

The implications of this research are multifaceted:

• RS Foundation Models: The proposed advancements significantly contribute to the development of RS foundation models. By equipping vision transformers with mechanisms tailored to RS characteristics, this research facilitates more effective deployment of AI systems in RS applications. This can potentially lead to improved Earth observation capabilities, enhancing the precision in land cover classification, maritime monitoring, and beyond.
• Scalability and Adaptability: The scalability and adaptability of the proposed RVSA model make it suitable for a range of RS tasks, from object detection to semantic segmentation. This flexibility is crucial for addressing diverse RS challenges, including processing large-scale imagery with variable object orientations.
• Theoretical Advancement: The introduction of adaptive and rotation-aware window attention mechanisms represents a theoretical advancement in the integration of vision transformers within specialized domains like RS. It opens avenues for further exploration of transformer applications in other domains with similar characteristics or requirements.

Future research could explore further optimization of these models, particularly in terms of scaling to even larger datasets or adapting them to other specialized domains. Additionally, investigations into the combination of foundation model strategies and domain-specific adaptations could yield further insights into the effective deployment of AI models in resource-constrained scenarios.

In conclusion, this paper presents substantial advancements in the adaptation of vision transformers to remote sensing tasks, proposing novel methodologies to enhance performance, computational efficiency, and applicability in RS domains.