- The paper presents a comprehensive survey evaluating over 300 remote sensing SSMs and their adaptive scanning strategies.
- The analysis showcases hybrid architectures that fuse SSMs with CNNs, Transformers, and GNNs, enhancing tasks like classification and segmentation.
- The study identifies challenges and future directions, including foundation model scaling, efficient edge deployment, and optimal feature fusion.
Comprehensive Survey of State Space Models in Remote Sensing
Introduction
State Space Models (SSMs) have recently gained prominence as a core architecture for sequence modeling, characterized by linear computational complexity and the capacity for robust long-range dependency extraction. While initially introduced for language and general vision tasks, the unique demands of remote sensing—dense predictions, multi-modal imaging, extensive temporal sequences—have catalyzed rapid adoption and innovation of SSMs in this domain. This survey systematically evaluates over 300 remote sensing SSMs, delineating task-specific adaptations, architectural advancements, and emerging challenges, with the aim of providing technical guidance for researchers seeking to optimize SSM design for remote sensing applications (2606.25329).
Evolution of SSMs for Remote Sensing
The foundational SSM structure, rooted in control theory, leverages state and output equations for sequential information integration. The S4 model marked SSM's entry into deep learning, outperforming self-attention on long sequences by exploiting memory mechanisms and optimal polynomial projections. The emergence of Mamba (S6) enhanced SSMs with selective state propagation and hardware-aware algorithms, delivering superior inference efficiency and competitive accuracy in language modeling. Vision-Mamba architectures subsequently replaced self-attention blocks in Vision Transformers with SSMs, pioneering two-dimensional and multi-path scanning strategies tailored for visual data, including in remote sensing.
SSMs Across Remote Sensing Tasks
Image Classification
Innovations in remote sensing image classification pivot around two axes. First, model scanning strategies (e.g., multi-directional, bidirectional, cross, hierarchical) are designed to mitigate causal bias, enhancing global context and small target separation. Second, hybrid designs integrate SSMs with CNNs, Transformers, and GNNs for spectral-spatial fusion. RSMamba and HSIMamba exemplify directional scanning and hybrid architectural approaches, respectively, consistently outperforming classical CNN, RNN, ViT, and Transformer models on datasets such as Houston2013 and Indian, achieving OA >99.5% and minimal performance degradation across imbalanced classes.
Semantic Segmentation
Dense prediction tasks in remote sensing benefit from dual strategies: SSMs fused with CNNs for global-local encoding and U-Net/-derived architectures with SSMs as skip connections or central blocks. RS3Mamba utilizes residual networks for local features and Mamba blocks for global dependencies, whereas CM-UNet and UNetMamba leverage multiscale convolution combined with SSM gating. SSM-based segmentation models, notably UNetMamba, attain MIoU scores exceeding classical and Transformer-based baselines on LoveDA, validating multi-scale enhancement.
Object Detection
SSMs in object detection exhibit a balance between long-sequence modeling for dense or small targets and multimodal fusion for data from disparate sensors. Pyramid SSM models (HTD-Mamba) downsample token sequences for hierarchical feature integration, while fusion-centric approaches (DMM, MGMF) deploy selective gating to suppress cross-modal redundancy. Empirical results indicate superior localization and robustness relative to Transformer-based detectors under high spatial variability.
Change Detection
SSMs in change detection utilize sequential, cross, and parallel token arrangements to capture spatiotemporal relationships. ChangeMamba demonstrates flexible token ordering and fusion methodologies, outperforming FC-EF, Swin-Unet, and ChangeFormer baselines on WHU, LEVIR-CD, and LEVIR-CD+, with F1 scores >95%. Subsequent works focus on multi-temporal fusion and adaptive guidance strategies, optimizing differential feature representation.
Super Resolution
SSM-based super-resolution methods integrate frequency-domain transforms (Fourier, wavelet) within SSM blocks for cross-domain feature extraction. FMSR and IRSRMamba models achieve substantial gains in reconstruction quality by coupling SSM's long-range modeling with local CNN refinement, although optimal fusion between frequency and spatial domains remains a challenge.
Denoising and Dehazing
Bidirectional and multi-path SSM scanning combined with spectral attention gating (HSIDMamba) and U-shaped designs (RSDeHamba) facilitate enhanced denoising and dehazing on hyperspectral and satellite imagery. The data-model interplay leverages global context for noise suppression, but adoption is limited by limited empirical validation and architectural maturity.
Pan-Sharpening
Cross-modal SSMs, exemplified by Pan-Mamba, deploy channel exchange and gating for efficient multispectral and panchromatic feature fusion, achieving high-fidelity pan-sharpened output. These designs depend critically on modality alignment and can require extensive tuning for generalization to other data sources.
Architectural Innovations in SSMs
Scanning Strategies
A taxonomy of over 20 scanning strategies (bidirectional, cross, continuous 2D, hierarchical, spatial-spectral, spatiotemporal) has been developed to address multimodal, temporal, and spatial heterogeneity of remote sensing data. Multi-path and adaptive sequences reduce directional bias, improving small target separability and balancing coverage, but introduce additional complexity and risk of redundancy.
High-Level Frameworks
Backbone-centric methods use SSMs as primary feature extractors with geometric-aware token alignment and dynamic weighting, while U-Net-based frameworks integrate SSM blocks for multi-scale fusion. The trade-off involves hierarchical feature efficiency versus native multiscale fusion complexity. Pruning, distillation, and quantization are essential for deployment on edge devices.
Hybrid Architectures
Integration paradigms combine SSMs in parallel/nested arrangements with CNNs, Transformers, and GNNs. CNNs enhance local perception, Transformers add global attention, GNNs allow relational abstraction for spatially structured data. The fusion unlocks cross-scale and multimodal synergy but challenges original SSM efficiency.
Component Refinement
Structural modification (deletion, addition, modification, complementation) at the block level optimizes computational cost and information flow. Adaptive residual branches and input supplementation strengthen feature extraction, with improper design risking performance collapse.
Multi-Modal Feature Fusion
Feature-level fusion, supported by selective gating and guided redundancy removal, achieves cross-temporal, spatial, and spectral alignment. Pixel-level and decision-level fusion in SSMs remain underexplored. Gated mechanisms (self-guidance, cross-guidance) filter noise and selective signal, critical for dense and multi-modal remote sensing tasks.
Frequency Domain Processing
Embedding frequency-domain transforms in SSM blocks facilitates noise suppression and structural feature modeling. Adaptive frequency attention (MambaFormerSR, IRSRMamba) generalizes across spectral and spatial domains, but the optimal fusion mechanism is unresolved.
Challenges and Future Directions
Remote Sensing Foundation Models
SSM-based foundation models are projected to overcome Transformer inefficiency for dense and high-resolution predictions, particularly by leveraging linear complexity and selective modeling. Key obstacles include scaling to global datasets, mitigating long sequence forgetting, and developing SSM-specific fine-tuning protocols (adapters, LoRA, incremental parameterization).
Edge Deployment and Accelerated Processing
Hardware-aware SSM acceleration and model lightweight approaches (distillation, pruning, quantization) are essential for large-scale and edge deployments. Tailored operator design and efficient data transmission algorithms (CPU-GPU interplay) optimize remote sensing throughput.
Vision-Language Integration
SSMs' selective modeling advantage supports contrastive vision-language alignment for extensive textual descriptions and autoregressive generative architectures. Research gaps exist in modality alignment and SSM-based sequence generation for remote sensing VLMs.
Limitations
SSMs exhibit sensitivity to scale variation, geometric heterogeneity, fine-grained texture modeling, and sequence stability at extreme lengths. Hybridization with CNNs and frequency-domain processing improves local detail capture, but systematic guidelines for optimal augmentation are lacking.
Conclusion
SSMs have instigated a technical paradigm shift in remote sensing by providing efficient, scalable architectures for multimodal, temporal, and dense prediction tasks. Their proliferation across classification, segmentation, detection, change detection, super-resolution, and pan-sharpening is defined by strong numerical performance and innovative design adaptations. Key future developments include SSM-based foundation models, hardware-constrained acceleration, vision-language synergy, and systematic architectural guidance for robust deployment across diverse remote sensing applications.