RemoteDet-Mamba for UAV Object Detection

• RemoteDet-Mamba is a hybrid deep neural architecture that integrates RGB and TIR data using parallel CSPDarkNet-53 encoders and a state-space fusion module.
• It employs a quad-directional selective scanning method to enhance small-object localization and inter-class discrimination in challenging scenes.
• Experimental results demonstrate improved mAP and 24 FPS performance on UAV platforms, highlighting its efficient and robust design.

RemoteDet-Mamba is a hybrid deep neural architecture developed for multi-modal object detection in challenging remote sensing images, with particular effectiveness for unmanned aerial vehicle (UAV) platforms. It leverages a combination of convolutional neural networks (CNN) for strong local feature extraction and a Mamba-based state-space module for efficient, global, patch-level cross-modal fusion. The architecture is characterized by a quad-directional selective scanning strategy that enhances small-object detection and inter-class discrimination while maintaining high processing speed and parameter efficiency (Ren et al., 2024).

1. Architecture and Data Flow

RemoteDet-Mamba integrates a Siamese CNN backbone with a cross-modal fusion Mamba (CFM) module. The two input modalities—most commonly visible (RGB) and thermal-infrared (TIR) imagery—are independently processed through parallel CSPDarkNet-53 CNN encoders. These extract multi-scale, local modality-specific feature maps, denoted $F_i^1, F_i^2$ at selected network layers ($i=2, 3, 5$). Each feature stream captures spatial gradients and textural cues crucial for small-object localization and background discrimination.

The fusion stage, realized via CFM, begins by normalizing and linearly projecting each feature map. Depthwise convolutions coupled with SiLU activation enhance intra-channel interactions. Each resulting feature map is then flattened in four orthogonal scan patterns (north, south, east, west), and per-scan fused across modalities by elementwise addition. These four 1D sequences are processed by lightweight state-space (S6) modules optimized for linear-time context propagation. After unflattening to 2D and summing, a global fused feature map $Y^{\mathrm{FUS}}$ is formed, which is residualized with the original CNN features before final linear transformations. The fused features are reintegrated into a YOLO-style detection head for bounding box regression and classification.

2. Quad-Directional Selective Scanning Fusion

The core innovation in RemoteDet-Mamba’s CFM is quad-directional selective scanning. Denote a normalized feature map as $f^1, f^2 \in \mathbb{R}^{H \times W \times C}$. Each scan direction ($$\text{flatten}_N, \text{flatten}_S, \text{flatten}_E, \text{flatten}_W$$) transforms this to a 1D sequence of length $HW$. For direction $d$:

$$f^1_d = \text{flatten}_d(f^1), \qquad f^2_d = \text{flatten}_d(f^2)$$

$f^{\mathrm{FUS}}_d = f^1_d + f^2_d$

$y_d = \mathrm{S6}_d(f^{\mathrm{FUS}}_d)$

$i=2, 3, 5$0

All directions are weighted equally ($i=2, 3, 5$1), and the summing merges context from all 2D axes. This approach is computationally efficient ($i=2, 3, 5$2 per block vs. quadratic for full self-attention) and propagates information across the entire spatial field in both local and non-local regimes. It also enables each category-relevant patch to receive context from all surrounding pixels—in both its own modality and the auxiliary, enhancing robustness against occlusions and background clutter.

3. Local Versus Global Feature Integration

RemoteDet-Mamba’s CNN backbone, augmented by stacked convolutional, C3, and SPPF blocks, separates local feature learning from global context fusion. Convolutions capture small receptive-field patterns, edges, and textures, critical for detection in high-density, small-object settings. The CFM module then addresses the limitations of locality by globally fusing patch-level information.

The 1D scanning and S6 processing allow dependency modeling over all spatial positions with linear computational complexity. Cross-modal elementwise addition prior to S6 ensures that both modalities contribute to the learned global representation, leading to improved recognition of objects with variable thermal or visible signatures.

4. Serial Mamba Processing: Algorithmic Mechanism

In each CFM invocation (per detection scale):

1. Each feature map undergoes layer normalization and linear projection.
2. Depthwise convolution and SiLU non-linearity yield intermediate features.
3. Each direction $i=2, 3, 5$3 is individually flattened, added across modalities, and processed by a direction-specific small state-space (S6) block.
4. Post-S6 outputs are unflattened, summed, and added residually to the original inputs.
5. The resulting maps are normalized and further transformed via feed-forward networks.
6. Fused features are reinjected into the detection pipeline.

Let $i=2, 3, 5$4 be the number of patches. Each S6 block operates in $i=2, 3, 5$5, and across four axes retains overall linear scaling. Empirical benchmarks show 24 frames per second on a single NVIDIA RTX 3090 (Ren et al., 2024).

5. Supervised Training and Loss Composition

Training employs classic YOLO-style detection objectives:

$i=2, 3, 5$6

• $i=2, 3, 5$7: Complete-IoU (CIoU) for bounding box localization.
• $i=2, 3, 5$8: Smooth Binary Cross-Entropy (Smooth BCE), incorporating an additional CIoU term for edge consistency.
• $i=2, 3, 5$9: Smooth BCE for class probabilities.

The optimizer is stochastic gradient descent (SGD), with initial learning rate $Y^{\mathrm{FUS}}$0, decay to $Y^{\mathrm{FUS}}$1, and standard data augmentations (color jitter, mosaic construction, horizontal flipping).

6. Experimental Evaluation

Evaluation on the DroneVehicle dataset (28,439 aligned RGB-TIR pairs, 953,087 vehicle instances, five categories: car, truck, bus, van, freight car) demonstrates the following outcomes at IoU=0.5:

Configuration	mAP$Y^{\mathrm{FUS}}$2	mAP$Y^{\mathrm{FUS}}$3	Model Size (M)	FPS
RGB only	75.7%	—	—	—
TIR only	79.4%	—	—	—
Simple addition fusion	80.8%	—	—	—
Bidirectional scanning	81.1%	—	—	—
RemoteDet-Mamba (quad-dir)	81.8%	58.9%	71.3	24
DMM (Mamba-based, SOTA)	79.4%	—	88	—
C²Former (Transformer-based)	74.2%	—	118	—
UA-CMDet	64.0%	—	234	9.1

RemoteDet-Mamba surpasses both unimodal and other multimodal baselines, providing a +2.4% improvement over pure TIR and +1.2% relative to the previous best (DMM) with fewer parameters and higher throughput (Ren et al., 2024).

7. Limitations and Future Extensions

The current implementation is validated only on the RGB-TIR DroneVehicle dataset. Cross-dataset generalization capabilities are untested. Modality misalignments are handled through enlarged bounding boxes; more sophisticated registration could further improve cross-modal synergy.

Potential extensions include:

• Incorporation of LiDAR or SAR data within the same SS2D framework.
• Learning adaptive fusion weights per scan direction.
• Integrating cross-modal gating or attention mechanisms within S6 blocks to mitigate false positives.
• Application of the hybrid design to other remote-sensing tasks (semantic segmentation, change detection).

A plausible implication is that similar hybrid CNN–Mamba fusions, with linear-time state-space modules, could scale to larger multi-modal systems or additional sensing tasks where global context and computational efficiency are required.

---

RemoteDet-Mamba demonstrates the viability of combining high-fidelity local encoding with computationally efficient, global context fusion using a Mamba-based state-space model, providing state-of-the-art small-object detection in remote sensing with strong parameter and resource efficiency (Ren et al., 2024).