- The paper introduces MoBaNet, a parameter-efficient framework combining cross-modal prompt adapters, discrepancy-guided gated fusion, and training-only modality masking to adapt frozen vision foundation models for RGB–DSM segmentation.
- The method achieves up to 84.37% mIoU on Vaihingen and 88.03% on Potsdam, while using only 6.18M trainable parameters and reaching 71.60 FPS with 403 MB memory in its lightweight configuration.
- The paper shows that modality masking substantially improves robustness: removing DSM causes only a 1.17-point mIoU drop with MCRM, compared with a 23.06-point collapse without it, demonstrating that symmetric architecture alone cannot prevent modality bias.
Overview
MoBaNet addresses two coupled problems in multimodal remote sensing semantic segmentation: the computational cost of adapting vision foundation models (VFMs) to multimodal inputs, and modality imbalance, whereby optimization converges to RGB-dominant shortcut solutions that suppress auxiliary-modality contributions. The framework is built on a largely frozen VFM backbone (SAM or DINOv2) arranged as a symmetric dual-stream encoder, with three trainable components: a Cross-modal Prompt-Injected Adapter (CPIA), a Difference-Guided Gated Fusion Module (DGFM), and a training-only Modality-Conditional Random Masking (MCRM) strategy. The full model trains with only 6.18M trainable parameters while achieving state-of-the-art results on ISPRS Vaihingen and Potsdam.
Motivation and positioning
The paper situates itself at the intersection of two research threads. First, feature-level multimodal fusion for RGB–DSM segmentation has progressed from parallel encoders toward interaction-enhanced designs, including cross-modal guidance (Gather-to-Guide), selective gated fusion (CMGFNet, CEGFNet), and Transformer-based interaction (FTransUNet, TMFNet). Second, parameter-efficient fine-tuning (PEFT) methods—adapters, LoRA, visual prompt tuning—are predominantly single-modality designs. The authors identify two bottlenecks when transferring PEFT to multimodal settings: independently configured adapters leave frozen branches weakly coupled, so deep cross-modal interaction is insufficient; and because frozen VFMs inherit a preference for optical cues, standard PEFT can still converge to optical-dominant solutions. A central claim of the paper is that architectural symmetry alone does not guarantee balanced modality contributions—an assertion supported empirically by their modality-missing ablation.
Architecture
The backbone is a pretrained ViT with shared, frozen self-attention and MLP weights across both streams. Patch embedding is asymmetric: the optical branch reuses the frozen pretrained embedding, while the auxiliary branch uses a trainable embedding to accommodate channel mismatch. Absolute positional encodings are shared and bicubically resized. The encoder is partitioned into four stages; CPIA is inserted before each stage and DGFM after it, producing multi-level fused features consumed by a UperNet decoder. Two lightweight auxiliary heads exist only during training for MCRM supervision.
CPIA enables cross-modal interaction under a frozen backbone. A Cross-modal Prompt Generator projects paired tokens to a reduced dimension, concatenates and fuses them into a shared semantic base, then applies a Task Feature Transformation—a per-modality channel-wise affine modulation—to produce modality-specific prompts. These prompts are injected as an additive bias inside bottleneck adapters (WdownX+WpromptP followed by ReLU and up-projection), establishing a symmetric semantic communication channel without disturbing the pretrained feature space.
DGFM performs post-stage fusion. Reduced features from both modalities are combined with an explicit discrepancy cue D=∣Rx−Ry∣, which is concatenated with the reduced features and passed through a lightweight gating network (1×1 conv, GroupNorm, GELU, 1×1 conv) to predict a sigmoid gating map applied channel-wise and spatially: F~=G⊙X~+(1−G)⊙Y~. The design rationale is that cross-modal disagreement directly indicates where one modality is unreliable, allowing adaptive selection rather than uniform averaging.
MCRM targets modality imbalance during training only. For a fraction r of each batch, local rectangular regions are zeroed on exactly one modality, with masked samples split nearly evenly between RGB-masked and aux-masked subsets. Auxiliary heads are supervised exclusively on hard pixels—those misclassified by the fused branch—so each modality-specific branch learns to compensate precisely where fusion fails. MCRM adds no inference cost; its auxiliary heads are discarded at test time.
Experimental results
On Vaihingen, MoBaNet (DINOv2-ViT-L) attains 91.90% OA, 91.39% mF1, and 84.37% mIoU, exceeding DC-Swin by 1.24 mIoU points and MANet (ViT-B) by 1.77 mIoU points. On Potsdam, MoBaNet (DINOv2-ViT-L) reaches 92.23% OA, 93.53% mF1, and 88.03% mIoU, improving over DC-Swin by 2.39 mIoU points and MANet by 2.95 mIoU points. Consistent gains across both SAM and DINOv2 backbones support the claim that the framework is backbone-agnostic.
The efficiency analysis is the paper's strongest quantitative result. With DINOv2-ViT-B on Potsdam, MoBaNet achieves 87.80% mIoU with 6.18M trainable parameters—the smallest count among compared methods—versus CMGFNet (43.68M), FTransUNet (203.40M), and MANet (20.42M), outperforming FTransUNet and MANet by 2.24 and 2.72 mIoU points respectively. Latency is 13.97 ms (71.60 FPS) with 403 MB memory, comparable to CMGFNet and far better than MANet (105.36 ms, 1742 MB). Lightweight optical-only models remain faster but lack DSM cues and score lower.
Ablations on Potsdam isolate each component's contribution:
| Variant |
Params (M) |
OA (%) |
mF1 (%) |
mIoU (%) |
| Base |
1.91 |
90.47 |
91.74 |
84.95 |
| + CPIA |
3.99 |
91.43 |
92.57 |
86.33 |
| + CPIA + DGFM |
5.00 |
91.73 |
93.02 |
87.15 |
| Full (+ MCRM) |
6.18 |
92.11 |
93.40 |
87.80 |
CPIA contributes the largest single gain (+1.38 mIoU), confirming that prompt-injected adapters materially improve cross-modal coupling despite the frozen backbone.
The most striking evidence concerns modality balance. Under RGB-only inference (DSM zeroed), the full MoBaNet drops only 0.75% OA / 1.17% mIoU relative to full-modality inference, whereas the same architecture trained without MCRM collapses by 19.41% OA / 23.06% mIoU. This roughly twenty-fold difference in degradation demonstrates that the unregularized dual-stream model becomes heavily dependent on DSM despite its symmetric structure—and that MCRM effectively corrects this. It also substantiates the paper's claim that architectural symmetry does not by itself yield balanced optimization.
Limitations and open questions
The paper concedes several constraints. Evaluation is restricted to RGB–DSM fusion on two urban benchmarks; generalization to multispectral, LiDAR, or SAR modalities, and to cross-city or cross-sensor transfer, remains untested. DGFM is deliberately lightweight, and the authors acknowledge its capacity for modeling complex higher-level cross-modal relationships may be limited. Challenging categories—low vegetation, trees, small objects such as cars—remain difficult, indicating unresolved fine-grained discrimination under class ambiguity. Additionally, the modality-missing evaluation tests only DSM removal at inference; robustness under more general incomplete-modality conditions is not characterized. Whether stage-wise prompt injection scales favorably to larger backbones or deeper interaction frequencies than four stages is also left open.
Conclusion
MoBaNet demonstrates that parameter-efficient adaptation of VFMs to multimodal remote sensing segmentation is viable when cross-modal interaction is engineered explicitly (via prompt-injected adapters), fusion is discrepancy-guided, and training-time masking enforces modality balance. The combination of state-of-the-art accuracy on ISPRS benchmarks with 6.18M trainable parameters and near-immunity to auxiliary-modality loss at inference constitutes a strong empirical case for the proposed design, while leaving extension to broader modality combinations and domain-shift settings as clearly stated open problems.