Prithvi-Complementary Adaptive Fusion Encoder

• The paper introduces a hybrid transformer–CNN model that uses a frozen Prithvi encoder with adapter tuning to achieve state-of-the-art flood segmentation.
• It employs complementary channel routing to split multispectral inputs, fusing global semantic features with local spatial details via CNN residuals and attention modules.
• Empirical results on Sen1Floods11 and FloodPlanet benchmarks show superior IoU performance with significant parameter efficiency and robust generalization.

Prithvi-Complementary Adaptive Fusion Encoder (CAFE) is a parameter-efficient hybrid transformer–CNN architecture designed to improve multispectral flood inundation mapping, particularly in remote sensing contexts where both global semantic context and local spatial detail are crucial. Developed to address the observed limitation that large geospatial foundation models (GFMs)—such as Prithvi and TerraMind—often fail to surpass conventional U-Net architectures on flood segmentation benchmarks, Prithvi-CAFE integrates a frozen Prithvi encoder fine-tuned via adapters with a parallel convolutional residual pathway enhanced by Convolutional Attention Modules (CAM). The result is an architecture that supports arbitrary multispectral/multimodal channel input and realizes state-of-the-art performance on the Sen1Floods11 and FloodPlanet flood mapping datasets (Kaushik et al., 5 Jan 2026).

1. Architectural Composition and Channel Routing

Prithvi-CAFE is constructed on the principle of complementary channel routing, wherein the input image stack $X \in \mathbb{R}^{B \times C \times H \times W}$ is explicitly split into two disjoint channel subsets: $X_{\text{AP}}$ for the transformer (Prithvi) branch and $X_{\text{CNN}}$ for the CNN residual branch, such that $X = [X_{\text{AP}} \mid X_{\text{CNN}}]$ and $X_{\text{AP}} \cap X_{\text{CNN}} = \emptyset$. For the Sen1Floods11 dataset, the indices $\mathcal{I}_{\text{AP}} = \{1,2,3,7,11,12\}$ correspond to the six spectral bands Prithvi was pretrained on, while the complement is processed by the CNN.

Each branch specializes: the transformer extracts spectrally rich, long-range representations; the CNN branch emphasizes fine local textures, edges, and modalities not natively supported by Prithvi. The outputs are later fused via a dedicated multi-scale, multi-level attention mechanism.

2. Adapted Prithvi Encoder and Efficient Adapter Tuning

The transformer backbone employs a frozen Prithvi encoder—pretrained on large remote-sensing corpora—as its context pathway. Rather than full model fine-tuning (which involves updating approximately 650M parameters), Prithvi-CAFE uses per-block, residual two-layer bottleneck adapters, reducing trainable parameters to 45.5M and utilizing only ~7% of the encoder's parameters for adaptation. Each adapter is a bottleneck MLP: $$f_{\text{adapter}}(x) = \sigma( W_2 \cdot \sigma(W_1 \cdot x + b_1) + b_2 )$$ with $W_1 \in \mathbb{R}^{d \times 32}$, $W_2 \in \mathbb{R}^{32 \times d}$, where $d$ is the embedding dimension.

Adapters are inserted at the start of each ViT block, yielding the block formula: $X_{\text{AP}}$0 This approach preserves pretrained knowledge, lowers GPU memory usage (50–80% reduction versus full tuning), and accelerates training. Features are extracted from layers $X_{\text{AP}}$1, mapped back to feature maps, and refined by hierarchical Feature Pyramid Networks (FPNs) with upsampling and progressive channel projection ($X_{\text{AP}}$2).

3. CNN Residual Branch and Convolutional Attention Modules

Complementing the transformer, the CNN branch processes $X_{\text{AP}}$3 through a four-stage residual hierarchy—channels $X_{\text{AP}}$4—optimized via ablation for sufficient capacity. Each residual block follows a conventional ResNet formulation with two $X_{\text{AP}}$5 convolutions, batch normalization, ReLU activations, and identity or $X_{\text{AP}}$6 projection shortcuts.

Crucially, each stage is augmented by a Convolutional Attention Module (CAM) that sequentially applies:

• Channel attention: leveraging global average and max pooling with an MLP and sigmoid activation,

$X_{\text{AP}}$7

• Spatial attention: convolution over pooled feature maps,

$X_{\text{AP}}$8

This suppresses irrelevant activations, enhances discriminative local responses, and heuristically reduces artifacts from atmospheric contamination. While precise hyperparameters for CAM (e.g., $X_{\text{AP}}$9, MLP widths) are not specified, the described ablation confirms CAM's benefit.

4. Multi-Scale, Multi-Level Feature Attention Fusion

Fusion is realized via the multi-scale, multi-level feature attention fusion (M\textsuperscript{2}FAF) module. At each pyramid level, CNN features are re-aligned (interpolated and projected by $X_{\text{CNN}}$0 conv) to match transformer feature dimensions. Features are concatenated and a learned sigmoid attention mask $X_{\text{CNN}}$1 is computed: $X_{\text{CNN}}$2 which is bias-stabilized: $X_{\text{CNN}}$3 The final fusion at each level: $X_{\text{CNN}}$4 A higher $X_{\text{CNN}}$5 biases toward transformer-semantic features, empirically shown to improve both stability and generalization.

5. Decoder, Training, and Evaluation Protocol

The decoder is a standard UPerNet, which applies a Pyramid Pooling Module (PPM) to the coarsest semantic feature, fuses via lateral connections ($X_{\text{CNN}}$6), and outputs a pixel-wise segmentation via a final conv head. The model is trained end-to-end—with adapters, CNN, CAM, fusion, and decoder learnable—using cross-entropy loss, AdamW optimizer, and StepLR scheduling.

Experiments are conducted on two primary datasets:

• Sen1Floods11: 446 Sentinel-1/Sentinel-2 image-label pairs, canonical splits.
• FloodPlanet: 298 Sentinel-2 images, labels from 3m PlanetScope, 4-fold cross-validation (70%/10%/20%).

All images and labels for FloodPlanet are resized to $X_{\text{CNN}}$7. While normalization and modality-specific preprocessing are not specified, input modalities are selected such that Prithvi receives only bands on which it was pretrained, and all others are passed to the CNN via the routing mechanism.

6. Empirical Results, Ablations, and Analysis

Prithvi-CAFE achieves state-of-the-art IoU and mIoU across both benchmarks:

Dataset	Prithvi-CAFE	Best Baseline (IoU)	Other Relevant Baselines (IoU)
Sen1Floods11 test	83.41	TerraMind: 82.90	Prithvi: 82.50; DOFA: 81.54; spectralGPT: 81.02
Hold-out site	81.37	U-Net: 70.57	Prithvi: 72.42
FloodPlanet	64.70	TerraMind: 62.33	U-Net: 60.14; DOFA: 59.15; Prithvi-600M: 61.91

In parameter-efficient tuning ablations on Sen1Floods11, Prithvi-CAFE outperforms both ViT Adapter (84.94 mIoU) and LoRA Adapter (87.57 mIoU) with 88.87 mIoU.

Ablation studies confirm:

• The combined effect of adapters, CNN residual blocks, CAM, and M\textsuperscript{2}FAF is required for optimal performance.
• Channel widths $X_{\text{CNN}}$8 provide the best CNN capacity.
• Increasing fusion bias $X_{\text{CNN}}$9 up to 0.8 improves fusion and accuracy.

Qualitative analysis and embedding visualization show that Prithvi-CAFE uniquely preserves both U-Net’s fine-grained boundary detail and the spatial-semantic coherence typical of GFMs, supporting robust performance especially in out-of-distribution generalization settings.

7. Limitations and Future Directions

Despite improved robustness, persistent failure modes include challenges with dense clouds and atmospherically corrupted pixels. The disparity between high-resolution (3m) PlanetScope labels and coarser (10m) Sentinel-2 imagery remains a source of local error, although Prithvi-CAFE outperforms prior solutions under these constraints.

A key axis for future improvement is the integration of SAR (Synthetic Aperture Radar) data into the input routing/fusion framework, exploiting the architecture’s flexibility for arbitrary channel input, and improving cloud-robustness. This direction is motivated by the model’s ability to naturally process multimodal sources via channel partitioning and adaptive fusion.

---

Prithvi-CAFE exemplifies a systematic approach to integrating pretrained transformer and CNN pathways for remote sensing segmentation, optimized for parameter-efficiency and local/global complementarity, and empirically validated through state-of-the-art results on challenging flood delineation tasks (Kaushik et al., 5 Jan 2026).