MM SAM-adapter for Multimodal Segmentation

• MM SAM-adapter is a multimodal semantic segmentation framework that leverages SAM’s RGB backbone while integrating auxiliary cues like depth, LiDAR, and thermal data.
• It employs a lightweight side branch using ConvNeXt encoders paired with bidirectional Injector-Extractor modules to selectively fuse complementary modality features.
• Empirical results on benchmarks such as DeLiVER and FMB highlight significant mIoU improvements, especially under adverse conditions like low visibility and occlusions.

MM SAM-adapter denotes a multimodal semantic segmentation framework that repurposes the Segment Anything Model (SAM) as the main RGB backbone and augments it with a lightweight multimodal side branch. Its defining premise is that RGB should remain dominant because foundation models such as SAM have learned strong and generalizable image representations, while auxiliary modalities such as depth, LiDAR, events, and thermal data should be injected selectively when they contribute complementary cues. The framework therefore discards SAM’s prompt encoder and mask decoder, retains the image encoder, and adapts the resulting representation through a multimodal fusion encoder, Injector-Extractor adapters, and a SegFormer decoder for dense semantic prediction (Curti et al., 12 Sep 2025).

1. Definition and terminological scope

In the strict sense established by "Multimodal SAM-adapter for Semantic Segmentation" (Curti et al., 12 Sep 2025), MM SAM-adapter is a multimodal semantic segmentation method rather than a generic “multi-task” or “multiple-prior” SAM variant. Its target problem is assigning a semantic class to every pixel under conditions such as poor lighting, occlusions, adverse weather, motion blur, and low visibility, where RGB-only segmentation degrades and auxiliary sensors can provide complementary information. The auxiliary observation may be depth, LiDAR, thermal/infrared, or events, depending on the benchmark.

This usage differs from several adjacent SAM-adapter lines. The original "SAM-Adapter" (Chen et al., 2023), "SAM2-Adapter" (Chen et al., 2024), and "SAM3-Adapter" (Chen et al., 24 Nov 2025) adapt SAM-family backbones to downstream tasks such as camouflaged object detection, shadow detection, and medical segmentation, but they are not multimodal in the strict sensor-modal sense. In particular, SAM3-Adapter defines task-specific input $F_i$ as a sum of guidance signals, yet its reported implementation uses visual priors only—specifically patch embeddings and high-frequency components—rather than RGB-plus-auxiliary-sensor fusion (Chen et al., 24 Nov 2025). Medical SAM adaptations such as SAM-DA and FSAM likewise address domain shift or modality-specific robustness, but they do not formulate multimodal semantic segmentation in the sense of jointly processing aligned RGB and auxiliary sensor streams (Tejero et al., 12 Jan 2025, Bui et al., 11 May 2026).

A central conceptual distinction follows from this terminology. MM SAM-adapter is not a prompt-learning extension of vanilla SAM, nor an instance-segmentation refinement of the original promptable decoder. It is a dense semantic segmentation architecture in which SAM contributes segmentation-specific RGB priors, while multimodal information is injected through an asymmetric side branch.

2. System architecture

The architecture has four major parts: a SAM image encoder as the main branch, a Multimodal Fusion Encoder as a secondary branch, an adapter that mediates information exchange between the fusion branch and SAM, and a segmentation head implemented as a SegFormer decoder (Curti et al., 12 Sep 2025). Only the SAM image encoder is reused; the prompt encoder and mask decoder are discarded because they are specific to prompt-guided instance segmentation.

The backbone is SAM ViT-Large with $L=24$ transformer layers. Following the ViT-Adapter design, the 24 layers are partitioned into $N=4$ blocks, and each block interacts with the side branch through an Injector-Extractor pair. The RGB image $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$ is sent both to the SAM image encoder and to an RGB-specific encoder in the fusion branch. The auxiliary signal $I^{X}$ is assumed pixel-aligned with RGB and is represented as $I^{X} \in \mathbb{R}^{H \times W \times 3}$; if the original auxiliary modality has one channel, it is replicated three times before entering the ConvNeXt encoder.

The multimodal side branch uses two ConvNeXt-Small encoders pretrained on ImageNet-22k, one for RGB and one for the auxiliary modality. They output four feature maps each,

$F_i^{\text{RGB}}, \quad F_i^{X}, \qquad i=1,2,3,4,$

with spatial sizes

$$\frac{H}{2^{i+1}} \times \frac{W}{2^{i+1}} \times D_i,$$

where $D_i = 96, 192, 384, 768$. These modality-specific features are fused by a Fusion Module to produce

$F_i^{\text{MM}}, \qquad i=1,2,3,4.$

The default fusion mechanism is the Road-Fusion block from RoadFormer+, which has separate global and local pathways. The global path uses transformer-style Global Feature Enhancers for each modality and a Global Feature Recalibration Module that performs cross-attention in both directions between RGB and auxiliary features. The local path uses convolutional Local Feature Enhancers and a Local Feature Fusion Module. A Feature Enhancement and Integration Module then combines global and local multimodal representations using weighted feature fusion and coordinate attention.

Channel alignment is essential because the SAM branch operates in embedding dimension $L=24$0. The fused multimodal outputs are therefore linearly projected to match SAM’s token dimension. On the SAM side, patch embedding and positional encoding produce the initial token sequence

$L=24$1

For the side branch, the initial multimodal token set is constructed from $L=24$2: each is projected to 1024 channels, flattened, and concatenated into

$L=24$3

This asymmetric organization is one of the framework’s defining principles. The main branch is a large SAM ViT-L RGB backbone, while the side branch is lighter and explicitly multimodal. The design is meant to preserve the strong generalization ability of RGB features while allowing auxiliary modalities to act as corrective signals.

3. Adapter mechanics and training regime

The adapter is implemented through two recurring modules: the Injector and the Extractor (Curti et al., 12 Sep 2025). The Injector performs deformable cross-attention from SAM to multimodal features. SAM tokens are the query, and multimodal tokens are the keys and values. Its update is

$L=24$4

where $L=24$5 is layer normalization, $L=24$6 denotes multi-scale deformable cross-attention, and $L=24$7 is a learnable $L=24$8-dimensional vector initialized to zero. The zero initialization suppresses multimodal influence at the beginning of training and stabilizes adaptation.

The Extractor performs the reverse interaction, updating the side-branch representation using the current SAM output: $L=24$9

$N=4$0

The feed-forward network includes linear layers and depth-wise convolution. Across the four SAM blocks, the architecture therefore performs bidirectional exchange: multimodal features are injected into SAM, and SAM semantics are fed back into the multimodal side branch.

Final prediction is produced by a SegFormer decoder. It uses the lowest-level fused feature $N=4$1 together with the final side-branch token set $N=4$2, which is sliced and reshaped back into spatial feature maps $N=4$3, $N=4$4, and $N=4$5. The model upsamples $N=4$6 and sums it with $N=4$7 to produce $N=4$8. Each resulting spatial feature is then enriched by summing it with the final SAM tokens $N=4$9, interpolated to the appropriate spatial resolution, yielding mixed representations $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$0, $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$1. These are decoded into the semantic prediction

$$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$2

Training uses OHEM cross-entropy loss. The optimizer is AdamW with weight decay $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$3. The learning-rate schedule is

$$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$4

with $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$5, $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$6, warm-up ratio $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$7, warm-up duration $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$8, and $$I^{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$9. Layer-wise learning-rate decay follows

$I^{X}$0

with $I^{X}$1.

A crucial implementation detail is that the image encoder is fine-tuned jointly with the adapter. This differs from many SAM-adapter methods that freeze the entire encoder. Training is run for 100 epochs on DeLiVER and MUSES and for 200 epochs on FMB, on two NVIDIA RTX 3090 GPUs with batch size 8 and gradient accumulation. Augmentations include random resize with ratio 0.5–2.0, random horizontal flipping, photometric distortion, Gaussian blur with probability 0.2, and random crop. Crop size is $I^{X}$2 for DeLiVER and MUSES, and $I^{X}$3 for FMB.

4. Benchmarks and empirical profile

MM SAM-adapter is evaluated on three benchmarks with different modality pairings and adverse-condition profiles: DeLiVER, FMB, and MUSES (Curti et al., 12 Sep 2025). DeLiVER is a synthetic urban dataset with RGB, LiDAR, depth, and event modalities at $I^{X}$4, with 25 classes. FMB contains 1,500 aligned RGB-Thermal pairs at $I^{X}$5, with 14 classes. MUSES is a real driving dataset with RGB, LiDAR, event, and radar, although radar is excluded because of excessive sparsity; it uses 19 Cityscapes-style classes.

Benchmark	Modality setting	MM SAM-adapter result
DeLiVER	RGB-Depth / RGB-LiDAR / RGB-Event	57.35 / 57.14 / 55.70 mIoU
FMB	RGB-Thermal	66.10 mIoU
MUSES	RGB-LiDAR / RGB-Event	81.07 / 79.92 mIoU

On DeLiVER, the model obtains 57.35 mIoU for RGB-Depth, 57.14 for RGB-LiDAR, and 55.70 for RGB-Event. On RGB-hard subsets, where the auxiliary modality is expected to matter most, the gains are especially pronounced: RGB-LiDAR reaches 45.46 mIoU, compared with 40.93 for GeminiFusion and 40.26 for RoadFormer+; RGB-Event reaches 44.67, compared with 41.98 for RoadFormer+ (Curti et al., 12 Sep 2025). RGB-Depth shows smaller gains, which the paper attributes to the synthetic dataset’s unusually clean and informative depth maps.

On FMB with RGB-Thermal, MM SAM-adapter achieves 66.10 mIoU overall, 68.45 on RGB-easy, and 62.59 on RGB-hard. This exceeds GeminiFusion at 64.75 overall and RoadFormer+ at 64.57 overall; on RGB-hard specifically it reaches 62.59 versus 61.79 for RoadFormer+ (Curti et al., 12 Sep 2025). On MUSES, overall performance is 81.07 for RGB-LiDAR and 79.92 for RGB-Event. For RGB-LiDAR, the model exceeds RoadFormer+ overall, by day, by night, and especially in fog, where it reaches 74.12 versus 69.64. For RGB-Event, it is best overall and in most conditions, though on the fog split it scores 68.97, slightly below RoadFormer+ at 69.29.

The paper also compares against all-modalities methods. On DeLiVER, the all-modalities CAFuser obtains 55.60 mIoU, while MM SAM-adapter reaches 57.35 with RGB+Depth, 57.14 with RGB+LiDAR, and 55.70 with RGB+Event. On MUSES, CAFuser obtains 78.18, while MM SAM-adapter reaches 81.07 with RGB+LiDAR and 79.92 with RGB+Event (Curti et al., 12 Sep 2025). This indicates that a two-modality foundation-model-centric design can outperform heavier all-sensor fusion pipelines.

A distinctive part of the empirical analysis is the RGB-easy/RGB-hard split. DeLiVER is divided into 1797 RGB-easy and 100 RGB-hard samples, while FMB is divided into 183 RGB-easy and 97 RGB-hard samples. This split is designed to test whether auxiliary modalities are used selectively rather than indiscriminately. The results support that claim: performance improves clearly on RGB-hard scenes while remaining strong on RGB-easy scenes.

5. Relation to the SAM-adapter lineage

MM SAM-adapter belongs to the broader SAM-adapter family in the sense that it reuses a pretrained SAM backbone and adds lightweight adaptation modules, but its design logic is different from the image-only task-adaptation lineage represented by SAM-Adapter, SAM2-Adapter, and SAM3-Adapter (Chen et al., 2023, Chen et al., 2024, Chen et al., 24 Nov 2025). Those works adapt SAM-family models to low-level or domain-specific segmentation tasks such as camouflaged object detection, shadow detection, and medical image segmentation, typically by keeping a large pretrained encoder mostly frozen, injecting lightweight task-specific modules, and fine-tuning a mask decoder. Their “composability” generally refers to combining multiple sources of visual prior information, not to RGB-plus-auxiliary-sensor fusion.

By contrast, MM SAM-adapter is explicitly multimodal and explicitly semantic. It discards SAM’s prompt encoder and mask decoder, keeps only the image encoder, and converts SAM into the dominant RGB branch of a multimodal dense prediction system. This distinguishes it both from prompt-conditioned SAM adaptations and from medical PEFT variants such as SAM-DA or FSAM, which are parameter-efficient but remain single-image or single-modality at inference time (Tejero et al., 12 Jan 2025, Bui et al., 11 May 2026).

The closest architectural analogy is to ViT-Adapter-style side tuning, not to classic bottleneck adapters inserted into every frozen transformer block. The multimodal side branch is lighter than the SAM backbone and uses repeated Injector-Extractor interaction rather than symmetric fusion. This asymmetry is deliberate. When the side-branch encoders are enlarged to create a more symmetric architecture, performance on DeLiVER RGB-LiDAR drops from 57.14 overall and 45.46 RGB-hard to 55.27 overall and 42.22 RGB-hard (Curti et al., 12 Sep 2025). That result reinforces the paper’s central argument: SAM’s RGB priors should remain primary, and auxiliary modalities should not be treated as equal partners in every scene.

Ablations also clarify the relation to other PEFT schemes. On DeLiVER RGB-LiDAR, a plain “SAM + Segmentation Head” baseline obtains 53.32 overall and 35.57 on RGB-hard. With RGB-only input, frozen SAM plus LoRA reaches 53.18, while SAM plus the adapter reaches 55.98. With multimodal RGB+LiDAR, multimodal LoRA reaches 53.97 overall and 40.22 RGB-hard, still below MM SAM-adapter at 57.14 overall and 45.46 RGB-hard (Curti et al., 12 Sep 2025). This indicates that repeated multimodal injection through Injector-Extractor interactions is more effective than a low-rank perturbation strategy in this setting.

6. Limitations and implications

The current framework has several explicit limitations (Curti et al., 12 Sep 2025). It supports only two input modalities at once, due mainly to constraints of the adopted Road-Fusion module. It still requires pixel-aligned modalities. It is computationally heavy because it uses SAM ViT-L plus two ConvNeXt encoders and cross-attention-based adapters. The paper does not provide explicit FLOP or latency comparisons. These characteristics place MM SAM-adapter closer to a high-capacity research architecture than to a minimal deployment model.

At the same time, its broader significance is methodological. It is the first work described as adapting SAM for multimodal semantic segmentation using an adapter network that injects fused multimodal features into SAM’s RGB feature stream (Curti et al., 12 Sep 2025). That result suggests a specific principle for foundation-model-based multimodal perception: a large pretrained segmentation model can remain the dominant RGB prior, while auxiliary modalities can be encoded in a lighter side branch and fused only where they improve scene understanding. The RGB-easy/RGB-hard analysis supports this interpretation by showing that multimodal information can improve adverse-condition performance without systematically degrading favorable-condition performance.

A common misconception is that any SAM-adapter with multiple guidance signals is already “multimodal.” The recent SAM-adapter literature shows that this is not generally the case. Some methods define composable task priors, sum visual signals such as patch embeddings and high-frequency components, or transfer across several medical modalities across experiments, yet still operate on one image stream at a time (Chen et al., 24 Nov 2025, Tejero et al., 12 Jan 2025). MM SAM-adapter is different because the auxiliary modality is a first-class input stream, fused and injected into SAM throughout the forward pass.

The framework is therefore most accurately characterized as a foundation-model-centric multimodal segmentation architecture: SAM supplies the dominant RGB representation, the multimodal side branch supplies aligned auxiliary evidence, and the adapter regulates information flow between them. Within the SAM-adapter landscape, that combination marks a transition from image-only downstream specialization to explicit multimodal scene understanding.