Modality-Spectral Adapters in Vision Models

• MSA are lightweight trainable modules that extend frozen RGB vision models to process diverse spectral modalities (e.g., NIR, SWIR, LWIR, hyperspectral) while preserving pretrained semantic structure.
• They achieve parameter-efficient adaptation by localizing modality-specific computation in dedicated components, inserted either within transformer blocks or via separate spectral-spatial modules.
• Empirical results from SpectraDINO and HSI-Adapter show significant performance gains, validating the approach of bridging spectral gaps via targeted adapter design.

Searching arXiv for the cited papers to ground the article in current preprints. arXiv search query: (Nalcakan et al., 4 May 2026) Modality-Spectral Adapters (MSA) are lightweight trainable modules that extend a frozen RGB-pretrained vision foundation model to non-RGB spectral modalities while preserving the backbone’s pretrained semantic structure. In the available formulations represented by "SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters" (Nalcakan et al., 4 May 2026) and "Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models" (Hurtado et al., 24 Sep 2025), the central problem is the spectral gap between RGB-centric pretrained models and sensing regimes such as Near-Infrared (NIR), Short-Wave Infrared (SWIR), Long-Wave Infrared (LWIR), and hyperspectral imaging. Both systems treat the frozen ViT as a strong semantic prior and localize modality-specific adaptation in small trainable components, but they do so with different architectural emphases: SpectraDINO uses per-modality stems, embeddings, and residual bottleneck adapters inside every transformer block, whereas HSI-Adapter uses a spectral transformer, a spectral-enhanced spatial prior module, and modality-aware interaction blocks that exchange information bidirectionally with a frozen ViT (Nalcakan et al., 4 May 2026, Hurtado et al., 24 Sep 2025).

1. Spectral adaptation as a parameter-efficient extension of frozen VFMs

The defining property of MSA-style systems in these papers is that the vision backbone remains substantially or entirely frozen while modality-specific computation is introduced in narrowly scoped modules. SpectraDINO extends DINOv2 ViT backbones to beyond-visible modalities through lightweight, per-modality bottleneck adapters, while preserving the rich representations of the frozen RGB backbone (Nalcakan et al., 4 May 2026). HSI-Adapter similarly plugs into a frozen vision foundation model so the model can perform semantic segmentation directly from hyperspectral cubes instead of RGB or pseudo-RGB (Hurtado et al., 24 Sep 2025).

This design directly addresses a domain gap that is spectral rather than merely photometric. SpectraDINO frames NIR, SWIR, and LWIR as complementary sensing modalities that are critical for robust perception in adverse conditions but remain mismatched to RGB-centric pretrained models (Nalcakan et al., 4 May 2026). HSI-Adapter frames hyperspectral imaging as a setting where dense wavelength bands encode material and illumination structure that cannot be captured by RGB-optimized architectures (Hurtado et al., 24 Sep 2025). In both cases, the adapter is not described as replacing the pretrained model; it is a bridge that allows the new modality to enter a representation space already shaped by large-scale RGB pretraining.

A plausible implication is that MSA denotes a design family rather than a single module topology. What is shared is the adaptation principle: small trainable spectral modules, frozen or largely frozen foundation backbones, and explicit mechanisms for aligning heterogeneous sensing statistics with pretrained visual semantics.

2. Recurring architectural pattern

Across the two systems, three recurrent design choices appear: modality-specific front-end processing, constrained interaction with a frozen ViT, and a downstream objective that benefits from preserving pretrained structure rather than overwriting it.

System	Frozen-backbone policy	Modality-specific trainable components
SpectraDINO	The core attention and MLP weights of the ViT are shared and unchanged across modalities; Stage III unfreezes the top half of the transformer blocks, and only 25% for ViT-G	Per-modality input stem, per-modality embedding vector, per-block bottleneck adapters
HSI-Adapter	The ViT backbone remains frozen, including patch embedding, positional embeddings, and transformer weights	Spectral Transformer, Spectral-Enhanced Spatial Prior Module, modality-aware interaction blocks, decoder and auxiliary head

The table highlights a key distinction. SpectraDINO inserts modality-specific computation around and inside every block of a shared ViT, using residual correction after each block (Nalcakan et al., 4 May 2026). HSI-Adapter keeps the ViT frozen and instead builds a fusion bridge that injects and extracts information at selected stage groups $[0,2], [3,5], [6,8], [9,11]$ (Hurtado et al., 24 Sep 2025).

This suggests that MSA mechanisms can be organized either as intra-backbone residual adapters or as external spectral-spatial processors with staged interaction points. In both cases, the architectural constraint is deliberate: adaptation capacity is concentrated in modules that learn to “speak the language” of the pretrained backbone rather than relearning the backbone itself.

3. SpectraDINO and per-modality residual correction

SpectraDINO implements the most literal MSA formulation in the provided material. The model consists of three modality-specific components around a shared ViT backbone: a per-modality input stem, a per-modality embedding vector, and per-block bottleneck adapters (Nalcakan et al., 4 May 2026).

At the input level, RGB uses a special $1\times 1$ convolution stem initialized near identity, while each single-channel infrared modality uses its own two-layer spatial stem with GELU and instance normalization:

$$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$

After patchification and positional encoding, a learned modality embedding $\mathbf{E}[m]\in \mathbb{R}^D$ is broadcast and added to all tokens so the shared transformer can condition on whether the input is RGB, NIR, SWIR, or LWIR (Nalcakan et al., 4 May 2026).

The adapter proper is a residual bottleneck MLP inserted into every transformer block:

$$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$

with

$$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$

where $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$ and $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$. Each transformer block has a separate adapter instance per modality, so there are four adapters per block: RGB, NIR, SWIR, and LWIR. For ViT-B/14, the paper states $12$ blocks $\times$ $1\times 1$0 modalities $1\times 1$1 $1\times 1$2 adapters, each with about $1\times 1$3 parameters, for a total overhead of roughly $1\times 1$4M parameters, far smaller than retraining a full ViT (Nalcakan et al., 4 May 2026).

Initialization is central to the method’s stability. Adapter weights use small Gaussian std $1\times 1$5 and biases are zero, making the initial network behave almost exactly like the frozen DINOv2 backbone. The paper identifies this as important for preventing early training from destabilizing the pretrained representation (Nalcakan et al., 4 May 2026).

The paper’s own contrast with neighboring strategies is also instructive. SpectraDINO does not branch into separate modality-specific transformers, does not duplicate attention or MLP weights per sensor, and does not rely on a sophisticated fusion network as the primary adaptation mechanism. Instead, it uses a single shared ViT plus small per-modality bottlenecks. In that sense, the method is close in spirit to adapter tuning, but it is explicitly spectral because it combines modality-specific stems for raw sensor normalization, modality embeddings for global conditioning, and separate adapters per modality at every block (Nalcakan et al., 4 May 2026).

4. HSI-Adapter and bidirectional hyperspectral interaction

HSI-Adapter instantiates the same broad MSA principle for high-dimensional hyperspectral input, but its mechanism is not residual per-block correction. Instead, it is a spectral-spatial feature extractor plus fusion bridge between HSI and a frozen RGB-pretrained ViT (Hurtado et al., 24 Sep 2025).

The architecture has three parts: a Spectral Transformer, a Spectral-Enhanced Spatial Prior Module (SPM), and Modality-Aware Interaction Blocks. The hyperspectral input cube $1\times 1$6 is processed in parallel by the Spectral Transformer and the SPM, while the frozen ViT backbone retains its patch embedding, positional embeddings, and transformer weights (Hurtado et al., 24 Sep 2025).

The Spectral Transformer models inter-band dependencies by treating spectral channels as a sequence at each spatial location. A Band Embedding layer maps each band to a $1\times 1$7-dimensional vector, and the embedded spectral sequence is processed by a Transformer encoder with $1\times 1$8 layers and $1\times 1$9 attention heads. The standard transformer attention form applies along the spectral dimension:

$$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$0

After spectral modeling, the outputs are average pooled across the spectral dimension to form $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$1, which is then projected by a linear layer into a $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$2-channel representation for the frozen ViT patch embedding (Hurtado et al., 24 Sep 2025).

The SPM supplies multiscale spatial structure. It begins with a depthwise separable $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$3 convolution across spectral channels, followed by batch normalization and ReLU, then a convolutional stem of three stacked $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$4 convolutions $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$5 BN $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$6 ReLU and a $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$7 max-pooling layer. Three hierarchical convolutional stages then produce $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$8 at $$\text{SpatialStem}(\mathbf{x}) = \text{IN}\!\left(\text{Conv}_{3\times3}^{16 \to 3}\!\left(\sigma\!\left(\text{Conv}_{3\times3}^{1 \to 16}(\mathbf{x})\right)\right)\right).$$9 resolution, $\mathbf{E}[m]\in \mathbb{R}^D$0 at $\mathbf{E}[m]\in \mathbb{R}^D$1, and $\mathbf{E}[m]\in \mathbb{R}^D$2 at $\mathbf{E}[m]\in \mathbb{R}^D$3, which are projected to a shared embedding dimension $\mathbf{E}[m]\in \mathbb{R}^D$4 using $\mathbf{E}[m]\in \mathbb{R}^D$5 convolutions before being passed to the interaction blocks (Hurtado et al., 24 Sep 2025).

The Modality-Aware Interaction Block is the core adapter mechanism. Injection is performed with deformable attention over three feature levels $\mathbf{E}[m]\in \mathbb{R}^D$6, with $\mathbf{E}[m]\in \mathbb{R}^D$7 heads and $\mathbf{E}[m]\in \mathbb{R}^D$8 sampling points per head per level. The paper then applies a learned modality gate,

$\mathbf{E}[m]\in \mathbb{R}^D$9

where $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$0 is a learned per-token coefficient. After passage through the frozen ViT layers, the updated ViT tokens attend back to adapter features via deformable cross-attention, followed by a cross-attention feedback block. The interaction is therefore bidirectional: adapter $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$1 ViT via injection, and ViT $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$2 adapter via extraction and feedback (Hurtado et al., 24 Sep 2025).

This architecture differs from a simple preprocessor. The paper explicitly states that the adapter is not merely a front-end spectral compression block; it is a mechanism for repeated alignment and exchange between heterogeneous feature spaces. That distinction is important for understanding why the method is categorized as a modality-spectral adapter rather than a spectral encoder attached to an otherwise unchanged segmentation model.

5. Training objectives, curricula, and empirical evidence

SpectraDINO uses a teacher-student setup in which a frozen DINOv2 teacher is fed only RGB images while the student processes both RGB and multispectral inputs through the shared backbone with modality-specific stems and adapters (Nalcakan et al., 4 May 2026). The total objective is a weighted sum of four losses:

$$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$3

The four terms are cosine distillation of the CLS token against the frozen RGB teacher, a symmetric InfoNCE contrastive objective between RGB-student and multispectral-student embeddings with $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$4, a patch-wise alignment loss with a patch-sample ratio of $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$5 in Stages I–II and $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$6 in Stage III, and the neighborhood-structure-preservation loss $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$7, which uses a FIFO queue of teacher embeddings with queue capacity $$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$8, top-$$\mathbf{x}' = \text{Block}(\mathbf{x}) + \text{Adapter}_{m}(\text{Block}(\mathbf{x})),$$9, and temperature $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$0 (Nalcakan et al., 4 May 2026).

The staged curriculum is integral to the paper’s explanation of trainability. Stage I freezes the entire ViT backbone and trains only stems, modality embeddings, per-block adapters, and the final layer norm, using $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$1, $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$2, $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$3, and no $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$4. Stage II keeps the backbone frozen and introduces $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$5 after a one-epoch queue warmup. Stage III unfreezes the top half of the transformer blocks, reduces $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$6 to $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$7, increases $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$8 to $$\text{Adapter}(\mathbf{x}) = \mathbf{W}_{\text{up}}\,\sigma\!\left(\mathbf{W}_{\text{down}}\mathbf{x}\right),$$9, increases $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$0 to $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$1, and keeps $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$2 at full weight (Nalcakan et al., 4 May 2026). The paper presents this progression as the mechanism that prevents catastrophic forgetting while still allowing limited backbone co-adaptation.

HSI-Adapter uses a simpler optimization objective tailored to semantic segmentation:

$$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$3

where both terms are standard pixel-wise cross-entropy losses. The ViT backbone remains frozen, and the trainable components are the Spectral Transformer, Spectral-Enhanced SPM, interaction blocks, decoder, and auxiliary head. Optimization details include batch size $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$4, cosine learning rate schedule, warm-up of $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$5 iterations, learning rate $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$6 for HSI-Drive and HCV2, $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$7 for HyKo2, random cropping at $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$8 for HSI-Drive and HyKo2 and $$\mathbf{W}_{\text{down}}\in\mathbb{R}^{D/4\times D}$$9 for HCV2, with no test-time augmentation and no multi-scale evaluation (Hurtado et al., 24 Sep 2025).

The reported empirical evidence directly supports the adapter role in both papers.

System	Benchmark	Reported result
SpectraDINO	RASMD, frozen DINOv2 backbone with no adapters	$\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$0 mAP
SpectraDINO	RASMD, stems plus adapters	$\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$1 mAP
SpectraDINO	RASMD, full SpectraDINO with half-unfreezing	$\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$2 mAP
SpectraDINO	RASMD, full unfreezing	$\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$3 mAP
SpectraDINO Large	RASMD	$\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$4 mAP
HSI-Adapter	HSI-DriveV2	mIoU $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$5, aAcc $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$6, mAcc $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$7
HSI-Adapter	HCV2	mIoU $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$8, aAcc $\mathbf{W}_{\text{up}}\in\mathbb{R}^{D\times D/4}$9, mAcc $12$0
HSI-Adapter	HyKo2-VIS	mIoU $12$1, aAcc $12$2, mAcc $12$3

In SpectraDINO, the largest single ablation gain comes from adding modality-specific stems plus adapters, and full unfreezing performs worse than half-unfreezing, which the paper interprets as evidence that too much backbone plasticity harms the transferred RGB structure (Nalcakan et al., 4 May 2026). In HSI-Adapter, the ablation sequence on HSI-DriveV2 progresses from $12$4 mIoU for a $12$5-channel ViT from scratch, to $12$6 for the Spectral Transformer $12$7 IN-22K, to $12$8 with the Spectral-Enhanced SPM, to $12$9 with HSS/interaction blocks $\times$0 DINOv2, indicating that spectral modeling, multiscale spatial priors, and bidirectional interaction each contribute distinct gains (Hurtado et al., 24 Sep 2025).

6. Scope, misconceptions, and limitations

A common misconception is that modality-spectral adaptation can be reduced to early spectral projection into three channels. The two papers reject that view in different ways. SpectraDINO confines modality-specific computation to stems, adapters, and the modality embedding table, but it inserts those components throughout the depth of the network rather than only at the input (Nalcakan et al., 4 May 2026). HSI-Adapter explicitly states that its adapter is not a simple front-end preprocessor; it is a spectral-spatial extractor plus a bidirectional interaction bridge with the frozen ViT (Hurtado et al., 24 Sep 2025). In both cases, adaptation is distributed and structurally coupled to the pretrained representation.

Another potential misconception is that spectral adaptation necessarily requires full backbone finetuning or separate modality-specific transformers. SpectraDINO is unusually conservative about backbone modification: it does not duplicate attention or MLP weights per sensor, and its ablations show that full unfreezing is inferior to partial unfreezing on RASMD (Nalcakan et al., 4 May 2026). HSI-Adapter keeps the entire ViT frozen, including patch embedding and positional embeddings, and attributes performance to explicit spectral modeling and modality-aware interaction rather than backbone modification (Hurtado et al., 24 Sep 2025).

The limitations are also specific. SpectraDINO trains on paired RGB-MS data only; there are no direct NIR-SWIR-LWIR pairings, so RGB serves as the alignment hub. Modalities are sampled round-robin during pretraining, which equalizes update counts but may under-optimize the harder thermal alignment case. The paper further notes that because there is no multi-infrared synchronized training, the model never learns direct inter-IR relationships, and suggests that tri- or quad-modal datasets could improve LWIR consistency (Nalcakan et al., 4 May 2026). HSI-Adapter does not present an analogous inter-modality pairing issue because it operates directly on hyperspectral cubes, but its analysis emphasizes that performance depends on narrow wavelength regions: zeroing bands in the $\times$1–$\times$2 nm region causes up to approximately $\times$3-point drops for pedestrians and $\times$4–$\times$5-point drops for painted metal and road markings (Hurtado et al., 24 Sep 2025).

Taken together, these works indicate that modality-spectral adapters are best understood as parameter-efficient mechanisms for preserving foundation-model priors while explicitly modeling the sensor-specific structure of spectral data. In SpectraDINO, that structure is captured through per-modality residual bottlenecks trained with global, local, and neighborhood-preserving alignment losses. In HSI-Adapter, it is captured through spectral attention, spectrum-aware spatial priors, and bidirectional interaction blocks. A plausible implication is that future MSA systems will continue to differ in module topology while retaining the same underlying principle: frozen pretrained semantics combined with narrowly targeted spectral adaptation.