Modality Adapters: Mechanisms & Applications

• Modality adapters are trainable modules that transform modality-specific features into a shared latent space, enabling compatibility among diverse systems.
• They facilitate cross-modal fusion, continual learning, and temporal adaptation by integrating and aligning features from different modalities.
• Recent designs use encoder-decoder bridges, sparsity-aware routing, and quantization calibration to improve performance metrics and system efficiency.

Modality adapters (MAs) are mechanisms that reconcile heterogeneity across modalities, components, or operating regimes. In the narrow architectural sense, an MA is a trainable module that maps one modality’s representation into a form expected by another pretrained component, such as a text decoder, a vision-language head, or a multimodal encoder block. In a broader systems sense, the same term is used for routing, sparsity, or calibration mechanisms that decide how modalities should be fused, compressed, quantized, or converted. Recent work therefore treats MAs as an umbrella concept spanning encoder–decoder bridges, multimodal fusion blocks, continual-learning memory modules, and inference-time controller layers (Zhao et al., 2022, Seputis et al., 2024, Yu et al., 2024, Yang et al., 3 Apr 2026, Hu et al., 5 Mar 2026).

1. Terminological scope and recurring design patterns

Across the cited literature, the term modality adapter is not canonical. One recurring meaning is representation adaptation: a module transforms modality-specific features into a shared or target-compatible latent space. This is the role played by M-Adapter in end-to-end speech-to-text translation, by the Modality-Agnostic Adapter in fine-grained scene classification, and by the Multi-Modal Adapter for CLIP-style models (Zhao et al., 2022, Wang et al., 2024, Seputis et al., 2024).

A second meaning is cross-modal collaboration control. In continual multimodal learning, adapters are used not merely to align features but to preserve prior modality knowledge while enabling transfer to a newly introduced modality. PathWeave’s Adapter-in-Adapter design and the cross-modality adapter of the continual-learning MMEncoder دونوں treat the adapter as a locus of plasticity, stability, and reuse rather than as a simple bottleneck (Yu et al., 2024, Chee et al., 10 Nov 2025).

A third meaning is temporal or reliability-aware adaptation. In multi-modal tracking, the Visual and Memory Dual Adapter couples cross-modal appearance transfer with memory-based temporal propagation. In multimodal speaker identification, the AMR adapters map audio and face embeddings into a common routing space so that a downstream router can estimate modality weights per sample (&&&10&&&, Zuo et al., 28 Jun 2026).

A fourth meaning is adapter-like systems control, where the mechanism is not a conventional parameter-efficient tuning layer. MSAO operationalizes modality adaptation as sparsity-aware offloading; MoMa uses modality-partitioned sparse experts inside an early-fusion transformer; MASQuant applies modality-specific smoothing and low-rank compensation during post-training quantization; and MATE realizes modality adaptation as interpreter-based routing among specialized conversion agents rather than as a neural block inserted into a backbone (Yang et al., 3 Apr 2026, Lin et al., 2024, Hu et al., 5 Mar 2026, Algazinov et al., 24 Jun 2025).

2. Encoder–decoder bridge adapters

In spoken-language systems, MAs often sit between a speech encoder and a decoder LM, producing a shorter sequence of “soft tokens” that the decoder can treat as LM-compatible inputs. The interpretability study of three spoken LLMs identifies two common MA families in this setting: a multi-layer perceptron that projects encoder outputs into the LM embedding space, and a Q-Former that learns fixed query tokens attending over encoder states (Ògúnrèmí et al., 2 Oct 2025).

The most explicit bridge design in the cited set is M-Adapter for end-to-end speech-to-text translation. Its motivation is the modality gap created when a model combines a pretrained speech encoder with a pretrained text decoder: speech encoder outputs are long, dense, acoustically redundant, and temporally fine-grained, whereas text decoders expect shorter, semantically structured sequences. M-Adapter addresses both sequence compression and representation adaptation without requiring transcription or phoneme supervision (Zhao et al., 2022).

Architecturally, M-Adapter replaces the standard self-attention block with Multi-head Pooled Self-Attention. Given speech features $\mathbf{X} \in \mathcal{R}^{L \times D}$, it forms $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, pools each with a 1D convolution, and reduces sequence length from $L$ to

$L'=\left\lfloor\frac{L+2 p - k}{s}\right\rfloor+1.$

Attention is then computed on the pooled sequences:

$$\text { Attention }(Q, K, V)=\operatorname{Softmax}\left(Q^{\prime} K^{\prime T} / \sqrt{d}\right) V^{\prime}.$$

Because $\mathbf{X}$ and the attention output have different lengths, an additional pooling layer $\operatorname{Pool_X}$ maps $\mathbf{X}$ to $\mathbf{Q}$0 so that residual addition and layer normalization remain valid. One layer reduces length by a factor of $\mathbf{Q}$1; stacking $\mathbf{Q}$2 layers reduces it by $\mathbf{Q}$3 (Zhao et al., 2022).

This design couples local and global modeling. Convolutional pooling extracts local patterns while downsampling, and self-attention preserves long-range dependencies. The reported ablations indicate that changing where pooling occurs degrades performance, and removing the global attention component at inference lowers BLEU, supporting the claim that adaptation requires more than local compression (Zhao et al., 2022).

Empirically, on MuST-C English$\mathbf{Q}$4German, the best M-Adapter configuration, W2V2-mAda$\mathbf{Q}$5-mBart, reaches 27.13 BLEU when the decoder is fine-tuned, compared with 26.12 for W2V2-cnn-mBart and 26.56 for W2V2-tran-mBart. With the decoder frozen, it reaches 27.73 BLEU versus 26.45 and 26.91, respectively. Across English$\mathbf{Q}$6German/Romanian/French, the paper reports an average gain of 0.78 BLEU over the CNN baseline, with about 1.01 BLEU on English$\mathbf{Q}$7German in the fine-tuned-decoder setting (Zhao et al., 2022).

3. Joint fusion, harmonization, and modality-agnostic integration

A major line of work uses MAs to make heterogeneous modalities comparable before fusion, or to adapt modalities jointly rather than independently. In fine-grained scene image classification, the Modality-Agnostic Adapter first removes distribution discrepancy with modality-specific MLPs,

$\mathbf{Q}$8

then concatenates aligned features,

$\mathbf{Q}$9

and processes them with a modality-agnostic Transformer encoder,

$\mathbf{K}$0

The Transformer uses modality embeddings rather than positional embeddings. On Con-Text and Crowd Activity, MAA reports 90.87 and 88.29 mAP with global, local, and text features, compared with 89.53 and 87.45 for Wang et al. (2022); the global+text-only variant MAA* reports 90.13 and 88.07. The ablation on “modal difference elimination” further shows that independent MLPs outperform both no alignment and shared MLPs (Wang et al., 2024).

The Multi-Modal Adapter for CLIP takes a different route: it leaves the CLIP backbone frozen and inserts a trainable module after the encoders. Text embeddings for class prompts and the image embedding are concatenated, downsampled, processed by masked multi-head attention, upsampled through linear layers with GELU, split back into text and image features, and then mixed residually with the original CLIP embeddings using $\mathbf{K}$1. The method uses 4 attention heads and about 107,712 trainable parameters, with Adam, learning rate 0.005, batch size 256, and early stopping with patience 10. Its main empirical claim is not uniformly highest base-class accuracy but better generalizability on unseen classes and a more balanced base/new trade-off; an ablation further shows that removing text adaptation hurts performance, supporting the claim that joint multimodal adaptation is stronger than image-only adaptation (Seputis et al., 2024).

CHARM, for modality-agnostic semantic segmentation, explicitly contests a common assumption behind prior multimodal adaptation: that success requires homogenizing features into a single aligned space. Instead it advocates “cooperative harmonization rather than homogenization.” Its effective adapter is the combination of a Mutual Perception Unit (MPU), which uses window-based cross-modal attention where modalities act as queries and contexts for each other, and a dual-path optimization strategy composed of Collaborative Learning Strategy (CoL) and Individual Enhancement Strategy (InE). The final objective uses two supervised segmentation losses,

$\mathbf{K}$2

without an explicit pairwise feature-alignment term in the final CHARM objective. On DELIVER with MiT-B2, CHARM reports 54.96 Average mIoU, 68.43 Top-1, and 28.76 Last-1, corresponding to a +9.92 gain in Average and +28.04 in Last-1 relative to Any2Seg; on MCubeS and MUSES with MiT-B2 it reports 46.58/54.33/38.50 and 46.18/59.36/25.28, respectively (Wen et al., 5 Aug 2025).

Taken together, these works identify three distinct fusion regimes. One aligns modalities into a common semantic space before shared attention; one updates modalities jointly through explicit cross-modal attention; and one rejects homogenization, preferring implicit content alignment while preserving modality-specific strengths. This suggests that MA design is often governed by the failure mode a paper seeks to avoid: hard-coded modality hierarchy, independent adaptation, or complementarity loss.

4. Continual, temporal, and reliability-aware adapters

In continual multimodal learning, adapters are often the primary mechanism for balancing plasticity and memory. PathWeave treats multimodal expansion as a continual-learning problem over modalities and introduces the Adapter-in-Adapter framework inside a frozen X-InstructBLIP-style backbone. For modality $\mathbf{K}$3, the uni-modal adapter is added in parallel to the pretrained Q-Former layer,

$\mathbf{K}$4

with

$\mathbf{K}$5

Cross-modal reuse is provided by inserting in-adapters into prior frozen uni-modal adapters,

$\mathbf{K}$6

and a soft MoE gate assigns weights

$\mathbf{K}$7

The method expands to new modalities using only uni-modal data, keeps the heavy backbone frozen, introduces the MCL benchmark over image, video, audio, depth, and point cloud, and reports a 98.73% reduction in parameter training burdens. In ablations, the full method improves average transfer performance by 4.3 points over directly using incremental adapters, while removing the In-Adapter or the gating module reduces performance by at least 1.1 and 4.0 points, respectively (Yu et al., 2024).

A more explicitly task-incremental formulation appears in the pre-trained model-based framework for multi-modal continual learning via cross-modality adapters. Here, the adapter inside each MMEncoder block is a sparsely gated MoE whose gate uses averaged representations from all modalities,

$\mathbf{K}$8

and each expert down-projects, mixes information from other modalities, and up-projects before residual addition. The framework pairs this adapter with a representation alignment loss and a preservation loss, and freezes experts whose activation frequency exceeds 10% of samples after a task. On AVE, UESTC-MMEA, and SAMSEMO, it reports 91.91%/2.82%, 95.54%/1.21%, and 70.90%/2.43% for accuracy/forgetting, respectively; an adapter ablation on AVE shows 87.57% with no adapter, 88.39% with modality-specific adapters, and 91.91% with the proposed cross-modality adapter (Chee et al., 10 Nov 2025).

Temporal adaptation is central in multi-modal object tracking. VMDA plugs into a frozen or mostly frozen ODTrack backbone and combines a visual adapter with a memory adapter. The visual adapter performs frequency-, spatial-, and channel-wise transfer from the auxiliary modality to the dominant RGB modality; the memory adapter maintains short-term, long-term, and permanent memory pools with 8, 8, and 3 tokens of dimension 768, respectively. The tracking loss is

$\mathbf{K}$9

with focal classification loss and regression based on $\mathbf{V}$0 and GIoU, using $\mathbf{V}$1 and $\mathbf{V}$2. On LasHeR, the ablation progresses from 0.659/0.518 for the baseline to 0.718/0.565 with the visual adapter, 0.689/0.545 with the memory adapter, and 0.726/0.571 with both; on RGBT234, the full method reports PR = 0.919 and SR = 0.689 (Xu et al., 30 Jun 2025).

Reliability-aware adaptation appears in AMR for polyglot speaker identification. Two adapters map audio and face embeddings into a common 256-dimensional space using two-layer ReLU MLPs, and their outputs are consumed only by a router,

$\mathbf{V}$3

which weights modality-specific logits:

$\mathbf{V}$4

Training uses four sample types—ORIGINAL, AUDIO_REPLACE, FACE_REPLACE, and NO_FACE—and supervises router weights with a KL term,

$\mathbf{V}$5

with $\mathbf{V}$6. On the POLY-SIM 2026 evaluation set, the full system reports 99.93%, 100.00%, 97.50%, and 98.83% across P3, P5, P4, and P6, with 99.07% average accuracy, compared with 66.34% for the FOP baseline (Zuo et al., 28 Jun 2026).

5. Routing, offloading, conversion, and quantization as adapter-like control layers

Several papers use the language of modality adaptation while moving away from classical trainable adapter blocks. MSAO is explicit about this shift: its modality-aware component is a lightweight prefix probing network that measures spatial, temporal, and modal sparsity and combines them into Modality Activation Sparsity,

$\mathbf{V}$7

which then constrains edge–cloud offloading, compression, and speculative execution. Reported gains include about 30% reduction in end-to-end latency, 30%–65% reduction in resource overhead, and 1.5× to 2.3× throughput improvement. The modality-aware module itself adds 4.2 ms to 15.3 ms latency, 0.47% to 1.23% FLOPs, and 0.12 GB to 0.28 GB memory. An ablation removing the modality-aware mechanism drops VQAv2 accuracy from 76.5% to 69.7% and MMBench from 75.8% to 68.2%, while removing collaborative scheduling increases latency from 2.9 s to 4.3 s on VQAv2 and from 3.1 s to 4.5 s on MMBench (Yang et al., 3 Apr 2026).

MATE pushes this systems interpretation further. It does not define an internal neural adapter layer at all; instead, modality adaptation is realized through an interpreter agent plus seven expert agents that perform explicit modality conversion such as TTS, TTI, STT, ITT, ATI, ITA, and VTT. The routing component, ModCon-Task-Identifier, is a fine-tuned BERT classifier trained on the ModConTT dataset and reports 0.917 accuracy, 0.924 precision, 0.917 recall, and 0.916 F1, outperforming classical ML baselines and the GPT-3.5-Turbo interpreter on the reported setup. Default expert models include Tacotron 2, Stable Diffusion v1-4, Whisper, and BLIP. Here the “adapter” is effectively a task-to-tool routing layer for accessibility-oriented modality conversion (Algazinov et al., 24 Jun 2025).

MoMa and MASQuant are likewise adapter-like without being standard adapters. MoMa partitions each MoE layer into modality-specific expert groups inside an early-fusion transformer,

$\mathbf{V}$8

so text tokens cannot route to image experts and vice versa. Under a 1-trillion-token budget, the 1.4B model with 4 text experts and 4 image experts reports 3.7× overall FLOPs savings, with 2.6× for text and 5.2× for image, compared with a compute-equivalent dense baseline; however, throughput is lower than the dense model because of routing and synchronization overheads (Lin et al., 2024).

MASQuant, by contrast, is a post-training quantization framework. Its Modality-Aware Smoothing defines a separate diagonal smoothing matrix $\mathbf{V}$9 for each modality and optimizes

$L$0

then restores a single shared quantized weight through Cross-Modal Compensation using SVD-whitened low-rank corrections. On Qwen2.5-Omni-3B at W4A8, SmoothQuant collapses on audio with Librispeech and Wenetspeech WER of 77.4 and 94.2, whereas MASQuant reports 3.6 and 8.7. The paper therefore treats modality-aware adaptation as quantization-time calibration rather than as representation learning in the usual PEFT sense (Hu et al., 5 Mar 2026).

6. Interpretability, misconceptions, and unresolved design choices

The most direct study of what MAs represent internally appears in the investigation of intermediate representations in spoken LLMs. By aligning each MA output vector with a ground-truth word and retrieving the nearest decoder-LM token using mean-centered cosine similarity,

$L$1

the paper identifies two distinct MA strategies. In Whisper-based systems such as SALMONN and Qwen2-Audio, the MA tends to encode meaning through an English-based interlingua; in Phi-4-Multimodal-Instruct, which does not use Whisper, the MA instead tends to preserve phonetics expressed with English words. The probe results sharpen this distinction: for SALMONN, the MA lowers phone and word accuracy relative to Whisper Large v2, from 84.3 to 69.7 and from 52.9 to 38.8, while raising SpokenSTS $L$2 from 0.47 to 0.63; for Phi-4-MI, the MA increases phone accuracy from 30.8 to 32.4, word accuracy from 12.3 to 32.3, and SpokenSTS from 0.21 to 0.54 (Ògúnrèmí et al., 2 Oct 2025).

One recurring misconception is that an MA must be a small bottleneck module attached to a frozen backbone. The surveyed literature contradicts that view. MSAO, MATE, MoMa, and MASQuant all operationalize modality adaptation through routing, sparse expert allocation, or quantization-time calibration rather than through a classical adapter layer (Yang et al., 3 Apr 2026, Algazinov et al., 24 Jun 2025, Lin et al., 2024, Hu et al., 5 Mar 2026). Another misconception is that successful modality adaptation necessarily requires explicit homogenization of feature spaces. CHARM argues that explicit homogenization can dilute modality-specific advantages, and its best results are obtained by implicit alignment plus protected modality-specific optimization rather than by pairwise alignment losses (Wen et al., 5 Aug 2025).

A second unresolved choice concerns where adaptation should occur. Some works adapt representations before decoding; others adapt interaction patterns between modalities; others adapt memory, temporal propagation, or reliability estimation; and still others adapt system execution policies. A plausible implication is that MA design is best understood not by parameter count or architectural style, but by the interface being regularized: encoder–decoder compatibility, cross-modal fusion, temporal continuity, task expansion, modality trust, or systems efficiency. The continual-learning evidence that cross-modality adapters outperform modality-specific adapters—91.91% versus 88.39% on AVE in one study—further suggests that the target of adaptation is often the relation among modalities, not merely the marginal distribution of each modality taken separately (Chee et al., 10 Nov 2025).