Cross-Modality Adapter in Multi-Modal Learning

Updated 17 November 2025

Cross-modality adapters are efficient modules that integrate heterogeneous data modalities in deep neural networks for robust multi-modal learning.
They employ lightweight bottleneck designs, partial weight sharing, and cross-attention mechanisms to fuse information across vision, language, audio, and more.
Their design enables scalable, low-resource adaptation in tasks such as retrieval, classification, and continual multi-modal learning.

A cross-modality adapter is a parameter-efficient architectural module designed to enable or optimize learning across heterogeneous data modalities within deep neural networks, particularly in multi-modal or cross-modal learning settings. In contrast to traditional adapters, which typically adapt unimodal representations toward new tasks, cross-modality adapters explicitly facilitate information integration, transfer, or alignment between diverse modalities such as vision, language, audio, thermal, or others. These modules are typically lightweight and introduced into pre-trained networks to enable multi-modal interaction, fusion, or adaptation while keeping most backbone parameters frozen. Contemporary cross-modality adapters are central to foundation model adaptation, continual multi-modal learning, and real-world applications involving incomplete, misaligned, or low-resource modal data.

1. Architectural Principles and Adapter Placement

Recent designs follow the paradigm of bottleneck adapters: small parameter-efficient modules are grafted into selected positions in pre-trained transformers or convolutional networks. In cross-modal architectures, adapters are introduced not only within independent modality-specific streams but also at cross-modal interaction points.

Examples:

Transformer-based Vision-LLMs: UniAdapter (Lu et al., 2023) inserts two-layer bottleneck adapters into each Transformer block of both the vision (ViT), text (BERT), and cross-modal encoder streams, leveraging partial weight sharing to minimize parameter overhead while preserving cross-modal alignment.
Video-Text Retrieval: MV-Adapter (Jin et al., 2023) follows a bottleneck+TRM (Transformer) structure in both video and text branches, with the video branch featuring additional temporal adaptation and cross-modality tying for shared weight calibration.
Specialized Cross-Attention/Fusion: Several adapters, such as the cross-modality attention adapter (CAA) (Shi et al., 2023), are placed at pre-defined fusion points, e.g., after several ViT backbone layers, to interleave modality streams and inject cross-modal information before downstream decoding or task heads.

Placement strategy is task-dependent. Adapters can be situated:

Alongside self-attention and feed-forward sublayers for per-layer interaction (UniAdapter, MV-Adapter).
Exclusively at late-stage fusion points for efficiency or robustness (CAA for medical images).
In both modality-specific and joint cross-modal branches, as in multi-adapter networks for RGB-Thermal tracking (Li et al., 2019, Lu et al., 2020).

A canonical cross-modality adapter can be described as follows:

Bottleneck Structure: Given feature $x \in \mathbb{R}^d$ , apply

$\text{Adapter}(x) = x + f((x W_{\mathrm{down}} + b_{\mathrm{down}})) W_{\mathrm{up}} + b_{\mathrm{up}}$

where $W_{\mathrm{down}} \in \mathbb{R}^{d \times r}$ , $W_{\mathrm{up}} \in \mathbb{R}^{r \times d}$ , $r \ll d$ .

Partial Weight Sharing: UniAdapter achieves parameter efficiency by sharing $W_{\mathrm{down}}$ across all modalities and maintaining modality-specific $W_{\mathrm{up}}$ , adding a split-branch up-projection for cross-modal tokens.
Cross-Attention and Modality Gating: Cross-modal adapters often utilize attention mechanisms to couple modalities. For example, MV-Adapter’s Cross Modality Tying (Jin et al., 2023) calibrates adapter weights using a shared low-dimensional factor, encouraging alignment between modalities in a shared subspace.
Mixture-of-Experts for Continual Learning: In continual multi-modal learning (Chee et al., 10 Nov 2025), adapters instantiate a set of $E$ experts, with a learned gating network $G$ assigning weights $w_e$ for expert-specific adaptive fusion across $K$ modalities:

$\tilde{f}^k = f^k + \sum_{e=1}^E w_e g^k_e$

where $g^k_e$ incorporates both intra- and inter-modal information.

Dynamic Parameter Generation: For cross-lingual retrieval, dynamic adapters generate layerwise parameters conditioned on disentangled semantic and form features of the input caption using a generator network, driven by coupled consistency and adversarial disentangling losses (Cai et al., 18 Dec 2024).
Optimal Transport and Cross-Modal Attention: OTA (Ji et al., 19 Mar 2025) introduces a cross-modal attention mechanism in the text encoder—explicitly "reading” visual tokens—and, by solving an entropy-regularized optimal transport problem, establishes a global alignment between sampled image and text features.

The following table compares representative adapter module forms in different settings:

Paper	Adapter Type	Fusion/Interaction
UniAdapter	Bottleneck + partial sharing	Intra-modality + cross-modal encoder ([CLS])
MV-Adapter	Bottleneck + TRM, CMT	Shared calibration factor in down-proj step
Multi-Adapter RGBT	Parallel residual branch, small kernels	Summed with large shared convs layerwise
CAA (SAM)	1×1 conv fusion, residual	Added after mid-ViT blocks to two modality streams
OTA (OTAT)	Cross-modal attention + OT	Global feature-level alignment (Sinkhorn)
DASD	Dynamic parameter gen	Input-conditioned, semantic style-adaptive

3. Training Protocols and Loss Functions

Cross-modality adapters are almost invariably optimized with the vast majority of backbone parameters frozen. Typical training regimes involve:

Contrastive losses for retrieval or alignment (e.g., InfoNCE in retrieval, supervised contrastive for representation alignment).
Task-specific objectives (cross-entropy for classification, sequence generation likelihood for radiology report generation (Chen et al., 20 Mar 2025), Dice plus CE for medical segmentation (Shi et al., 2023)).
Auxiliary divergence or regularization losses:
- Hierarchical divergence loss (HD Loss) via MK-MMD maximizes separation of modality-specific adapters, while minimizing the gap in modality-shared adapters (Lu et al., 2020).
- Optimal Transport Losses compute alignment costs between sets of visual and textual features, solved using the Sinkhorn algorithm, and regularized by entropy-aware or difficulty-weighted losses (Ji et al., 19 Mar 2025).
- Adversarial and semantic consistency losses to ensure representation disentangling and stable transfer across modalities/languages (Cai et al., 18 Dec 2024).
- Knowledge preservation and representation alignment losses in continual learning enforce stability across tasks and robust cross-modal clustering (Chee et al., 10 Nov 2025).

Adapter training typically requires an order of magnitude fewer trainable parameters and less GPU memory than full fine-tuning, enabling efficient multi-task deployment.

4. Empirical Effectiveness and Parameter Efficiency

Empirical benchmarks consistently demonstrate that cross-modality adapters attain competitive, often superior, performance with a fraction of the parameter cost and faster convergence:

Video-text retrieval: UniAdapter tunes only 2.2% of a 337 M parameter BLIP but achieves or surpasses R@1 of full-tuned methods (Lu et al., 2023); MV-Adapter tunes ≈2.4% of CLIP, achieving best-in-class retrieval on five datasets (Jin et al., 2023).
Multi-modal continual learning: Cross-modality adapters using mixtures of experts show large reductions in forgetting (expressed as Fgt metric) and accuracy gains over uni-modal baselines and prior prompt-based approaches (Chee et al., 10 Nov 2025).
Cross-lingual retrieval: DASD significantly outperforms static adapter baselines (e.g., +7.3 points mAR on MSCOCO multi-lingual) thanks to dynamic input-conditioned generation (Cai et al., 18 Dec 2024).
Medical and domain adaptation: Cross-modality adapters enable adaptation of large-scale vision-LLMs (e.g., CLIP) to radiology data (Chen et al., 20 Mar 2025), yielding BLEU and METEOR gains while tuning only 1–2% of the parameters.
Modality fusion reliability: In missing-modality regimes, designs using residual adapters in both self-attention and MLP paths maintain stable accuracy above single-modality models (Li et al., 2023).

Comprehensive ablation studies validate that cross-modal adapters in both unimodal streams and fused branches yield the highest accuracy, and that strategies such as partial weight sharing or dynamic parameterization further boost parameter efficiency and generalization.

5. Design Variants, Extensions, and Tradeoffs

Contemporary cross-modality adapters demonstrate several orthogonal axes of design:

Parameter sharing (e.g., UniAdapter) vs. modality-specific branches (e.g., Multi-Adapter RGBT).
Static vs. dynamic parameterization (DASD introduces semantic-disentangled, input-conditioned dynamic adapters (Cai et al., 18 Dec 2024)).
Local (token- or patch-level) attention/fusion vs. global (prototype- or instance-level) alignment (e.g., optimal transport adapters (Ji et al., 19 Mar 2025)).
Early vs. late fusion (early fusion via layer-wise adapters or late fusion via head-level cross-modal attention).
Mixture-of-experts vs. bottleneck-only designs, wherein expert selection and freezing provide architectural lifelong learning capability (Chee et al., 10 Nov 2025).

Each design entails tradeoffs in adaptation speed, resource usage, and the scope of cross-modal information flow. Adapter stacking deep within encoders supports fine-grained interaction but increases memory and sequence length; late fusion is more efficient but may underutilize semantic overlap.

6. Applications, Limitations, and Outlook

Cross-modality adapters have been adopted in a range of real-world and research domains:

Vision-language retrieval and classification (e.g., UniAdapter, MV-Adapter, OTA)
Continual multi-modal learning: preserving knowledge and enabling new modality/task addition (Chee et al., 10 Nov 2025)
Medical imaging/diagnosis: parameter-efficient adaptation to radiology or multimodal MRI segmentation (Chen et al., 20 Mar 2025, Shi et al., 2023)
Cross-lingual cross-modal retrieval: dynamic adaptation across language and modality gaps (Cai et al., 18 Dec 2024)
Few-shot transfer and low-resource domains: adapters are crucial for efficient downstream task adaptation where annotation is scarce.

Limitations arise in scenarios with extremely high modality imbalance or misalignment, tasks needing very long-range temporal dependencies, or where dynamic adaptation complexity may outweigh parameter savings.

Future directions suggest greater integration of dynamic, input-aware adapter structures (as in DASD), the evolution of learnable expert selection/freezing for lifelong continual learning, and the extension of cross-modality adapters to new modalities (e.g., audio, geospatial, structured data) in foundational multi-modal systems. The modularity and efficiency of cross-modality adapters position them as essential components for scalable, robust, and adaptable multi-modal AI.