Cross-Modal Adapter for Efficient Multimodal Fusion

• Cross-modal adapters are lightweight modules that integrate parameter-efficient transformation mechanisms to fuse vision and language data.
• They employ bottlenecked residual architectures with cross-attention and dynamic gating, reducing the need for extensive tuning.
• Their practical applications span retrieval, continual learning, and domain adaptation, enabling efficient adaptation of large pre-trained models.

A cross-modal adapter is a parameter-efficient architectural module, typically inserted into or alongside large frozen backbones, that enables enhanced fusion, transfer, or alignment of multiple modalities—most commonly vision and language—by facilitating information flow between them. These adapters inject lightweight, often bottlenecked, transformation and attention mechanisms that (unlike purely unimodal adapters) explicitly model cross-modal interactions while minimizing the number of tunable parameters required for adaptation, fine-tuning, or continual learning. The cross-modal adapter paradigm supports scaling, quick adaptation to new modalities or tasks, efficiency in low-resource or federated learning, and is actively employed in retrieval, reasoning, tracking, and multi-modal LLM frameworks.

1. Core Design Patterns and Mathematical Principles

Cross-modal adapters generalize the bottlenecked residual architecture from standard unimodal adapters to the multimodal setting. The canonical design applies a down-projection (bottleneck) to each modality’s features, optionally performs cross-attention or gating, then re-projects to the original dimension. Key mathematical constructs include:

• Residual bottleneck: For feature $x\in\mathbb{R}^d$, the adapter applies

$$\mathrm{Adapter}(x) = x + s\, \sigma(x W_{\text{down}}) W_{\text{up}}$$

where $W_{\text{down}}\in\mathbb{R}^{d\times r}$, $W_{\text{up}}\in\mathbb{R}^{r\times d}$, $r\ll d$, $s$ is a scaling constant, and $\sigma$ a nonlinearity.

• Cross-modal fusion: Adapters employ cross-attention or gating between representations. For example, in “V-expert” X-adapters (Zhang et al., 2023), PLM hidden state $x$ is fused with top-$K$ CLIP image embeddings $V$ using multi-head attention:

$$\text{head}_i = \mathrm{Attn}(Q=u_0 W_q^i,\; K=V W_2 W_k^i,\; V=V W_2 W_v^i)$$

• Mixture-of-Experts (MoE): To capture diverse cross-modal mappings and avoid catastrophic forgetting in continual learning, parallel expert modules with a learned gating network are used:

$$\tilde{f}^{(k)} = f^{(k)} + \sum_{i=1}^E w_i g_i^{(k)} \quad (w_i=\mathrm{softmax}(\mathcal{G}(x)))$$

as in multi-modal continual learning settings (Chee et al., 10 Nov 2025, Xia et al., 1 Apr 2025).

• Dynamic parameter generation: Adapters can dynamically generate their parameters conditioned on input semantics, improving flexibility for cross-lingual and cross-modal generalization (Cai et al., 18 Dec 2024).
• Hybrid or dual adapters: Progressive or hierarchical fusion—such as spatio-temporal, shallow-deep, or unimodal-to-cross-modal adapters—enables multistage information exchange and alignment (Li et al., 3 Aug 2025, Ji et al., 19 Mar 2025).

2. Parameter Efficiency and Transfer Learning

Cross-modal adapters are motivated by the need to adapt large-scale vision-language pre-trained models (VLPs) without incurring prohibitive compute and storage costs. Typical strategies include:

• Freezing the backbone: Only adapter parameters are updated, preserving all pre-trained knowledge; e.g., X-adapter on BERT/CLIP (Zhang et al., 2023), UniAdapter on BLIP (Lu et al., 2023), CROME in MLLMs (Ebrahimi et al., 13 Aug 2024), and XMAdapter on CLIP (Yang et al., 19 Apr 2024).
• Minimal trainable parameters: Adapters usually account for <10% (often 1–6%) of model parameters (Lu et al., 2023, Zhang et al., 2023, Chen et al., 20 Mar 2025). For example, CROME adapter comprises ≈5M trainable weights versus 7–13B in the LLM backbone (Ebrahimi et al., 13 Aug 2024).
• Decoupled local/global adaptation: In federated or personalized settings, adapters separate client-local up/down projections from globally shared cross-modal projections, drastically reducing communication (Ghiasvand et al., 7 Jul 2025).
• Efficiency in memory, compute, and convergence time: Adapter-tuning avoids the full-gradient backpropagation and large-parameter checkpoint storage of fine-tuning. Empirically, adapter-based tuning yields ≈2–3× faster training and 25–40% less memory usage (Lu et al., 2023, Zhang et al., 2023).

3. Architectural Variants and Domain-Specific Extensions

Cross-modal adapters exhibit a wide diversity of architectural instantiations depending on task and domain:

• Video-text alignment: Cross-modal adapters are placed in both visual and text transformers, often enabling early or late fusion (see “UniCrossAdapter” for radiology (Chen et al., 20 Mar 2025), cross-modal text-video retrieval (Jiang et al., 2022)).
• Instance-conditioned adaptation: For compositional generalization in egocentric video, adapters compute a per-instance vector that is added to every text embedding, enabling video-conditioned classification without retraining heavy encoders (Kukleva et al., 28 Mar 2024).
• Dual-branch progressive adapters: DMTrack employs (i) self-prompting spatio-temporal adapters per modality, followed by (ii) shallow and deep cross-modality adapters for pixel-level complementarity (Li et al., 3 Aug 2025).
• Optimal transport-based adapters: OTA fuses image and text streams via parallel adapters and a cross-modal attention block, embedding the alignment as an optimal transport problem solved by Sinkhorn iterations, boosting few-shot generalization (Ji et al., 19 Mar 2025).
• Dynamic or input-conditional adapters: DASD dynamically generates adapter weights for every target-language caption, guided by a semantics disentangling module that separates style/content factors (Cai et al., 18 Dec 2024).
• Cache and prompt-based decoupling: XMAdapter leverages both trained image/text caches and cross-modal projection MLPs, dynamically fusing image and text similarity metrics with an adaptive ratio for few-shot classification and domain generalization (Yang et al., 19 Apr 2024).
• MoE adapters with knowledge preservation: Cross-modality MoE adapters support continual learning by gating, expert freezing, and relation-regularized knowledge transfer (Chee et al., 10 Nov 2025, Xia et al., 1 Apr 2025).

4. Training Objectives and Loss Functions

Cross-modal adapters are supervised via modular objectives tailored to cross-modal fusion and transfer:

• Standard cross-entropy/classification loss: Used in most supervised settings and downstream fine-tuning (e.g., radiology report generation (Chen et al., 20 Mar 2025), person ReID (Xie et al., 1 Jul 2025)).
• Contrastive losses: NT-Xent / InfoNCE (cosine contrast) between cross-modal representations for retrieval and matching tasks (Lu et al., 2023, Yang et al., 19 Apr 2024, Cai et al., 18 Dec 2024).
• Feature/logit distillation losses: Knowledge distillation between teacher and student adapters in missing-modality or weak-label scenarios (Nguyen et al., 17 Nov 2025).
• Cross-modal attention/reconstruction: Optimal transport loss (OTA) using entropy-regularized Sinkhorn solutions (Ji et al., 19 Mar 2025), with entropy-aware sample weighting to regularize convergence.
• Continual alignment and knowledge preservation: Representation-alignment (anchor-to-joint) loss and inter-sample relation regularization (Chee et al., 10 Nov 2025, Xia et al., 1 Apr 2025).
• Dynamic adaptation losses: Semantic consistency (alignment with source-language features) and adversarial disentangling in dynamic settings (Cai et al., 18 Dec 2024).
• Federated/asymmetric optimization: Personalization uses local adapters; only shared cross-modal projections are averaged and communicated, minimizing overhead (Ghiasvand et al., 7 Jul 2025).

5. Empirical Performance and Scope of Application

Cross-modal adapters have been empirically validated across a range of multimodal tasks:

Task Domain	Performance Gains	Adapter Type
Video-Text/Image-Text Retrieval	+2–6% R@1/+mAR over previous SOTA (Lu et al., 2023, Zhang et al., 2023)	UniAdapter, X-adapter, XMAdapter
Cross-modal generalization (CLIP, audio-text, etc.)	1–2 pts R@1 gains on zero-shot retrieval; +3–6 BLEU in TIMT	CMoE-Adapter (Xia et al., 1 Apr 2025), modal adapter (Ma et al., 2023)
Continual Learning & Knowledge Retention	+3.5–4.3% multi-task accuracy, lower forgetting (Chee et al., 10 Nov 2025)	Cross-modality MoE adapters
Multimodal LLMs (Instruction, VQA)	Zero-shot/fine-tuned SOTA on 6/8 MLLM benchmarks (Ebrahimi et al., 13 Aug 2024)	CROME-adapter
Federated Learning (personalization/generalization)	+6–8% unseen-class accuracy; 100x comms reduction (Ghiasvand et al., 7 Jul 2025)	pFedMMA multi-modal adapters
Few/Zero-shot remote sensing classification	+1.8% (1-shot) / +3.2% (5-shot) over full fine-tune (Ji et al., 19 Mar 2025)	OT-adapter
Cross-lingual retrieval	Dynamic adapter: +16.9 mAR vs. static (Cai et al., 18 Dec 2024)	Dynamic adapter + SDM
Egocentric action recognition across datasets	+5–10 pts noun/verb harmonic mean (Kukleva et al., 28 Mar 2024)	X-MIC instance-conditioned

Adapters consistently match or surpass fully fine-tuned or prompt-tuned alternatives while tuning orders of magnitude fewer parameters and enabling scalable, plug-and-play adaptation.

6. Open Problems, Generalizations, and Future Directions

The design of cross-modal adapters increasingly addresses new challenges:

• Scalability to many modalities: MoE architectures and Pseudo-Modality Replay enable continued expansion to arbitrary sensory data (e.g., audio, speech, video, image, LiDAR) without catastrophic forgetting (Xia et al., 1 Apr 2025).
• Dynamic context and low-resource adaptation: Dynamic adapters with semantic disentangling enable cross-lingual adaptation using per-caption parameterization (Cai et al., 18 Dec 2024).
• Test-time and training-free adaptation: Techniques such as test-time distribution learning and cache-fusion adapters enable adaptation without any retraining steps, crucial in few-shot and open-world settings (Zhang et al., 10 Mar 2024, Yang et al., 19 Apr 2024).
• Prompt and adapter hybridization: Efficient architectures combine prompt learning, cache-based retrieval, and cross-modal adapters for enhanced flexibility (Jiang et al., 14 Dec 2024).
• Robustness to domain shift and personalization: Asymmetric adaptation, selective expert freezing, and cross-modal residuals collectively address generalization and overfitting in federated, cross-domain, or continual learning (Ghiasvand et al., 7 Jul 2025, Chee et al., 10 Nov 2025).

A plausible implication is that adapters will remain central to maintaining a tractable adaptation cost, knowledge preservation, and sample efficiency as the dimensionality and diversity of multimodal data and pre-trained models continues to grow.

7. Representative Research Contributions

Paper	Adapter Mechanism	Key Contributions
"Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained LLMs with Cross-Modal Adapters" (Zhang et al., 2023)	X-adapter (V-/T-expert)	Inject CLIP image/text into PLMs; ~3.7% param budget; 7.6 BLEU gain
"UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling" (Lu et al., 2023)	Unified cross-modal	Bottleneck w/ partial sharing; fusion adapter; matches/exceeds full-tune
"Optimal Transport Adapter Tuning for Bridging Modality Gaps..." (Ji et al., 19 Mar 2025)	OTA (CMAM+Sinkhorn OT)	CPAM attention; entropy-weighted OT loss; SOTA on FS-RSSC
"CROME: Cross-Modal Adapters for Efficient Multimodal LLM" (Ebrahimi et al., 13 Aug 2024)	Gated pre-LLM adapter	5M parameters, SOTA 0-shot and fine-tune, robust to scale-up
"Continual Mixture of Experts Adapter" (Xia et al., 1 Apr 2025)	MoE with codebook/PMR	Continual learning, dynamic codebook, EWC, replay, multi-modal
"Dynamic Adapter with Semantics Disentangling..." (Cai et al., 18 Dec 2024)	Dynamic/conditional	Per-example parameter generation, cross-lingual retrieval
"pFedMMA: Personalized Federated..." (Ghiasvand et al., 7 Jul 2025)	Local-global split	Federated MMA, asymmetric comms, SOTA personalization/generalization

These architectural paradigms collectively define the contemporary landscape of cross-modal adapter research, shaping both efficient adaptation and scalable multimodal cognition.