Parameter-Efficient Modality Adaptation

• Parameter-efficient modality adaptation is a design approach that utilizes lightweight modules to enable multimodal models to adapt to new tasks with minimal additional parameters.
• Techniques such as prompt-based adaption, low-rank adapters, and explicit cross-modal fusion mechanisms minimize computational overhead while achieving competitive performance.
• This strategy enhances deployment in resource-constrained scenarios and applications like vision-language, audio-visual, and medical imaging by maintaining strong generalization with frozen backbones.

Parameter-efficient modality adaptation refers to strategies designed to enable large-scale multimodal models to adapt to new modalities, domains, or tasks by introducing only a small number of new trainable parameters, while freezing the majority of the backbone weights. These techniques facilitate practical deployment of massive models in multimodal settings—such as vision-language, audio-visual, and medical imaging—by enabling cross-modal information fusion, robust handling of missing modalities, and domain adaptation, all within tight computational and memory budgets. Recent research has advanced the architectural and algorithmic foundations of parameter-efficient adaptation across diverse scenarios.

1. Core Principles and Objectives

The primary objective of parameter-efficient modality adaptation is to minimize the number of newly introduced or updated parameters required for effective transfer or extension of pre-trained multimodal models. This is achieved by:

• Inserting lightweight adaptation modules (e.g., adapters, prompt vectors, low-rank modules) at key points in the model architecture, while keeping the bulk of the model—transformer layers, encoder backbones, classifier heads—frozen.
• Promoting modularity and extensibility to support rapid adaptation across new tasks or modalities, including the ability to handle scenarios where modalities may be missing or variable at inference time.
• Maintaining, or even enhancing, the performance and generalization abilities of the full model adaptation, as shown empirically in multiple benchmarks (Lu et al., 2023, Guo et al., 2024, Wei et al., 5 Jun 2025, Zhao et al., 9 Nov 2025, Saadi et al., 2024).

Distinct classes of adaptation mechanisms are employed, ranging from input-centric or prompt-based modules (Liang et al., 2022, Wang et al., 2023), through residual adapters and low-rank factorization approaches (Wei et al., 5 Jun 2025, Saadi et al., 2024, Zhou et al., 26 Mar 2025), to sequence-level and outer-product fusion adapters for rich cross-modal interactions (Guo et al., 2024). Modality-robust design, bidirectional knowledge transfer (common vs. specific updates), and hybrid sharing patterns (e.g., partially shared and private adapters) are crucial for ensuring both unimodal and cross-modal knowledge transfer.

2. Adapter Classes and Methodological Taxonomy

Parameter-efficient modality adaptation encompasses a range of design philosophies and concrete adapter realizations. Key approaches include:

A. Prompt-based and Input-centric Adaptation

• Learns a small fixed set of prompt vectors or “pseudo-tokens” prepended or inserted into the input stream, which guide cross-modal alignment without altering the backbone (Liang et al., 2022, Wang et al., 2023).
• Modular extension to new modalities is possible by adding new prompt blocks, each with negligible (<0.02%) parameter overhead relative to full models.

B. Bottleneck and Low-rank Adapters

• Adapters are typically two-layer feed-forward (down-projection, non-linearity, up-projection) modules inserted after attention or MLP layers; only the bottleneck weights are trained (Wang et al., 2023, Wei et al., 5 Jun 2025, Lu et al., 2023).
• Low-rank adaptation (LoRA) directly factorizes updates as $B A$ (with $A \in \mathbb{R}^{r \times k}$, $B \in \mathbb{R}^{d \times r}$, $r \ll d, k$), and often maintains separate sets for each modality or for modality-common and modality-specific factors (Saadi et al., 2024, Zhao et al., 9 Nov 2025).

C. Cross-modal Interaction Mechanisms

• Recent methods introduce explicit cross-modal modules, e.g., lightweight cross-attention blocks, tensor (outer-product) fusion adapters, or auxiliary experts for interaction between non-text and text tokens (Guo et al., 2024, Wei et al., 5 Jun 2025).
• Some frameworks (e.g., MokA) strictly separate unimodal low-rank adaptation and cross-modal enhancement to maximize both within- and between-modality transfer (Wei et al., 5 Jun 2025).

D. Proxy and Modality-robust Tokens

• Cross-modal proxy tokens (e.g., mask tokens) are trained to synthesize “hallucinated” class tokens for missing modalities, improving robustness in cases of partial observation (Reza et al., 29 Jan 2025).

E. Feature-wise Modulation

• Element-wise scale-and-shift adapters (analogous to FiLM) inserted after key linear/convolutional blocks can adapt intermediate representations to new modalities with $\ll1\%$ parameter overhead (Reza et al., 2023).

3. Algorithmic and Architectural Formulations

Several seminal architectural and mathematical strategies underlie state-of-the-art parameter-efficient adaptation:

Modular Adapter Placement

Adapter Location	Example Approaches	Parameter Impact
Input/Prompt	PromptFuse, AdaLink	$<0.02$\% overhead
Per-layer Adapter	LoRA, UniAdapter, PEMMA, MoRA	$\sim$0.1–8\% overhead
Fusion/Output	Wander, Cross-modal Proxy	$\sim$0.1–2\% (task-tuned)

Low-Rank Adapter Equation (LoRA, MoRA, PEMMA, MokA):

$W = W_0 + BA$

where $W_0$ is frozen and $A$, $B$ are the only trainable parameters (per adapter instance).

Cross-Modal Adapter (MokA):

In each adapted layer, for $n$ modalities and token sequence $\mathbf{x}$ split as $[\mathbf{x}^1; \dotsc; \mathbf{x}^n]$: $$\Delta W\,\mathbf{x} = [B(z^1 + \mathrm{CA}^1); \dotsc; B(z^m + \mathrm{CA}^m); \dotsc; Bz^t]$$ with $z^i = A^i \mathbf{x}^i$ and $\mathrm{CA}^i$ cross-attention from text tokens.

CP-Decomposed Sequence Fusion (Wander):

Efficient outer-product sequence fusion with CANDECOMP/PARAFAC: $$\tilde H_t = \sum_{r_t=1}^{R_t} \sum_{r_h=1}^{R_h} \bigotimes_{m=1}^M \left( \mathbf{w}_{t,m}^{(r_t)} h_{m} \mathbf{w}_{h,m}^{(r_h)T} \right)$$ No explicit high-order tensor instantiated; complexity is $O(R d \sum_m d_m)$.

Proxy Token Alignment (U2A):

Train a single mask token per modality with alignment loss: $$\mathcal{L}_{\mathrm{align}} = \left\|\hat{\mathcal{T}}_{\mathrm{MT},m_1} - \hat{\mathcal{T}}_{\mathrm{CLS},m_2}\right\|^2 + \left\|\hat{\mathcal{T}}_{\mathrm{MT},m_2} - \hat{\mathcal{T}}_{\mathrm{CLS},m_1}\right\|^2$$

4. Fusion, Robustness, and Multimodal Challenges

Practical parameter-efficient adaptation is governed by three interlinked challenges:

A. Cross-Modal Fusion Efficiency:

Careful selection of fusion sites (prompt, fusion layer, or blockwise sequence) enables context-dependent, token-level, or expert-mediated information interaction. Canonical mechanisms include low-rank adapters at fusion boundaries (Zhou et al., 26 Mar 2025, Guo et al., 2024), outer-product/shared-factorization (Guo et al., 2024), or cache-based cross-modal retrieval with adaptive weighting (Yang et al., 2024).

B. Robust Adaptation under Missing Modalities:

To address missing-modal inference:

• MoRA uses modality-common Gram-matrix low-rank factors for bidirectional transfer across frozen encoders (Zhao et al., 9 Nov 2025).
• Proxy-token frameworks (U2A) synthesize missing modality representations via aligned mask tokens, trained with explicit alignment loss (Reza et al., 29 Jan 2025).
• Scale & Shift adaptation learns tiny, per-layer feature-wise affine parameters, restoring performance to near full-modality baselines (Reza et al., 2023).

C. Parameter/Latency Trade-offs:

Most advanced frameworks (e.g., UniAdapter, MoRA, MokA, PEMMA) achieve $<2\%$ of backbone parameters actively tuned, minimizing inference overhead, with empirical evidence showing negligible or modest increases in runtime over prompt-based or adapter-free baselines (Lu et al., 2023, Zhao et al., 9 Nov 2025, Wei et al., 5 Jun 2025, Saadi et al., 2024).

5. Applications and Experimental Evidence

Parameter-efficient modality adaptation has demonstrated notable gains across a broad spectrum of multimodal scenarios:

• Vision-language retrieval, captioning, and VQA: UniAdapter achieves or exceeds full fine-tuning recall at $\leq2\%$ of parameters, enabled by hybrid adapter placement and partial weight sharing (Lu et al., 2023).
• Audio-visual and egocentric video: Ego-VPA leverages a small basis-prompt bank for joint frame and text adaptation, outperforming prompt-tuning and matching or surpassing full fine-tuning at $0.84\%$ added parameters (Wu et al., 2024).
• 3D understanding: Any2Point adapts text or vision models to point cloud tasks using virtual projection and guided adapters, boosting 3D accuracy at $<1\%$ parameter cost (Tang et al., 2024).
• Medical imaging and prognosis: PEMMA orchestrates LoRA/DoRA adapters per modality to support CT–PET–EHR fusion, yielding +28% Dice improvement on PET and +23% C-index on EHR, with $\leq8\%$ parameter overhead (Saeed et al., 18 Apr 2025).
• Multilingual multimodal ASR: Adapter-centric transfer in SeamlessM4T demonstrates $>$90% parameter savings and up to 17% WER reduction in zero-shot transfer (Gupta et al., 2024).
• Missing-modality robustness: Both MoRA and U2A surpass prompt-based and baseline architectures by 2–5% in scenarios with up to 90% missing modality rates (Zhao et al., 9 Nov 2025, Reza et al., 29 Jan 2025).

Table: Parameter Overhead and Representative Results

Method	Param. Overhead	Notable Benchmarks	Key Result(s)
MoRA	$\sim$0.11%	MM-IMDb, Food101, Hateful Memes	+5.24% F1 vs. prompt SOTA at 0% latency
Ego-VPA	0.84%	Charades-Ego, EGTEA	Outperforms full FT, +1 pt top-1 Acc.
Wander	1–4M ($<$1%)	CMU-MOSI, IEMOCAP, MSRVTT	Matches full FT, scalable to ≥3 modalities
UniAdapter	1–2%	MSR-VTT, VQAv2, MSCOCO	Exceeds full FT on several metrics
U2A (mask)	0.14%	Food101, MM-IMDb, Kinetics-Sound	$>$2% top-1 boost for missing modalities

6. Design Trends, Limitations, and Future Directions

Recent developments have unified several trends:

• Specialization of adaptation modules by modality (e.g., distinct adapter banks or basis prompts) is crucial for minimizing cross-modal entanglement and catastrophic forgetting (Saeed et al., 18 Apr 2025, Wei et al., 5 Jun 2025, Guo et al., 2024).
• Cross-modal interaction must be explicit and tunable—hybrid designs outperform both naive prompt/adapters and pure LoRA inserts (Wei et al., 5 Jun 2025, Zhou et al., 26 Mar 2025).
• Adapter merging and continual learning strategies, such as CoPA-Merging, have been developed to aggregate low-rank adapters from multiple experts or tasks, preserving principal directions and scaling appropriately via pruning and complementary scaling, and achieving stronger zero-tuning generalization (Zeng et al., 24 Feb 2025).
• Non-intrusiveness in adaptation (e.g., AdaLink) streamlines deployment and mitigates serving complexity, though with some trade-off in expressiveness for highly complex tasks (Wang et al., 2023).

Limitations:

• Most current methods are validated primarily on vision-text (and increasingly audio-visual) benchmarks; fewer results exist for highly heterogeneous modality combinations (e.g., sensor–tabular–text) or streaming inputs.
• Adapter capacity and placement require task- and backbone-specific tuning; over- or under-provisioned adaptation modules may bottleneck transfer.
• Real-world data often exhibit partial, corrupted, or hierarchically structured modality loss; extending these frameworks to handle arbitrary missing-data patterns remains ongoing research (Zhao et al., 9 Nov 2025, Reza et al., 2023).

Future Directions:

• Scalable compositionality: Dynamic instantiation and routing of adapters for compositional (e.g., instruction-following) tasks in MLLMs.
• Modality discovery and automatic sharing: Learning when and where to share vs. specialize adapters for emergent modalities.
• Efficient continual and federated adaptation: Adapter merging and reparameterization under constrained and decentralized data regimes (Zeng et al., 24 Feb 2025).

7. Synthesis and Outlook

Parameter-efficient modality adaptation represents a convergence of architectural modularity, information-theoretic fusion, and scalable optimization. The paradigm has shifted multimodal model deployment from monolithic retraining toward flexible, plug-in modules with clear theoretical and empirical advantages. As foundational multimodal models become more pervasive, the efficiency, robustness, and universality of such schemes—across missing-modality, resource-limited, and rapidly evolving task landscapes—will continue to drive both practical impact and deep theoretical questions (Wei et al., 5 Jun 2025, Guo et al., 2024, Zhou et al., 26 Mar 2025, Zhao et al., 9 Nov 2025, Saeed et al., 18 Apr 2025, Wang et al., 2023, Lu et al., 2023).