Parameter-Efficient Scaling in Multi-Modal Research

• Parameter-efficient scaling in multi-modal research is a paradigm that integrates heterogeneous data sources like text, speech, and images while optimizing trainable parameters and computational resources.
• The approach extends classical uni-modal scaling laws with explicit cross-modal interaction terms, demonstrating how synergistic effects can overcome modality competition.
• Techniques such as adapters, LoRA, and mixture-of-experts, coupled with efficient distributed training, enable scalable, resource-conscious deployment of advanced multi-modal systems.

Parameter-efficient scaling in multi-modal research addresses the challenge of constructing neural models that process heterogeneous data sources—such as text, speech, images, code, and others—while optimizing the use of trainable parameters, compute, and memory. This paradigm is crucial for both advancing foundational models that must handle diverse input streams and for practical deployment where computational resources are constrained. Core advances have centered on generalizing scaling laws to mixed-modal regimes, designing efficient architectural modules for adaptation, and empirically characterizing the interaction between competing or synergistic modalities.

1. Unified and Mixed-Modal Scaling Laws

Scaling laws for uni-modal models typically express task loss $L$ as a function of the model parameter count $N$ and the size of the modality-specific training data $D_j$, with the canonical Chinchilla-like form:

$$L(N, D_j) = E_j + \frac{A_j}{N^{\alpha_j}} + \frac{B_j}{\lVert D_j \rVert^{\beta_j}},$$

where $E_j$ is the irreducible loss, and the exponents $\alpha_j$, $\beta_j$ are modality-specific (bounded by 1/2) (Aghajanyan et al., 2023).

For mixed-modal settings with two modalities ($D_i$, $D_j$), the empirical law extends additively with an explicit interaction term

$$L(N, D_i, D_j) = \frac{1}{2}[L(N, D_i) + L(N, D_j)] - \mathcal{C}_{(i,j)} + \frac{A_{(i,j)}}{N^{\alpha_{(i,j)}}} + \frac{B_{(i,j)}}{(\lVert D_i \rVert + \lVert D_j \rVert)^{\beta_{(i,j)}}}.$$

Here, $\mathcal{C}_{(i,j)}$ quantifies synergy (when negative) or competition (when positive) between modalities, thus predicting when cross-modal fusion yields gains or induces loss barriers. This unified formulation captures the empirical finding of an optimal regime—typically achieved at large model and data scales—where synergistic effects overcome competitive interference, as seen in a 30B-param, 45B-token Speech–Text model outperforming its unimodal variants (Aghajanyan et al., 2023).

2. Parameter-Efficient Adaptation Techniques

Parameter-efficient fine-tuning (PEFT) methods—such as adapters, prompt-tuning, Low-Rank Adaptation (LoRA), BitFit, and their multi-modal extensions—are foundational to efficient scaling in multi-modal regimes.

• Adapters: Lightweight modules (e.g., added after transformer sub-blocks) trained while freezing the backbone. In Pre-CoFactv2, adapters are injected in large pre-trained models (Swinv2, DeBERTa) for multi-modal fact verification, minimizing parameter updates while enabling rich multi-modal integration (Du et al., 2023).
• LoRA and Contextualized Variants: LoRA injects low-rank updates into projection matrices, $W \leftarrow W + AB$, updating only $$A \in \mathbb{R}^{d \times r}, B \in \mathbb{R}^{r \times d}$$, $r \ll d$. Context-PEFT extends this by conditioning updates on per-token context (modality), allowing distinct adaptation per modality in a frozen backbone (Hadji-Kyriacou et al., 2023).
• Prompt-based Adaptation: In visual prompt-based multi-modal tracking (ViPT), learned modal prompts are injected at various layers in the frozen transformer, tuning <1% of parameters and outperforming full fine-tuning in RGB+Depth, RGB+Thermal, and RGB+Event settings (Zhu et al., 2023). Bi-directional adapters further generalize this, allowing cross-modal prompt fusion that dynamically adapts to dominant modality changes (Cao et al., 2023).
• Efficient LayerNorm Tuning: Efficient multi-modal adaptation is possible by tuning only LayerNorm parameters, viewed as domain adaptors, yielding competitive or better performance than LoRA or full finetuning—even when the adaptation parameter count is as low as 0.003–0.004% of total parameters (Zhao et al., 2023).

3. Scaling Laws for Mixture-of-Experts (MoE) in Multi-Modal Contexts

MoE architectures scale parameter efficiency by conditionally routing tokens through a subset of multiple “experts”—independent sub-networks—such that only a small active submodel is evaluated per input. They enable models with far greater total capacity without linearly increasing compute or memory usage (Ludziejewski et al., 7 Feb 2025, Zhao et al., 28 Sep 2025).

A comprehensive scaling law for MoEs considers not just data size ($D$) and total parameter count ($N$) but also the activated parameter count per forward pass ($N_a$), number of active experts per token ($G$), and the shared expert ratio ($S$):

$$L(N, D, N_a, G, S) = (e \cdot G + \frac{f}{G} + m S^2 + n S) \left( \frac{1}{N^\alpha} + \frac{k}{N_a^\alpha} + h \frac{N_a}{N} \right) + \frac{a}{N^\alpha} + \frac{b}{D^\beta} + \frac{c}{N_a^\alpha} + \epsilon$$

(Zhao et al., 28 Sep 2025).

Empirical experiments (n=446) demonstrate:

• Optimality is achieved for $G \approx 7$, $S \approx 0.31$, and an activated ratio $N_a/N$ that decreases (“sparser” activation) as total size increases.
• MoE configurations can outperform dense models—with lower memory requirements to reach the same loss—when trained on roughly $E$ times more tokens for $E$ experts at constant total parameter budget (Ludziejewski et al., 7 Feb 2025).

This systematic optimization enables practitioners to tune MoE models for deployment in resource-constrained, multi-modal environments.

4. Architectural and Optimization Strategies for Scaling

Beyond core adaptation techniques and scaling laws, several system-level and architectural optimizations are necessary for parameter-efficient scaling:

• Efficient Distributed Training: Application of parallelism strategies (data, model, tensor parallelism) and optimizer partitioning (e.g., DeepSpeed ZeRO stages) enable very large multi-modal networks to train with bounded memory and compute (Benington et al., 2023). For multi-modal models, distributed strategies may be aligned with modality-specific towers, with ZeRO partitioning gradients and parameters by modality, and communication costs modeled as $T_{\text{comm}} = \alpha \log(P) + \beta n$ for $P$ devices.
• One-Stage Unified Training Paradigms: SPHINX-X replaces multi-stage pipelines with a one-stage format, training on a mixed, multi-task dataset using all available data in a conversational setup. In synergistic configurations, the use of complementarily pre-trained visual experts (CNN + ViT, with redundant encoders removed) and learnable skip tokens further reduces parameter overhead, preserving spatial fidelity without unnecessary computation (Liu et al., 8 Feb 2024).
• Module Sharing and Selective Fusion: The use of a single shared encoder (with modality-specific classification heads) and shared cross-attention modules can halve the parameter count for multi-modal classification while maintaining or improving performance—demonstrated in skin lesion diagnosis (Tang et al., 28 Mar 2024). Loss functions can be biased toward the dominant modality based on prior domain knowledge for improved representational efficiency.
• Efficient Attention Skipping: In MLLMs, the Efficient Attention Skipping (EAS) mechanism evaluates and skips redundant Multi-Head Attention modules, inserting lightweight propagation-of-information adapters (PIAs) that are merged into feed-forward networks for zero extra latency. Empirical results show >2x inference speedup and 23.8% reduction in updated parameters with accuracy maintained (Wu et al., 22 Mar 2024).

5. Empirical Phenomena and Optimization Insights

Empirical studies across tasks have revealed several phenomena pertinent to efficient scaling:

• Emergent Coordinate-Ascent Dynamics: During mixed-modal training, optimization can alternate focus between modalities, producing loss plateaus in some but not all modalities. This effect weakens at larger scales as capacity suffices to fit both modalities jointly (Aghajanyan et al., 2023).
• Marginality of PET Design Differences at Scale: As model scale increases, the specific design of PEFT modules becomes less critical; large models (due to redundancy and higher degrees of freedom) are robust to suboptimal module placement, and performance differences between prompt, adapter, BitFit, and LoRA narrow (Su et al., 2023).
• Minimal “Low Threshold” for Tunable Parameters: Empirical results indicate a minimal parameter threshold must be surpassed before fine-tuning outperforms random initialization across tasks—once surpassed, full-finetuning performance can be nearly achieved with a relatively small number of tunable parameters (Su et al., 2023).
• Multi-Modal Scaling Law Hypotheses: Model performance in multi-modal contexts is predicted by the sum over modalities of the log of effective training tokens (i.e., $\sum_i \log(T_i / C_i)$, where $T_i$ is raw data size, $C_i$ is the modality-specific compression factor), plus $\log(P)$. Leveraging efficiently compressed, abundant multi-modal data can offset the need for very large models and facilitate deployment on edge devices (Sun et al., 10 Sep 2024).

6. Applications and Future Directions

The above principles yield concrete benefits in diverse applications:

• Medical image segmentation models leverage LoRA/DoRA and modular transformer backbones to add new modalities without catastrophic forgetting or retraining—achieving +28% Dice score in PET scans for an 8% parameter cost relative to early fusion and with robust continual learning across imaging, EHR, and beyond (Saadi et al., 21 Apr 2024, Saeed et al., 18 Apr 2025).
• Parameter-efficient, side-tuning adapters and prompt-based architectures enable scalable multi-modal tracking and visual-linguistic tasks (e.g., referring expression comprehension) with only 2.1–9% of the full parameter count and drastically reduced memory/compute requirements (Liu et al., 1 Jul 2024).
• Unified transformer architectures (e.g., Transfusion) jointly model both text (via next-token prediction) and images (via diffusion loss) with only lightweight modality encoders and highly compressed image tokens—processing, for example, 256×256 images as just 16 tokens—achieving state-of-the-art cross-modal generation at a fraction of the compute of quantized baselines (Zhou et al., 20 Aug 2024).
• In-context learning for MLLMs is enabled at scale by transforming multi-modal demonstrations into “virtual tokens” via a frozen backbone and lightweight projection layer, supporting efficient and effective in-context reasoning independent of the base model architecture (Gao et al., 11 Jun 2024).

Theoretical scaling relationships (e.g., $N \propto D^{0.44}$ for efficient LLMs) provide design guidance, including the expected regime where emergent skills (such as cross-modal alignment) manifest. MoE scaling analyses further indicate that optimal active expert count and sharing ratios are largely architecture- and data-independent, considerably simplifying the search for efficient multi-modal MoE configurations (Ludziejewski et al., 7 Feb 2025, Zhao et al., 28 Sep 2025, Kausik, 22 Feb 2024).

Looking forward, parameter-efficient scaling principles are expected to further inform the design of universal, deployable, multi-modal systems, with continuing refinement of joint scaling laws as models, data diversity, and application requirements co-evolve.