---
title: Multimodal Model Merging
url: https://www.emergentmind.com/topics/multimodal-model-merging
type: topic
---

# Multimodal Model Merging

Multimodal Model Merging is the process of algorithmically fusing multiple deep neural models—each trained on different modalities or multimodal tasks—into a single parameter set that expresses the complementarity of the original networks. This approach enables the construction of unified, zero-shot capable large models that leverage the diversity of independently optimized experts for language, vision, audio, and other modalities, without resource-intensive joint fine-tuning or access to the original pretraining data. The field integrates advances in weight-space interpolation, sparsification, attention alignment, and importance-driven fusion, with recent work extending to parameter-efficient, heterogeneous-architecture, and lifelong (temporal) multimodal settings. Key challenges include mitigating catastrophic forgetting, balancing modality-specific and cross-modal capabilities, and reducing performance interference among constituent experts.

## 1. Foundations and Motivations

Model merging emerged to address practical limitations in large-scale multimodal systems: the cost and inflexibility of joint retraining, the need for rapid integration of modality/domain specialists, and the desire to preserve or superimpose fine-grained expert capabilities within a single deployable backbone. In the multimodal setting, this means combining models such as vision–language, audio–language, or point cloud–language LLMs, or deeper fusions of visual, audio, and coding LLMs into a single MLLM. 

The foundational principle is the weight-space connectivity of deep networks: independently fine-tuned models, originating from a shared initialization, typically reside in a connected (flat) loss basin, permitting direct parameter-space interpolation or arithmetic to fuse expertise [2408.07666]. Empirical studies confirm that simple forms of layerwise or vectorial merging suffice to approximately preserve performance on both constituent modalities and joint tasks, even in the absence of further data [2304.14933, 2510.01845, 2601.07645].

## 2. Core Methodologies

Multimodal model merging now encompasses a rich taxonomy of parameter fusion strategies, most of which can be organized into the following categories:

**A. Linear and Arithmetic Merging**

The simplest approach linearly interpolates two (or more) parameter sets sharing the same architecture and initialization:
\[
\theta_{\text{merge}} = \alpha \theta_1 + (1-\alpha) \theta_2,\ \alpha\in[0,1]
\]
or, more generally, for N experts:
\[
\theta_{\text{merge}} = \sum_{i=1}^N \lambda_i \theta_i \qquad \text{with}\ \sum_{i=1}^N \lambda_i=1
\]
Task arithmetic operates in task-vector space relative to a shared base:
\[
\theta_{\text{merge}} = \theta_{\text{base}} + \sum_{i=1}^N \lambda_i (\theta_i - \theta_{\text{base}})
\]
This family underlies classic model soups, task vector addition, and layerwise averaging [2408.07666, 2510.01845, 2505.19892].

**B. Parameter-Efficient and PEFT Merging**

When specialist models are trained via parameter-efficient methods (e.g. LoRA, adapters), merging is confined to low-rank update matrices:
\[
\Delta\theta^{\rm merge} = \sum_{i=1}^N \lambda_i \Delta\theta_i
\]
CoPA-Merging introduces structured pruning of singular vectors, scaling, and cross-task normalization to maintain directionality and balance contributions of each expert under LoRA adaptation [2502.17159].

**C. Alignment and Modular Decoupling**

Methods such as DAMC [2402.12750] and MMER [2505.17110] decouple subnetworks or generate masks to isolate modality-specific parameters. MMER’s process:
- Merge fine-tuned models in the task-vector domain using sign-consistent summation (e.g., TIES merge).
- Construct binary masks $M_i$ for each modality to isolate only coordinates whose directional contribution matches the merged vector and exceeds a significance threshold.
- At inference, input for modality $i$ is routed via the corresponding mask to avoid destructive interference.

**D. Neuron and Layerwise Selective Fusion**

Locate-then-Merge and Neuron-Fusion [2505.16703] apply neuron-level selection, preserving large-shift (“vision-specialist”) neurons and suppressing diffused shifts that induce forgetting. Layer-level or blockwise merging—such as plateau-guided merging in PlaM [2601.07645]—restores base LLM weights in late stages to recover language reasoning while preserving cross-modal alignment earlier.

**E. Heterogeneous and Architecture-Aware Merging**

AdaMMS merges MLLMs with different architectures (e.g., layer duplication, additional QKV heads) by mapping, linearly interpolating corresponding tensors, and searching for optimal interpolation coefficients via unsupervised consistency on unlabeled data [2503.23733].

**F. Optimization-Based and Chunked Merging**

Expert Merging [2509.25712] and its extension, Expert Merging++, learn a small set of per-layer (or chunk) coefficients via loss alignment of hidden states/logits with the experts, regularized for stability, often using only a modest calibration set.

| Method Name         | Key Idea/Formula                                 | Notable Applications    |
|---------------------|--------------------------------------------------|------------------------|
| Linear Interp./TA   | $\theta_{\text{merge}} = \sum \lambda_i \theta_i$ | Language/vision fusion [2510.01845, 2304.14933] |
| MMER                | Masked, sign-consistent task vector merging & decoupling | Multimodal expansion/retention, catastrophic forgetting mitigation [2505.17110] |
| Neuron-Fusion       | Select large-shift neurons; suppress/restore     | Mitigate language drop in MLLMs [2505.16703] |
| AdaMMS              | Mapping-based blockwise merge + unsupervised coeff. search | Heterogeneous backbone merges [2503.23733] |
| Plateau-guided      | Restore LLM weights in plateau (late) layers     | Language retention, visual grounding [2601.07645] |
| Expert Merging(++)  | Layerwise/chunkwise coefficients via alignment   | Multi-expert LLM/MLLM [2509.25712] |

## 3. Advanced Strategies: Interference, Sparsification, and Regularization

Central challenges in multimodal merging are performance interference, gradient conflicts, and catastrophic forgetting—especially pronounced when fusing highly heterogenous tasks or modalities. Several classes of solutions have been proposed:

- **Sparsity and Masking**: EMR-Merging [2405.17461] elects a unified task vector using majority sign, per-task masks incompatible directions, and rescales magnitude such that each expert’s core features are recovered without detrimental interference.
- **Optimization-Based Losses**: OptMerge [2505.19892] denoises task vectors via low-rank projection, then minimizes an interference loss penalizing overlap in parameter-space “energy.”
- **Adaptive/Regularized Coefficients**: AMM [2510.08987] combines static task importance weights with dynamic compatibility and projection penalties, ensuring per-layer merges remain in directions aligned with specialist tasks.

A typical empirical ablation demonstrates additive improvement from sparsity, adaptive scaling, and projection penalties, especially for LoRA-based (PEFT) merges [2502.17159, 2405.17461].

## 4. Merging for Heterogeneous and Lifelong Multimodal Systems

Recent merging approaches address emerging scenarios in multimodal systems: heterogeneity in model architectures and continual or temporal integration of new modalities.

**Heterogeneous Models**: AdaMMS [2503.23733] generalizes weight-space merging to mismatched layers and modular structure by mapping corresponding modules and restricting merging to aligned components. Unmapped (extra) blocks are carried forward unaltered.

**Temporal Merging**: The TIME framework [2412.06712] analyzes continual integration, proposing “init_EMA + deploy_EMA”: experts are recursively merged via exponential moving averages, with simple weight/arithmetic merges sufficing for strong retention and adaptation across many tasks through time.

**Extensibility and Data-Free Fusion**: Composition approaches such as NaiveMC and DAMC [2402.12750] allow arbitrary extensibility: to add a new modality, merge only the LLM backbone and plug in the new encoder, with parameter decoupling to minimize interference. These strategies underpin data-free, ongoing expansion of MLLM capabilities.

## 5. Empirical Evaluations and Benchmarks

A suite of benchmarks and domain-specific metrics is standard for evaluating merged multimodal models:

- **Task Suites**: MCUB (commonality reasoning) [2402.12750], multimodal reasoning (VQA, Geometry, ChartQA, Grounding, OCR) [2505.19892], coding+vision (MMCode, InfiBench-V) [2508.09945].
- **Zero-Shot and Transfer**: Model merging approaches (MAM [2312.14378]) can enable zero-shot gains in speech/audio from pretraining on text/image, highlighting the benefits of attention transfer even when labeled data is scarce.
- **Catastrophic Forgetting**: Locate-then-Merge and MMER report near-perfect (≥99%) retention of previous unimodal abilities after merging, outperforming naive and training-based baselines [2505.16703, 2505.17110].
- **Performance Trade-offs**: Merged models can surpass individual experts in joint or unseen modalities (“complementarity”), often outperform supervised mixture-trained models in benchmark averages [2505.19892, 2509.25712].

| Model/Approach   | Benchmark                   | SOTA/Notable Results             |
|------------------|----------------------------|----------------------------------|
| VisCodex [2508.09945] | InfiBench-V, ChartMimic      | +1–5 points over no-merge, close to GPT-4o |
| OptMerge [2505.19892] | InternVL2.5, Qwen2-VL benchmarks | +1–4 pts over prior merges, matches mixture SFT |
| MMER [2505.17110]      | MCUB, MUSIC-AVQA               | +1.9 points MCUB, ≥99% retention |
| PlaM [2601.07645]  | MMStar, MMMU, GQA, MME      | +0.8–1.2 points over base |

## 6. Theoretical Principles and Open Problems

The theoretical basis of multimodal model merging predominantly rests on:
- **Linear Mode Connectivity**: Most pretrained/fine-tuned models remain in a flat basin, justifying interpolation and arithmetic in weight/task-vector space [2408.07666].
- **NTK/Disentanglement Theory**: Modality/task updates that inhabit orthogonal subspaces can be jointly merged without interference [2408.07666].
- **Empirical Indicators**: Weight distance and sign-alignment metrics can predict merge success [2304.14933].
- **Limitations & Open Questions**: Merging highly heterogeneous models (dissimilar architectures, pretraining, or data) remains challenging; robustly establishing theoretical conditions for when merged models preserve all expert skills (without retraining or data calibration) is unresolved [2408.07666].

Among outstanding challenges are:
- Data-free fusion across more than four modalities and arbitrarily large parameter spaces.
- Improved automatic selection and tuning of layerwise or chunkwise merge coefficients [2503.23733, 2509.25712].
- Robustness to adversarial interference and dynamic, input-conditional merges [2408.07666].
- Extensible approaches for architectures without aligned component mapping [2408.07666, 2503.23733].

## 7. Applications and Future Directions

Multimodal model merging is now a primary technique for:
- Training-free composition of “omnimodal” LLMs spanning vision, audio, video, code, and geometric inputs [2505.19892, 2402.12750].
- Catastrophic forgetting mitigation under sequential task additions [2505.16703, 2505.17110, 2412.06712].
- Lifelong/continual learning within decentralized and federated model repositories [2412.06712].
- Building lightweight, high-accuracy, unified reasoning models (e.g., Tiny-R1V with AMM [2510.08987]) for edge or real-time multimodal inference.

The combination of merging (for “fast path” fusion of experts), masking (for retention/decoupling), and targeted regularization (for conflict mitigation) is likely to remain foundational as the scale and scope of multimodal models expand. Benchmarks such as MCUB, InfiBench-V, and the FoMo-in-Flux suite will enable sustained, comparable empirical progress. Advances in unsupervised hyperparameter search, layer/chunk-aware merging, and dynamic/soft routers represent active research directions.

Ongoing work aims for robust, scalable, and theory-grounded model merging frameworks that can efficiently support continual multimodal expansion with minimal catastrophic forgetting and maximal synergy [2408.07666, 2601.07645, 2505.19892].

Source: https://www.emergentmind.com/topics/multimodal-model-merging