---
title: Model Merging Methods
url: https://www.emergentmind.com/topics/model-merging-methods
type: topic
---

# Model Merging Methods

Model merging methods comprise a diverse set of algorithmic techniques for integrating multiple pre-trained or fine-tuned neural network models into a single set of parameters, with the goal of inheriting and consolidating their capabilities, often without access to original training data or expensive retraining. This capability is critical for scenarios including multi-task learning, federated deployment, privacy-preserving model fusion, and continuous capability composition in foundation models. Approaches span from simple weight-space averaging and arithmetic to sophisticated optimization, geometric, sparsification, and input- or data-driven schemes, and increasingly address both homogeneous and heterogeneous settings across architectures and modalities.

## 1. Foundational Principles and Motivation

Model merging is fundamentally distinguished by direct parameter-space aggregation of models, enabling the construction of multitask or cross-domain systems under conditions where multi-task training or ensembling is impractical due to data silos, privacy, or scale constraints [2503.08998]. The main motivations are:

- **Resource efficiency**: Avoidance of redundant retraining or ensembling by creating a single model.
- **Knowledge aggregation**: Fusion of complementary task or domain expertise.
- **Deployment flexibility**: Merged models can be deployed on edge or in federated systems.
- **Data privacy and accessibility**: Operations on released model weights without access to proprietary datasets.

Challenges arise due to possible **parameter interference** (conflicting updates from different experts), **permutation invariance** (non-aligned neurons or layers), **heterogeneous architectures**, and the presence of both **shared** and **exclusive** knowledge in component models [2503.08998, 2406.15479].

## 2. Algorithmic Taxonomy and Method Classes

The model merging landscape is taxonomized along several orthogonal axes, reflecting the granularity of parameter mixing, the use of task/activation information, and the extent of architectural alignment [2503.08998, 2408.07666, 2503.08998, 2505.10833]. Principal method families include:

### (a) Parameter Averaging and Task Arithmetic
Simple **weight averaging** ("Model Soup") linearly combines expert weights:  
$\theta_{\mathrm{merged}} = \sum_i \alpha_i \theta_i$, with $\sum_i \alpha_i = 1$ [2505.10833].  
**Task arithmetic** represents expert models via task vectors $\tau_i = \theta_i - \theta_0$ and composes them additively:  
$\theta_{\mathrm{merged}} = \theta_0 + \sum_i \lambda_i \tau_i$ [2505.10833, 2408.07666].

### (b) Sparsification and Conflict Mitigation
**TIES-Merging** prunes low-magnitude task vector entries and elects sign-consensus per coordinate, merging only non-conflicting large updates [2505.10833].  
**DARE** randomly drops entries in $\tau_i$ with scaling for the remainder [2505.10833].  
**Breadcrumbs, DELLA, EMR** and hybrid approaches apply data- or activation-driven pruning, layer-wise or neuron-wise [2503.08998].

### (c) Optimization-Based Methods
**Fisher-weighted averaging**: weights the combination by per-parameter Fisher information [2505.10833]:  
$\theta_\mathrm{merge} = (\sum_i F_i)^{-1} \sum_i F_i \theta_i$ (with $F_i$ diagonal Fisher) [2503.08998].  
**Regression Mean (RegMean)** aligns expert activations under merged weights via a data-driven least-squares procedure [2312.04339, 2505.10833].  
**MaTS** (Matching in Task Parameter Subspaces) solves a linear system reflecting each model’s principal parameter subspace, often via conjugate gradient when closed forms are infeasible [2312.04339].  
**SuperMerge, AdaMerging, DOGE** and others optimize layer-wise or per-parameter coefficient matrices via gradient descent on small validation sets, sometimes with adaptive coefficients and projected updates to retain shared knowledge [2412.10416, 2501.01230].

### (d) Geometric and Manifold-Based Approaches
**Orthogonal Model Merging (OrthoMerge)** merges in the space of orthogonal transformations (OFT adapters), using Riemannian mean or Cayley parameterization to avoid geometric distortion of pretrained weights [2602.05943].  
**ACTMat** (Data-Free Covariance Estimation) formulates merging as minimization of activation interference; it accurately estimates required covariance matrices from expert task differences without using any data [2604.01329].

### (e) Input- and Activation-Driven Merging
**APL (Activated Parameter Locating)** uses causal interventions to estimate parameter importance for task performance, guiding importance-weighted pruning [2408.09485].  
**MERGE (Modular Expert Recombination)** performs component-wise bi-objective search (accuracy vs storage), constructing libraries of modular experts and using routers for input-adaptive recombination [2602.06552].  
**Twin-Merging** modularizes shared and exclusive knowledge, performs low-rank compression, and dynamically routes between shared and per-task residuals [2406.15479].  
**ES-Merging** leverages embedding space signals derived from probe inputs to estimate both layer-wise and fine-grained per-parameter merging coefficients, especially for multimodal or biological expert models [2603.14405].

### (f) Evolutionary, Reinforcement-Learning, and Search-Based Approaches
**Reinforced Model Merging (RMM)** formalizes layer-wise operator selection as an MDP, training a PPO agent to compose the final model by per-layer actions, enabling non-gradient-based search with massive evaluation speedups via reward approximation [2503.21272].  
**FW-Merging** (Frank-Wolfe) treats the set of candidate models as a convex hull, iteratively selecting relevant experts through a conditional gradient and allowing the integration of arbitrary merging subroutines [2503.12649].

### (g) Heterogeneity-Aware Methods
**AdaMMS** aligns and linearly merges weights across models differing in architecture by constructing a mapping and selecting merge coefficients via unsupervised generation-consistency search; capable of merging heterogeneous MLLMs without labeled data [2503.23733].  
**Training-free Heterogeneous Model Merging** addresses depth/width misalignments with layer segmentation and elastic neuron zipping [2501.00061].

## 3. Theoretical Underpinnings

The efficacy of merging is intimately tied to several geometric and functional properties:

- **Linear Mode Connectivity (LMC)**: Fine-tuned models sharing an initialization often admit low-loss linear interpolations in parameter space, supported theoretically and empirically, underpinning weight averaging and task arithmetic [2603.09938, 2408.07666].
- **Permutation invariance and alignment**: Parameter-space symmetries, particularly neuron permutations, necessitate alignment pre-processing—by assignment (e.g., Hungarian/OT)—to avoid destructive interference [2503.08998, 2501.00061].
- **Loss landscape smoothness and subspace separation**: Optimization and subspace-based approaches benefit from flat, well-connected minima and orthogonality among task subspaces (as in tangent-space merging and MA-TS) [2312.04339, 2503.08998].
- **Theoretical error bounds**: Under $L$-smoothness, merging error is quadratically bounded in the parameter-space distance between models [2603.09938].
- **Interpretability of knowledge composition**: Decomposition of weights into shared and exclusive components, as in Twin-Merging, is justified by observed overlaps and disjointness in task subspaces [2406.15479].

## 4. Empirical Results and Comparative Evaluation

Empirical comparisons across vision, natural language, and multimodal tasks demonstrate that choice of merging method is driven by data/resource availability, architectural homogeneity, and performance–efficiency trade-offs [2505.10833, 2503.08998, 2412.10416, 2602.06552]. Key findings include:

| Method Class                 | Data Needed | Notable Results (examples)                                            |
|------------------------------|-------------|----------------------------------------------------------------------|
| Model Soup, Task Arithmetic  | None        | Task Arithmetic: 45.8% MedQA (med LLMs, 2511.13373)                 |
| Sparsification (TIES, DARE)  | None        | TIES: up to +2.5% over averaging in vision/NLP [2503.08998]         |
| Activation/causal pruning    | Few-shot    | APL surpasses Ties on OOD tasks with >95% parameter drop [2408.09485] |
| Optimization-based (Fisher, MaTS) | Validation | MaTS: 90.2% acc (ViT-L/14) vs 85.9% TIES, 81.5% (NLP full) [2312.04339] |
| Input-aware/dynamic routing  | Few-shot    | Twin-Merging: +28.3% normalized GLUE over Ties-Merging [2406.15479]  |
| Data-free covariance (ACTMat) | None       | ACTMat: 89.5% (ViT-B/16), 79.8% (T5-L) vs prior data-free baselines [2604.01329] |
| Hetero merging (AdaMMS)      | None        | AdaMMS: +26.84 points over baseline on Qwen2-Lava MLLM [2503.23733]  |
| RL/evolution/search (RMM)    | Validation | RMM: +4.1–14.6% over baseline on vision; +7–10% on NLP [2503.21272]  |

In large-scale LLM settings, Localize-and-Stitch and consensus task arithmetic yield >89% normalized multitask performance on 8–9B models; data-based versions outperform dataless by 1–2% [2505.10833].

Practical recommendations are method-specific [2505.10833, 2503.08998, 2511.13373]:
- Use Model Soup or Task Arithmetic for lowest compute/memory.
- Prefer sparsification (TIES, dataless LAS) with minimal validation data.
- Employ optimization-based or dynamic methods (e.g., MaTS, SuperMerge) if accuracy is paramount and validation data are available.
- For heterogeneous or multimodal cases, apply architecture- or input-aware schemes (AdaMMS, ES-Merging, zipping).

## 5. Extensions for Heterogeneous and Modular Architectures

Contemporary merging research increasingly addresses heterogeneity in both architecture and task domain [2501.00061, 2503.23733, 2602.06552]:

- **Layer/depth alignment**: Segmentation and matching of layers when models differ in depth, enabling their fusion [2501.00061].
- **Elastic neuron zipping**: Projection and progressive merging of mismatched-width neurons onto common dimensionality [2501.00061].
- **Partial or non-shared module mapping**: AdaMMS defines partial mappings and unmerged parameter carryover for multimodal expert integration [2503.23733].
- **Modular expert recombination (MERGE)**: Formulation of a bi-objective (accuracy, cost) component-wise search with offline Pareto-optimal modular library construction and batch-inference via dynamic routing at deployment [2602.06552].
- **Twin-Merging/EMR**: Extraction, compression, and reassembly of shared/exclusive knoweldge via dynamic, input-conditional routers [2406.15479].

These approaches offer scalability, reduced storage/inference cost, and robustness to input/task heterogeneity, and are increasingly required in practical multi-expert/federated/continual learning settings.

## 6. Limitations, Open Problems, and Future Directions

Key open challenges include [2503.08998, 2603.09938, 2408.07666, 2505.10833]:

- **Task interference and performance scaling**: Quality degrades as the number and diversity of merged experts increase, due to unresolved conflicts or over-pruning.
- **Theoretical understanding**: Explanations for the boundary between successful and failed merging, especially for heterogeneous or randomly initiated models, remain incomplete.
- **Combinatorial and memory complexity**: Alignment (e.g., permutation/OT) and coefficient optimization are expensive at extreme scale.
- **Modality and architecture generality**: Comprehensive solutions for architecture- or domain-misaligned models are not universal.
- **Standardized evaluation and benchmarking**: There is a recognized need for community benchmarks and tools (e.g., MergeBench, FusionBench, MergeKit) and metrics beyond in-domain accuracy, including OOD retention, interference, and cost [2505.10833, 2603.09938].
- **Integration with training pipelines**: The use of merging as a core component rather than post-hoc recipe (e.g., within RLHF, continual learning) remains underexplored.

Planned advances include predictive compatibility metrics, continual/incremental merging frameworks, next-gen methods for cross-architecture/model property adaptation, theory for safety-certified merges, and methods for efficient merging under strict compute/memory constraints.

## 7. References to Representative Methods and Surveys

The following table provides a sampling of recent, representative approaches across principal categories:

| Method/Framework                        | Key Ideas                               | arXiv ID         |
|-----------------------------------------|-----------------------------------------|------------------|
| RMM (RL-based merging)                  | Layer-wise MDP; PPO agent               | 2503.21272       |
| MaTS (linear system in subspaces)       | Subspace matching; CG solution          | 2312.04339       |
| AdaMMS (heterogeneous, unsupervised)    | Partial mapping; gen-consistency search | 2503.23733       |
| ACTMat (data-free covariance)           | Covariance from $\Delta^\top\Delta$     | 2604.01329       |
| SuperMerge (gradient-based layer weights)| Layer-wise trainable merging; $\tanh$   | 2412.10416       |
| Twin-Merging                            | Shared/exclusive modularization; routing| 2406.15479       |
| OrthoMerge (manifold/orthogonal merge)  | Lie-group averaging on O(d)             | 2602.05943       |
| MERGE (modular comp-weight search)      | Bi-obj. Pareto, surrogate-NSGA-II       | 2602.06552       |

Comprehensive surveys and taxonomies: [2503.08998], [2603.09938], [2408.07666], [2505.10833].

---

Model merging is thus a central, rapidly developing paradigm in modern deep learning, straddling theoretical insights into loss landscapes and connectivity, combinatorial and geometric algorithmics, and system-level challenges in knowledge integration, privacy, and deployment. Future advances are expected to further unify theory and practice, address scalability, heterogeneity, and trustworthiness, and establish model merging as a standard tool for modular, flexible, and efficient AI systems.

Source: https://www.emergentmind.com/topics/model-merging-methods