---
title: Model Merging in Neural Networks
url: https://www.emergentmind.com/topics/model-merging
type: topic
---

# Model Merging in Neural Networks

Model merging is the process of combining several fine-tuned neural network models—each specialized on different tasks or domains—into a single multi-task model, typically without requiring retraining on the original data. The primary objective is to aggregate expertise efficiently, reduce storage and inference costs compared to ensembling, and enable cross-domain generalization. Model merging underpins a broad range of scenarios, from multi-task LLMs to multimodal foundation models in vision, language, and beyond.

## 1. Mathematical Foundations and Theoretical Guarantees

Model merging typically assumes a shared pre-trained backbone $f_0$ and $T$ fine-tuned variants $f_1,\dots,f_T$ whose parameters are $\theta_0,\,\theta_1,...,\theta_T$. The merging problem is formalized as searching for a set of weights $w=(w_1,\dots,w_T)$ such that the merged model $f_{\rm merge}(x) = f_0(x) + \sum_{t=1}^T w_t \, r_t(x)$ matches each $f_t$'s outputs as closely as possible, where $r_t(x) = f_t(x) - f_0(x)$ is the pointwise residual [2605.29101].

A key advance is casting merge-weight selection as a convex quadratic program (QP) over the residuals. The squared-output calibration objective is:
\[
\min_{w \in \mathbb{R}^T} \sum_{i=1}^N \left\|f_0(x_i) + \sum_{t=1}^T w_t r_t(x_i) - f_t(x_i) \right\|_2^2 + \lambda\|w\|_2^2,
\]
where $\{x_i\}_{i=1}^N$ is a calibration set (labels optional). The solution $w^*$ is global and unique due to convexity. This QP subsumes and generalizes many heuristic merges such as model soups (uniform averaging), task arithmetic, TIES, and DARE as special cases [2605.29101].

Further theoretical tools include projection diagnostics. The fraction of residual energy captured by a chosen basis, $\gamma = \frac{\mathrm{tr}(S P)}{\mathrm{tr}(S)}$, with $S$ the total residual-energy matrix, quantifies how much of the multi-task variation is accessible to the merge and robustly predicts merge success [2605.29101].

## 2. Merging Algorithms: Taxonomy and Methodological Advances

A diverse algorithmic ecosystem has crystallized, with methods grouped as follows:

- **Weight-Space Averaging:** Uniform or greedy averaging of checkpoints (“model soups”) is effective when all models lie in the same contiguous basin of the loss landscape but degrades if fine-tuned directions diverge [2603.09938].
- **Task Vector Arithmetic:** Merges are performed by summing and scaling task vectors $\tau_t = \theta_t - \theta_0$, supporting operations such as addition, negation, and scaling [2603.09938, 2605.29101].
- **Sparsification-Enhanced Merging:** Methods like TIES enforce parameter or sign sparsity to promote consensus, while DARE applies random drop-out regularization to task vectors, often improving interference control [2605.29101, 2603.09938].
- **Optimal Transport and Alignment:** Permutation matching across neurons or optimal transport over parameter alignments remedies architectural symmetries and parameter entanglement [2604.27155].
- **Geometric and Manifold Approaches:** Fréchet averaging defines merges via geodesic means on a Riemannian or quotient manifold, yielding algorithms invariant to architectural symmetries (e.g., LoRA adapters as points on a gauge space) [2604.27155].
- **Stochastic and Search-Based Approaches:** Mixup Model Merge (M$^3$) draws random interpolation ratios from a Beta distribution, exploring nontrivial merges unattainable by fixed $\lambda$ [2502.15434]. RL-based and evolutionary methods (e.g., Reinforced Model Merging) treat merge configuration search as a Markov decision process [2503.21272].

The selection of merging method and its optimization hyperparameters is commonly tied to assumptions about the geometry of the loss landscape (linear mode connectivity, basin widths), the independence or correlation among fine-tuned updates, and the nature of the task vectors [2603.09938, 2605.29101, 2601.06672].

## 3. Extensions: Multi-Layer, Modular, and Dynamic Merging Strategies

Layer-wise and modular merging workflows extend merging flexibility and performance:

- **Sequential Layer-wise Merging:** For deep architectures, independent QPs are solved per layer in a greedy sequence. At each $\ell$, compute residuals $\{\delta_t^{(\ell)}\}$, solve the local QP, and update only layer $\ell$ [2605.29101]. This approach addresses non-convex interactions across layers and is highly effective when only the final few layers are fine-tuned.
- **Component-Wise and Modular Recombinations:** Fine-grained merging decomposes models into submodules—e.g., attention, MLP, normalization layers—and searches for optimal groupings or expert recombination patterns, often using Pareto-front optimization to trade off performance vs. storage [2602.06552].
- **Dynamic and Input-Conditional Methods:** Methods such as SE-Merging adapt merge weights dynamically for each test sample using representation similarity, yielding per-input merged models that exhibit both task separation and instance-wise adaptation without further training [2506.18135]. Slim dynamic frameworks (e.g., DiDi-Merging) leverage differentiable rank allocation in low-rank modules, balancing shared and expert parameters for aggressive storage reduction [2605.18904].

Task heterogeneity and module-level differences in mergeability necessitate modular or dynamic approaches for efficient and scalable multi-task deployment [2602.06552, 2605.18904, 2409.19173].

## 4. Theoretical and Empirical Limits of Mergeability

Rigorous analyses have identified concrete limits on mergeability:

- **Upper Bound on Experts:** The total number of experts meaningfully merged is bounded by the effective parameter space and their mutual correlation. For $n$ experts with pairwise correlation $\rho$, variance reduction under uniform merging saturates at $\sigma^2 \rho$ as $n\to\infty$; thus, marginal benefits diminish strictly as a function of Gaussian width [2505.21226].
- **Diminishing Returns:** Performance gains from adding experts are concave due to geometric constraints of the loss basin, and heavy correlation among task vectors rapidly saturates improvement [2505.21226].
- **Accuracy-Aware Weighted Merging:** Mergeability correlates strongly with the base model’s prior knowledge; knowledge that is easily accessible to the base merges more robustly. Weights should be modulated based on task familiarity to prevent rare or weak tasks from being overwhelmed in the merged model [2601.06672].

Overaggressive merging risks performance collapse due to interference, and monitoring metrics such as the marginal reduction in variance or energy-capture ratio is necessary to identify the optimal merging point [2505.21226, 2605.29101].

## 5. Geometry, Symmetry, and Invariance in Model Merging

Naïve parameter averaging fails in the presence of architectural symmetries (e.g., neuron permutation, LoRA gauge). Fréchet averaging on manifolds provides a symmetry-invariant, geometry-aware solution [2604.27155]. For low-rank adapters, alignment and averaging must respect the quotient geometry induced by invertible gauge groups, necessitating specialized algorithms (e.g., GeoMerge with Stiefel and SPD metrics).

The choice of geometry (Euclidean, Fisher–Rao, product manifolds) fundamentally shapes the feasible merge space and determines whether statistical or architectural pathologies can be avoided [2604.27155]. Empirically, symmetry-aware merges reliably outperform parameter-space heuristics, especially in highly-adapted or large-scale domains.

## 6. Practical Considerations, Scalability, and Deployment

Model merging is attractive due to computational and practical efficiencies. Modern methods can:

- Avoid retraining or require only minimal unlabeled calibration data (e.g., 100-shot set).
- Be implemented in a training-free (data-free) fashion by analytically estimating covariance via difference matrices [2604.01329] or leveraging statistical alignment between activations and weight updates [2605.12843].
- Scale to tens of models and hundreds of millions of parameters with favorable computational costs, e.g., $O(Td^3)$ per layer for data-free covariance approaches, or $O(Nd)$ memory for Frank–Wolfe scaling [2604.01329, 2503.12649].
- Integrate preference-aware and multi-objective optimization to present users with a Pareto set of trade-off solutions accommodating application-specific priorities (e.g., high accuracy on selected tasks, bounded storage) [2408.12105].

Dynamic routing, evolutionary search, and Bayesian hyperparameter coordination are common tools for practical hyperparameter-free deployment, as is open-source tool support (e.g., MergeKit) [2603.09938].

## 7. Benchmarks and Empirical Performance

State-of-the-art methods are benchmarked across vision and language domains:

- **Vision:** On ViT-B/32, the QP-based merge matches or exceeds the performance of all competitors, especially in the challenging multi-task regime [2605.29101, 2605.12843].
- **Language:** LLaMA-based fusions for instruction-following, coding, and math show that output-space QP, BMM, and geo-aware methods yield largest accuracy and robustness improvements; M$^3$ enhances OOD and adversarial performance across all baseline merges [2502.15434, 2605.12843].
- **Efficiency:** Modern merging frameworks often achieve accuracy retention of 98–99% of individual expert models with as little as 1.24x parameter overhead—including in dynamic and storage-constrained settings [2605.18904].
- **Multi-objective:** Pareto Merging methods deliver user-controllable trade-offs across all tasks, with a single run generating the spectrum of compromise solutions [2408.12105].

The collective findings demonstrate that principled model merging—rooted in convex optimization, geometric invariance, and efficient algorithmic design—enables multi-expert composition previously only practical by computationally costly or data-prohibitive methods.

---

**References:**
- "Model Merging by Output-Space Projection" [2605.29101]
- "Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation" [2502.15434]
- "Generalizing the Geometry of Model Merging Through Frechet Averages" [2604.27155]
- "Why Do More Experts Fail? A Theoretical Analysis of Model Merging" [2505.21226]
- "Will it Merge? On The Causes of Model Mergeability" [2601.06672]
- "Bayesian Model Merging" [2605.12843]
- "Dynamic Model Merging Made Slim" [2605.18904]
- "SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging" [2506.18135]
- "Fine-Grained Model Merging via Modular Expert Recombination" [2602.06552]
- "Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging" [2408.12105]
- "Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions" [2603.09938]
- "Model Merging via Data-Free Covariance Estimation" [2604.01329]
- "FW-Merging: Scaling Model Merging with Frank-Wolfe Optimization" [2503.12649]
- "Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking" [2505.23117]
- "From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches" [2503.08998]

Source: https://www.emergentmind.com/topics/model-merging