Model Merging Methods

Updated 20 April 2026

Model merging methods are algorithmic techniques that integrate multiple neural network models into a single parameter set, inheriting combined strengths without needing original training data.
Key approaches include weight averaging, sparsification, optimization-based techniques, geometric methods, and input-driven schemes, addressing both homogeneous and heterogeneous architectures.
Empirical findings show that these methods enhance resource efficiency and deployment flexibility while managing challenges such as parameter interference, alignment, and scalability.

Model merging methods comprise a diverse set of algorithmic techniques for integrating multiple pre-trained or fine-tuned neural network models into a single set of parameters, with the goal of inheriting and consolidating their capabilities, often without access to original training data or expensive retraining. This capability is critical for scenarios including multi-task learning, federated deployment, privacy-preserving model fusion, and continuous capability composition in foundation models. Approaches span from simple weight-space averaging and arithmetic to sophisticated optimization, geometric, sparsification, and input- or data-driven schemes, and increasingly address both homogeneous and heterogeneous settings across architectures and modalities.

1. Foundational Principles and Motivation

Model merging is fundamentally distinguished by direct parameter-space aggregation of models, enabling the construction of multitask or cross-domain systems under conditions where multi-task training or ensembling is impractical due to data silos, privacy, or scale constraints (Ruan et al., 12 Mar 2025). The main motivations are:

Resource efficiency: Avoidance of redundant retraining or ensembling by creating a single model.
Knowledge aggregation: Fusion of complementary task or domain expertise.
Deployment flexibility: Merged models can be deployed on edge or in federated systems.
Data privacy and accessibility: Operations on released model weights without access to proprietary datasets.

Challenges arise due to possible parameter interference (conflicting updates from different experts), permutation invariance (non-aligned neurons or layers), heterogeneous architectures, and the presence of both shared and exclusive knowledge in component models (Ruan et al., 12 Mar 2025, Lu et al., 2024).

2. Algorithmic Taxonomy and Method Classes

The model merging landscape is taxonomized along several orthogonal axes, reflecting the granularity of parameter mixing, the use of task/activation information, and the extent of architectural alignment (Ruan et al., 12 Mar 2025, Yang et al., 2024, Ruan et al., 12 Mar 2025, 2505.10833). Principal method families include:

(a) Parameter Averaging and Task Arithmetic

Simple weight averaging ("Model Soup") linearly combines expert weights: $\theta_{\mathrm{merged}} = \sum_i \alpha_i \theta_i$ , with $\sum_i \alpha_i = 1$ (2505.10833). Task arithmetic represents expert models via task vectors $\tau_i = \theta_i - \theta_0$ and composes them additively: $\theta_{\mathrm{merged}} = \theta_0 + \sum_i \lambda_i \tau_i$ (2505.10833, Yang et al., 2024).

(b) Sparsification and Conflict Mitigation

TIES-Merging prunes low-magnitude task vector entries and elects sign-consensus per coordinate, merging only non-conflicting large updates (2505.10833). DARE randomly drops entries in $\tau_i$ with scaling for the remainder (2505.10833). Breadcrumbs, DELLA, EMR and hybrid approaches apply data- or activation-driven pruning, layer-wise or neuron-wise (Ruan et al., 12 Mar 2025).

(c) Optimization-Based Methods

Fisher-weighted averaging: weights the combination by per-parameter Fisher information (2505.10833): $\theta_\mathrm{merge} = (\sum_i F_i)^{-1} \sum_i F_i \theta_i$ (with $F_i$ diagonal Fisher) (Ruan et al., 12 Mar 2025). Regression Mean (RegMean) aligns expert activations under merged weights via a data-driven least-squares procedure (Tam et al., 2023, 2505.10833). MaTS (Matching in Task Parameter Subspaces) solves a linear system reflecting each model’s principal parameter subspace, often via conjugate gradient when closed forms are infeasible (Tam et al., 2023). SuperMerge, AdaMerging, DOGE and others optimize layer-wise or per-parameter coefficient matrices via gradient descent on small validation sets, sometimes with adaptive coefficients and projected updates to retain shared knowledge (Yang et al., 2024, Wei et al., 2 Jan 2025).

(d) Geometric and Manifold-Based Approaches

Orthogonal Model Merging (OrthoMerge) merges in the space of orthogonal transformations (OFT adapters), using Riemannian mean or Cayley parameterization to avoid geometric distortion of pretrained weights (Yang et al., 5 Feb 2026). ACTMat (Data-Free Covariance Estimation) formulates merging as minimization of activation interference; it accurately estimates required covariance matrices from expert task differences without using any data (Hameed et al., 1 Apr 2026).

(e) Input- and Activation-Driven Merging

APL (Activated Parameter Locating) uses causal interventions to estimate parameter importance for task performance, guiding importance-weighted pruning (Kong et al., 2024). MERGE (Modular Expert Recombination) performs component-wise bi-objective search (accuracy vs storage), constructing libraries of modular experts and using routers for input-adaptive recombination (Qiu et al., 6 Feb 2026). Twin-Merging modularizes shared and exclusive knowledge, performs low-rank compression, and dynamically routes between shared and per-task residuals (Lu et al., 2024). ES-Merging leverages embedding space signals derived from probe inputs to estimate both layer-wise and fine-grained per-parameter merging coefficients, especially for multimodal or biological expert models (Lee et al., 15 Mar 2026).

(f) Evolutionary, Reinforcement-Learning, and Search-Based Approaches

Reinforced Model Merging (RMM) formalizes layer-wise operator selection as an MDP, training a PPO agent to compose the final model by per-layer actions, enabling non-gradient-based search with massive evaluation speedups via reward approximation (Han et al., 27 Mar 2025). FW-Merging (Frank-Wolfe) treats the set of candidate models as a convex hull, iteratively selecting relevant experts through a conditional gradient and allowing the integration of arbitrary merging subroutines (Chen et al., 16 Mar 2025).

(g) Heterogeneity-Aware Methods

AdaMMS aligns and linearly merges weights across models differing in architecture by constructing a mapping and selecting merge coefficients via unsupervised generation-consistency search; capable of merging heterogeneous MLLMs without labeled data (Du et al., 31 Mar 2025). Training-free Heterogeneous Model Merging addresses depth/width misalignments with layer segmentation and elastic neuron zipping (Xu et al., 2024).

3. Theoretical Underpinnings

The efficacy of merging is intimately tied to several geometric and functional properties:

Linear Mode Connectivity (LMC): Fine-tuned models sharing an initialization often admit low-loss linear interpolations in parameter space, supported theoretically and empirically, underpinning weight averaging and task arithmetic (Song et al., 10 Mar 2026, Yang et al., 2024).
Permutation invariance and alignment: Parameter-space symmetries, particularly neuron permutations, necessitate alignment pre-processing—by assignment (e.g., Hungarian/OT)—to avoid destructive interference (Ruan et al., 12 Mar 2025, Xu et al., 2024).
Loss landscape smoothness and subspace separation: Optimization and subspace-based approaches benefit from flat, well-connected minima and orthogonality among task subspaces (as in tangent-space merging and MA-TS) (Tam et al., 2023, Ruan et al., 12 Mar 2025).
Theoretical error bounds: Under $L$ -smoothness, merging error is quadratically bounded in the parameter-space distance between models (Song et al., 10 Mar 2026).
Interpretability of knowledge composition: Decomposition of weights into shared and exclusive components, as in Twin-Merging, is justified by observed overlaps and disjointness in task subspaces (Lu et al., 2024).

4. Empirical Results and Comparative Evaluation

Empirical comparisons across vision, natural language, and multimodal tasks demonstrate that choice of merging method is driven by data/resource availability, architectural homogeneity, and performance–efficiency trade-offs (2505.10833, Ruan et al., 12 Mar 2025, Yang et al., 2024, Qiu et al., 6 Feb 2026). Key findings include:

Method Class	Data Needed	Notable Results (examples)
Model Soup, Task Arithmetic	None	Task Arithmetic: 45.8% MedQA (med LLMs, (Timilsina et al., 17 Nov 2025))
Sparsification (TIES, DARE)	None	TIES: up to +2.5% over averaging in vision/NLP (Ruan et al., 12 Mar 2025)
Activation/causal pruning	Few-shot	APL surpasses Ties on OOD tasks with >95% parameter drop (Kong et al., 2024)
Optimization-based (Fisher, MaTS)	Validation	MaTS: 90.2% acc (ViT-L/14) vs 85.9% TIES, 81.5% (NLP full) (Tam et al., 2023)
Input-aware/dynamic routing	Few-shot	Twin-Merging: +28.3% normalized GLUE over Ties-Merging (Lu et al., 2024)
Data-free covariance (ACTMat)	None	ACTMat: 89.5% (ViT-B/16), 79.8% (T5-L) vs prior data-free baselines (Hameed et al., 1 Apr 2026)
Hetero merging (AdaMMS)	None	AdaMMS: +26.84 points over baseline on Qwen2-Lava MLLM (Du et al., 31 Mar 2025)
RL/evolution/search (RMM)	Validation	RMM: +4.1–14.6% over baseline on vision; +7–10% on NLP (Han et al., 27 Mar 2025)

In large-scale LLM settings, Localize-and-Stitch and consensus task arithmetic yield >89% normalized multitask performance on 8–9B models; data-based versions outperform dataless by 1–2% (2505.10833).

Practical recommendations are method-specific (2505.10833, Ruan et al., 12 Mar 2025, Timilsina et al., 17 Nov 2025):

Use Model Soup or Task Arithmetic for lowest compute/memory.
Prefer sparsification (TIES, dataless LAS) with minimal validation data.
Employ optimization-based or dynamic methods (e.g., MaTS, SuperMerge) if accuracy is paramount and validation data are available.
For heterogeneous or multimodal cases, apply architecture- or input-aware schemes (AdaMMS, ES-Merging, zipping).

5. Extensions for Heterogeneous and Modular Architectures

Contemporary merging research increasingly addresses heterogeneity in both architecture and task domain (Xu et al., 2024, Du et al., 31 Mar 2025, Qiu et al., 6 Feb 2026):

Layer/depth alignment: Segmentation and matching of layers when models differ in depth, enabling their fusion (Xu et al., 2024).
Elastic neuron zipping: Projection and progressive merging of mismatched-width neurons onto common dimensionality (Xu et al., 2024).
Partial or non-shared module mapping: AdaMMS defines partial mappings and unmerged parameter carryover for multimodal expert integration (Du et al., 31 Mar 2025).
Modular expert recombination (MERGE): Formulation of a bi-objective (accuracy, cost) component-wise search with offline Pareto-optimal modular library construction and batch-inference via dynamic routing at deployment (Qiu et al., 6 Feb 2026).
Twin-Merging/EMR: Extraction, compression, and reassembly of shared/exclusive knoweldge via dynamic, input-conditional routers (Lu et al., 2024).

These approaches offer scalability, reduced storage/inference cost, and robustness to input/task heterogeneity, and are increasingly required in practical multi-expert/federated/continual learning settings.

6. Limitations, Open Problems, and Future Directions

Key open challenges include (Ruan et al., 12 Mar 2025, Song et al., 10 Mar 2026, Yang et al., 2024, 2505.10833):

Task interference and performance scaling: Quality degrades as the number and diversity of merged experts increase, due to unresolved conflicts or over-pruning.
Theoretical understanding: Explanations for the boundary between successful and failed merging, especially for heterogeneous or randomly initiated models, remain incomplete.
Combinatorial and memory complexity: Alignment (e.g., permutation/OT) and coefficient optimization are expensive at extreme scale.
Modality and architecture generality: Comprehensive solutions for architecture- or domain-misaligned models are not universal.
Standardized evaluation and benchmarking: There is a recognized need for community benchmarks and tools (e.g., MergeBench, FusionBench, MergeKit) and metrics beyond in-domain accuracy, including OOD retention, interference, and cost (2505.10833, Song et al., 10 Mar 2026).
Integration with training pipelines: The use of merging as a core component rather than post-hoc recipe (e.g., within RLHF, continual learning) remains underexplored.

Planned advances include predictive compatibility metrics, continual/incremental merging frameworks, next-gen methods for cross-architecture/model property adaptation, theory for safety-certified merges, and methods for efficient merging under strict compute/memory constraints.

7. References to Representative Methods and Surveys

The following table provides a sampling of recent, representative approaches across principal categories:

Method/Framework	Key Ideas	arXiv ID
RMM (RL-based merging)	Layer-wise MDP; PPO agent	(Han et al., 27 Mar 2025)
MaTS (linear system in subspaces)	Subspace matching; CG solution	(Tam et al., 2023)
AdaMMS (heterogeneous, unsupervised)	Partial mapping; gen-consistency search	(Du et al., 31 Mar 2025)
ACTMat (data-free covariance)	Covariance from $\Delta^\top\Delta$	(Hameed et al., 1 Apr 2026)
SuperMerge (gradient-based layer weights)	Layer-wise trainable merging; $\tanh$	(Yang et al., 2024)
Twin-Merging	Shared/exclusive modularization; routing	(Lu et al., 2024)
OrthoMerge (manifold/orthogonal merge)	Lie-group averaging on O(d)	(Yang et al., 5 Feb 2026)
MERGE (modular comp-weight search)	Bi-obj. Pareto, surrogate-NSGA-II	(Qiu et al., 6 Feb 2026)

Comprehensive surveys and taxonomies: (Ruan et al., 12 Mar 2025, Song et al., 10 Mar 2026, Yang et al., 2024, 2505.10833).

Model merging is thus a central, rapidly developing paradigm in modern deep learning, straddling theoretical insights into loss landscapes and connectivity, combinatorial and geometric algorithmics, and system-level challenges in knowledge integration, privacy, and deployment. Future advances are expected to further unify theory and practice, address scalability, heterogeneity, and trustworthiness, and establish model merging as a standard tool for modular, flexible, and efficient AI systems.