---
title: Large Language Model Merging
url: https://www.emergentmind.com/topics/large-language-model-merging
type: topic
---

# Large Language Model Merging

Large language model merging is the process by which two or more large language models (LLMs), typically fine-tuned for distinct tasks or domains but sharing a backbone architecture, are combined into a single unified parameter set capable of multi-domain or multi-task inference—without requiring full retraining. The recent proliferation of specialized LLM variants and increasing focus on modular AI systems have spurred extensive methodological innovation, scaling studies, and a diverse ecosystem of tools and benchmarks. The theoretical and practical landscape of LLM merging is characterized by a spectrum of approaches, from simple weight averaging to advanced sensitivity-guided methods and gradient-based coefficient optimization, all grounded in the empirical finding that many LLM fine-tuning trajectories lie within a connected, low-loss basin of the parameter space [2603.09938], [2511.21437].

## 1. Theoretical and Empirical Foundations

The foundational insight for model merging is the linear mode connectivity of neural network loss landscapes: fine-tuned LLMs initialized from the same checkpoint often admit low-loss interpolation paths in parameter space, enabling the arithmetic blending of weights to yield viable merged solutions [2603.09938], [2511.21437]. Formally, if $\theta_0$ is a pretrained base, and $\theta_1$, $\theta_2$ are fine-tuned variants, task “delta” vectors are computed as
\[
\delta_i = \theta_i - \theta_0
\]
and merged weights by
\[
\theta_\text{merged} = \theta_0 + \sum_{i=1}^k \lambda_i \delta_i
\]
where $\lambda_i$ are mixing coefficients.

Mode connectivity manifests as empirically observed flat or nearly monotonic loss along these linear or geodesic paths in weight space [2603.09938]. This underpins the reliability of “Task Arithmetic” (uniformly summing “task deltas”) for LLMs—a result validated at both the small-model and multi-billion-parameter scale [2511.21437].

Scaling laws further quantify merging returns: merging $k$ expert models into a base of capacity $C$ yields loss
\[
L(k, C) = L_\infty(C) + \frac{A(C)}{k + b}
\]
with $L_\infty(C)$ the irreducible floor that decreases as a power law in $C$, and a $1/(k+b)$ tail expressing diminishing returns from additional experts. These laws are robust across merging methods, model families, and task heterogeneity [2509.24244].

## 2. Merging Methodologies and Algorithms

LLM merging algorithms span a range of strategies, which can be organized as follows:

- **Task Arithmetic and Averaging**: The simplest and most robust, task arithmetic (TA) applies equal weighting to each task’s parameter delta [2603.09938], [2511.21437]. Variants include Fisher-weighted averaging and trajectory (Stochastic Weight Averaging, SWA) approaches.

- **Sparsification and Interference-aware Schemes**: To mitigate destructive interference among conflicting task deltas, methods like TIES-Merging (Trim-Elect-Sign) and DARE (Drop-And-REscale) sparsify or mask deltas by magnitude trimming, random drop, and sign consensus [2509.24244], [2511.21437]. However, large-scale benchmarking finds these approaches often degrade performance compared to TA when many diverse, strongly fine-tuned checkpoints are merged [2511.21437].

- **Subspace, Low-rank, and SVD-based Merging**: Singular vector decomposition (TSV-Merge, Iso-C, SB) restricts updates to low-rank or orthogonalized subspaces, motivated by the hypothesis of disjoint task subspaces. These techniques have produced reliable gains in smaller models and classifiers but consistently underperform compared to TA in modern LLMs [2511.21437].

- **Sensitivity-guided and Activation-informed Merging**: Recent advancements use parameter gradient sensitivities [2502.12420] or activation-space statistics [2502.02421] to assign layer/task-specific mixing coefficients, preserving critical weights and improving task retention, especially when combined with classical delta-based methods.

- **Gradient-based Merge Coefficient Optimization**: SuperMerge introduces layer/task-specific, learnable merging weights, updating only $\mathcal{O}(k \cdot n_\text{layers})$ parameters using validation sets [2412.10416]. This method achieves merged accuracy rivaling fully fine-tuned and multi-task models at minimal computational cost.

- **Distribution-based and Output-level Merging**: The MoD (Mixture of Distributions) framework merges models at the output distribution level, forming a convex mixture of next-token probabilities and linearly combining the logits. This approach is architecture-agnostic and strong on specialist retention, especially in domains like mathematical reasoning [2411.00406].

- **Cross-Architecture Merging**: When merging heterogeneous or multimodal models, mapping schemes (e.g., AdaMMS) or OT-based neuron alignment (Transport & Merge) infer cross-model correspondences and interpolate mapped weights [2503.23733], [2602.05495], enabling knowledge transfer without requiring shared layer structure.

The majority of real-world merging is performed at the level of adapter weights (e.g., LoRA), allowing modularity, memory efficiency, and direct deployment in compositional scenarios [2601.16127].

## 3. Practical Applications and Empirical Outcomes

Large language model merging delivers:

- **Multi-task and Multi-domain Generalization**: By judiciously merging checkpoints fine-tuned for distinct capabilities (instruction, code, math, multilingual, safety), LLMs can approach or surpass multi-task fine-tuned baselines without additional gradient steps [2506.12379], [2511.21437].

- **Alignment and Safety**: Parameter-level merging of alignment-tuned models (helpfulness, honesty, harmlessness) achieves superior trade-offs compared to data-level mixture, particularly under conflicting objectives. The RESM algorithm further enhances balanced alignment via outlier-aware SVD weighting and sparsity-adaptive truncation [2502.06876].

- **Efficiency and Maintenance**: Merging substantially reduces retraining and maintenance costs in multilingual and multi-task setups. For example, adapter-based language-merge pipelines cut initial fine-tuning time by up to 50% and maintenance time/cost by 60–70% compared to full retrain [2601.16127].

- **Pretraining and Model Recovery**: Checkpoint merging at the pretraining stage (PMA) can emulate the effects of cosine learning-rate decay, stabilize optimization, and recover from loss spikes, resulting in higher final test accuracy and faster tuning [2505.12082].

- **Personality and Attribute Modulation**: Weight-space arithmetic over “personality vectors” extracted by trait-specific fine-tuning grants continuous, compositional, and cross-domain control over LLM personality traits [2509.19727].

- **Compression and On-the-fly Routing**: 1bit-Merging fuses dynamic routing with bitwise-compressed task vectors, balancing performance and storage efficiency for memory-limited deployments [2502.10743].

- **Federated and Continual Learning**: Model merging underpins branch-train-merge strategies, federated aggregation (FedAvg and beyond), and streaming incremental expert integration [2603.09938].

## 4. Limitations, Scaling Laws, and Failure Modes

Extensive empirical studies reveal that, for modern LLMs:

- **Task Arithmetic is Uniquely Reliable**: Only the oldest and simplest delta arithmetic reliably yields merged models that consistently outperform both the base and individual experts as the number of merged checkpoints grows [2511.21437]. More complex methods often damage performance, especially when merging highly divergent or strongly fine-tuned models.

- **Law of Diminishing Returns**: Merging gains rapidly saturate—most of the benefit accrues within the first 5–6 experts, with marginal returns decaying as $1/(k+b)$. Cross-domain and in-domain merges both obey this floor-plus-tail scaling law, which enables predictive planning, budget optimization, and principled stopping criteria [2509.24244].

- **Model Kinship and Merging Gains**: The “kinship” (similarity) between task deltas predicts merge gains: low-kinship (more orthogonal) pairings yield larger gains and enable escape from local optima in iterative, top-$k$ greedy merges. High-kinship merges quickly saturate and further merges add negligible value [2410.12613].

- **Interference and Over-pruning**: Aggressive sparsification, subspace, or orthogonalization methods—premised on strong independence assumptions—fail when actual task deltas overlap in substantial parameter subspaces. Monitoring the $\ell_2$ displacement of merged weights from the base is critical to ensure merges remain within the model’s low-loss region [2511.21437].

- **Complex, Strongly Fine-tuned Deltas**: When deltas have large, module-specific amplitudes (common with strong domain specialists), dynamic pruning and amplification (DPPA) can yield superior fusions at high sparsity [2403.02799].

## 5. Benchmarks, Ecosystem, and Best Practices

A mature merging ecosystem includes:

- **Toolkits**: MergeKit (Arcee), FusionBench, and similar libraries standardize delta arithmetic, sparsification, and evaluation pipelines, lowering barriers for practical merging [2603.09938].

- **Benchmarks**: Platforms such as the Open LLM Leaderboard and standardized suites (MMLU, GSM8K, HumanEval, TruthfulQA, FusionBench) allow rigorous assessment of merged model capability, retention rate, and interference matrices [2511.21437].

- **Calibration Protocols**: Even simple merging benefits from post-hoc validation, adaptive coefficient selection (Sens-Merging, activation-informed merging), and, in complex scenarios, supervised or unsupervised hyperparameter tuning (e.g., AdaMMS, SuperMerge) [2502.12420], [2503.23733], [2412.10416].

- **Guidelines**: Empirically, practitioners are advised to default to task arithmetic with modest delta norms, avoid merging highly similar or highly divergent models without calibration, and to exploit activation-space or sensitivity signals for challenging merges [2511.21437], [2502.02421].

## 6. Open Challenges and Future Directions

While significant progress has been achieved, several challenges and research frontiers remain:

- **Theory**: General theoretical explanations for the prevalence of linear mode connectivity and the generalization properties of merged billion-parameter LLMs are lacking [2603.09938].

- **Scalability**: As model size and the number of experts increase (>100B parameters, $k > 10$), both search and alignment costs rise super-linearly. Advanced approximation techniques, evolutionary search, and automated coefficient learning are required.

- **Heterogeneous and Modular Merging**: Cross-architecture and multimodal merging—where models differ in structural details—are active areas, with OT-based correspondence and mapping techniques showing early promise [2602.05495], [2503.23733].

- **Safety and Security**: Merging exposes new attack surfaces, such as “merge hijacking” via malicious backdoored models. Detection, certification, and adversarially robust merging objectives remain underexplored.

- **Standardization and Benchmarks**: Protocols for interference/error reporting, emergent composite capability, and safety under merging are yet to be standardized.

- **Continual and Federated Merging**: Automation of streaming expert integration and merging-aware fine-tuning regimes is an open engineering challenge for ongoing deployments [2603.09938].

In sum, large language model merging recasts the development of advanced LLM systems from monolithic retraining to modular compositional assembly. Its continued evolution will depend on both methodological innovation and deeper understanding of neural loss geometry, interference dynamics, and task transfer in high-dimensional spaces.

Source: https://www.emergentmind.com/topics/large-language-model-merging