MergeKit: Toolkit for LLM Parameter Merging

• MergeKit is an open-source toolkit that enables parameter-level merging of large language models to synthesize composite capabilities and enhance multitask performance.
• It systematizes static merging algorithms like linear averaging, task arithmetic, and SLERP to combine specialized checkpoint deltas with reproducibility and efficiency.
• MergeKit supports scalable experiments via YAML configuration, out-of-core tensor loading, and integration with Hugging Face, underpinning its research and deployment utility.

MergeKit is an open-source toolkit and software infrastructure for merging LLMs and related deep learning models at the parameter level. It systematizes methodologies for combining pretrained or fine-tuned model checkpoints, allowing researchers and practitioners to synthesize models with composite capabilities, enhance multitask competence, and facilitate experimentation with a broad array of merging algorithms. MergeKit occupies a central role in the open-weight model ecosystem, underpinning the reproducible, efficient, and scalable application of model merging in both research and downstream deployment contexts (Goddard et al., 2024, Yadav et al., 2024).

1. Core Principles and Rationale for Model Merging

Model merging refers to the direct synthesis of a new model by parameter-wise combination of two or more pretrained or fine-tuned models—typically from the same base initialization—without additional gradient optimization. The central motivation is to enable construction of multitask or domain-adapted models by reusing and aggregating the strengths of specialized expert checkpoints, thus addressing several prominent challenges:

• Resource efficiency: Merging obviates the need for expensive retraining and leverages existing open checkpoints, which is especially impactful given the high costs of training frontier-scale LLMs. For example, training Mistral-7B from scratch is cited at $2–3$ million USD (Goddard et al., 2024).
• Catastrophic forgetting avoidance: Parameter merging seeks to combine new specialties (from task/domain fine-tunes) with retained general performance, mitigating the degradation observed in further fine-tuning.
• Multitask and continual learning: By systematically aggregating task-specific deltas (task vectors), merged models can inherit diverse competencies and serve as robust initializations for continual domain adaptation.
• Modular extension: Merging is often the only practical way to combine models when original data is unavailable or unsuited for joint supervised training.

The operational assumption in MergeKit-style workflows is the availability of model checkpoints with compatible architectures and shared initializations—conditions commonly satisfied in the open-source ecosystem (Goddard et al., 2024, Yadav et al., 2024).

2. Supported Merging Algorithms and Methodological Taxonomy

MergeKit implements and generalizes a spectrum of parameter-space merging strategies, most of which fall into the category of static, training-free aggregation. The principal families include:

• Linear weight averaging: Component-wise interpolation between models, e.g., $$\theta_{\text{merge}} = \alpha\theta_1 + (1-\alpha)\theta_2$$. Used extensively for "model soups."
• Task arithmetic: Merging via addition and scaling of task vectors, $$\Delta\theta = \theta_{\text{task}} - \theta_{\text{base}}$$, with arbitrary scaling and summation.
• SLERP (Spherical Linear intERPolation): Interpolation on a geodesic in parameter space (unit sphere), e.g., $\text{SLERP}(\theta_1,\theta_2, t)$, empirically shown to sometimes traverse lower-loss regions than linear paths (Goddard et al., 2024).
• TIES (TrIm-Elect-Sign) and DARE (Drop and REscale): Variants that prune, sign-align, and combine task deltas to minimize destructive interference, supporting competitive multi-model merges.
• Fisher-weighted averaging: Combining models with weights informed by per-parameter Fisher information, requiring task data.
• Algorithmic extensions: The taxonomy in (Yadav et al., 2024) also recognizes merging via permutation alignment (e.g., Git Re-Basin), optimal transport, and other advanced strategies, though these are less universally supported.

The toolkit operationalizes these recipes through YAML configuration files, abstracting technical details and enabling reproducibility (Goddard et al., 2024). MergeKit does not natively implement adaptive, data-driven routing or inference-time expert selection; its focus is parameter aggregation as a downstream (post-expert-training) operation (Yadav et al., 2024).

3. Practical Applications and Empirical Impact

MergeKit enables, documents, and underpins diverse real-world merging use cases, including:

• Base-versus-expert interpolation: As in the Hala Arabic-centric instruction models, where fine-tuned Arabic-specialized checkpoints are slerp-interpolated with their base at $t=0.5$, recovering general capability while enhancing target-language performance. Gains of up to $+7.6$ on Arabic benchmarks are reported for the 350M parameter model, with consistent improvements at all scales (Hammoud et al., 17 Sep 2025).
• Meta-ability alignment in reasoning models: Weighted parameter averaging (e.g., $$\Theta_{\text{merge}} = 1.0\Theta^{(d)} + 0.2\Theta^{(i)} + 0.1\Theta^{(a)}$$) is used to consolidate deduction, induction, and abduction specialists, improving math average scores by $+2.2$ (7B) and $+4.4$ (32B) over baselines (Hu et al., 15 May 2025).
• Code-mixed multilingual adaptation: MergeKit-based task arithmetic and TIES are used to merge code-mix-adapted checkpoints with base multilingual models, outperforming both full fine-tuning and CPT by $1$–$$\theta_{\text{merge}} = \alpha\theta_1 + (1-\alpha)\theta_2$$0 F1 on code-mixed classification tasks (Kodali et al., 22 Oct 2025).
• Model competition leaderboard impact: Models merged via MergeKit have constituted 20–34% of top-performing open-source leaderboard entries in several size categories (Goddard et al., 2024).

These applications demonstrate robust effectiveness across tasks—machine translation, reasoning, code-mixed NLP, and instruction following—validating the generality and practical utility of the parameter merging paradigm.

4. Toolkit Architecture and Ecosystem Integration

MergeKit is designed for extensibility, efficiency, and broad ecosystem compatibility (Goddard et al., 2024):

• YAML-based interface: Users specify merging strategies, source checkpoints, and per-layer or per-block hyperparameters via configuration files, enabling modular, reproducible experiments.
• Out-of-core tensor loading: The system schedules operations using a directed acyclic graph (DAG) over model tensors, ensuring that only active components occupy RAM and GPU memory. This design supports merging at arbitrary scale, including on memory-constrained hardware.
• Modular Python codebase: The architecture exposes classes for merge planning, tensor scheduling, and method extension, documented for contributor adaptability.
• Transformers and Hugging Face Hub support: Seamless integration with standard model formats and orchestration of community-shared checkpoints.
• Support for token surgery: Integration with mergekit-tokensurgeon facilitates cross-tokenizer merging via OMP-based embedding alignment (Goddard et al., 7 Jun 2025), essential for merging models with incompatible vocabularies or custom tokenization.

In comparative tool surveys, MergeKit is positioned as engineering infrastructure—rather than an algorithmic solution—serving as a programmatic and batch-capable foundation for merging and fusion workflows (Yadav et al., 2024). It is distinct from GUI tools (e.g., ComfyUI) or routing libraries focused on inference-time expert selection.

5. Methodological Developments and Automated Merging

The search for optimal merging strategies has motivated algorithmic and automation extensions beyond static recipes:

• Gradient-based and supervised merging: Methods like SuperMerge (Yang et al., 2024) optimize per-layer combination weights via gradient descent on a small validation set, providing more expressive control than hand-tuned global mixing coefficients. These approaches can be integrated into the MergeKit pipeline when validation data are available.
• Automated model merging: Search frameworks treat merge configuration selection as a hyperparameter optimization problem, exploring layerwise fusion and depthwise integration spaces via multi-fidelity resource allocation (Su et al., 6 Feb 2025). This approach automates recipe discovery, often achieving superior multi-task and single-task performance compared to manual merging.
• Principled formalization: Output-space projection methods (Evans et al., 27 May 2026) cast model merging as a convex quadratic program over residual updates, minimizing a squared-output calibration objective. This envelops common heuristics (task arithmetic, TIES, etc.) as special cases with mathematical guarantees on calibration loss minimization. The fraction of residual energy captured by the chosen basis serves as a predictive diagnostic for merge success.

These methodological advances are either directly supported in MergeKit or are implementable within its framework.

6. Limitations, Security Considerations, and Future Directions

Model merging with MergeKit and its extensions is subject to well-understood limitations and emerging risks:

• Architecture and initialization constraints: Reliable merging typically assumes shared or compatible model architectures and initializations. Merges across distinct architectures require specialized strategies (e.g., permutation alignment, partial merging).
• Hyperparameter sensitivity and conflict resolution: The effectiveness of merges is sensitive to coefficient selection and merge rule localization (e.g., layerwise vs. global, block-level granularity), and parameter conflicts can arise in the absence of sign/magnitude alignment (e.g., for TIES).
• Tokenization alignment: Successful merging typically presupposes a shared vocabulary; OMP-based token surgery methods solve this for many cases but are susceptible to performance collapse when, for example, numeric tokenization schemes diverge (Goddard et al., 7 Jun 2025).
• Security and supply-chain risk: Token transplant pipelines in MergeKit are exposed to "breaker token" adversarial attacks, where a malicious donor tokenizer can surreptitiously inject a high-salience feature into the base model during transplantation, with persistence through subsequent merges, and stealth with respect to outlier detection (Liu et al., 31 Dec 2025).
• Copy-paste limitations and catastrophic interference: Parameter merging is not universally additive; functionally similar weights do not always sum to improved competence, particularly when merging from heterogeneous experts without explicit task disambiguation.

Ongoing research targets the discovery of reliable diagnostics, adaptive methods for layerwise/inter-layer conflict resolution, and the development of standard auditing and verification pipelines to detect and mitigate adversarial attacks in open-weight model composition.

---

References (arXiv IDs):

• (Goddard et al., 2024) Arcee's MergeKit: A Toolkit for Merging LLMs
• (Yadav et al., 2024) A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
• (Yang et al., 2024) SuperMerge: An Approach For Gradient-Based Model Merging
• (Su et al., 6 Feb 2025) Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
• (Hu et al., 15 May 2025) Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models
• (Goddard et al., 7 Jun 2025) Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
• (Hammoud et al., 17 Sep 2025) Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
• (Kodali et al., 22 Oct 2025) Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
• (Liu et al., 31 Dec 2025) The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition
• (Evans et al., 27 May 2026) Model Merging by Output-Space Projection