Expert Merging: Combining Specialized Models

• Expert merging is a technique that aggregates task-specific expert models by computing parameter offsets from a common pretrained backbone to preserve specialized capabilities.
• It employs diverse methods—including task arithmetic, sparse MoE fusion, and Bayesian refinement—to merge experts without full multi-task retraining.
• Recent studies highlight the importance of structured alignment and activation calibration in mitigating parameter interference and enhancing overall performance.

Expert merging denotes a family of parameter-space and module-space procedures for combining multiple specialized experts into a single model, a smaller expert set, or an adaptive composition mechanism, typically without joint multi-task retraining. In the most common formulation, a pretrained base model with parameters $\theta_{\mathrm{pre}}$ or $\theta_{\mathrm{base}}$ is adapted into task-specific experts $\{\theta^{(t)}\}$, and merging constructs $\theta_{\mathrm{merged}}$ from their offsets $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$ while attempting to preserve the specialists’ capabilities across tasks. The term is also used inside sparse Mixture-of-Experts (SMoE) architectures, where the merged objects are not whole models but expert submodules, expert clusters, or task-conditioned low-rank components. Recent work spans training-free arithmetic, training-light behavior alignment, Bayesian and evolutionary optimization, MoE-specific expert fusion, continual model merging, and online task-aware expert adaptation (Zhang et al., 30 Sep 2025, Zhou et al., 3 Feb 2025, Nguyen et al., 26 Feb 2025).

1. Formal setting and scope

A standard model-merging setup starts from a pretrained backbone $\theta_{\mathrm{pre}}$ and task-specific experts $\{\theta^{(t)}\}_{t=1}^T$, each obtained by fine-tuning from that backbone. The canonical task vector is

$\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}},$

and merging is expressed as an aggregation operator

$$\theta_{\mathrm{merged}} = \theta_{\mathrm{pre}} + \mathcal{A}\big(\tau^{(1)},\dots,\tau^{(T)}\big).$$

This formulation underlies task arithmetic, sparsity-aware merges, anchor-based Bayesian refinement, and several gradient-free search methods (Li et al., 13 May 2026).

A second formulation arises inside MoE layers. There, an expert is a submodule such as an FFN block with parameters $\mathbf{E}_i$, and routing combines expert outputs or activates a small subset. In SMoE notation, the output for token $\theta_{\mathrm{base}}$0 can be written as

$\theta_{\mathrm{base}}$1

or, in merging-based variants, as a single merged expert whose parameters are constructed from multiple experts rather than selected one by one (Nguyen et al., 26 Feb 2025).

The literature therefore uses “expert merging” in several closely related senses.

Setting	Merged object	Representative papers
Task-vector model merging	Full fine-tuned models	(Zhang et al., 30 Sep 2025, Li et al., 13 May 2026, Zhang et al., 27 Aug 2025)
MoE construction from dense experts	Non-FFN layers, FFNs, routers, projectors	(Zhou et al., 3 Feb 2025)
SMoE expert fusion/compression	Expert FFNs inside a layer	(Nguyen et al., 26 Feb 2025, Li et al., 29 Jun 2025, Nguyen et al., 17 Oct 2025)
Dynamic or continual merging	Modular experts or low-rank components	(Qiu et al., 6 Feb 2026, Du et al., 17 May 2026, Qiu et al., 24 Apr 2026)
Online adaptive merging	Merging weights over experts	(Han et al., 24 Sep 2025)

A further distinction is between local and non-local merging. Local merging assumes experts remain in a neighborhood of a shared initialization; non-local merging covers experts derived from different foundation models or experts separated by larger pretraining drift, where simple linear interpolation becomes unreliable (Sharma et al., 2024).

2. Training-free task-vector merging and parameter interference

The classical baseline is Task Arithmetic: $\theta_{\mathrm{base}}$2 with fixed global coefficients $\theta_{\mathrm{base}}$3. Many follow-ups, including TIES, DARE, TSV, Iso-C, and related methods, manipulate task vectors in parameter space, usually without explicit behavior matching. In this regime, a central failure mode is parameter interference: conflicting signs, large-magnitude influential parameters mixed with small-magnitude redundant ones, and the assumption that all layers or modules are equally mergeable (Zhang et al., 30 Sep 2025).

For homogeneous MoE construction, MergeME makes the same interference diagnosis for non-FFN layers. It replaces unweighted averaging of non-FFN parameters with TIES or DARE, keeps FFN layers as separate experts, and inserts sparse routing. The paper shows that attention task vectors of code and math experts are not highly similar, which contradicts the assumption that attention can safely be averaged. On its 4-expert homogeneous MoE benchmark, BTX achieves average 11.72, Ties MoE 12.52, and Dare MoE 12.86 across MBPP, HumanEval, MATH, GSM8K, NQ, and TriviaQA (Zhou et al., 3 Feb 2025).

The non-local model merging work sharpens this limitation. It argues that when experts do not share a common foundation model, or when pretraining changes are large, standard merging techniques fail even after permutation alignment. The paper identifies variance collapse in deep representations as a major source of degradation and proposes task-specific activation correction on top of standard merging as a strong baseline for the non-local regime (Sharma et al., 2024).

A plausible implication is that parameter-space arithmetic remains effective primarily when the experts lie in a sufficiently aligned basin and when the merged directions are not dominated by layer-wise or module-wise conflicts. Outside that regime, additional structural alignment becomes necessary.

3. Data-guided, behavior-aware, and Bayesian refinement

A major recent direction replaces fixed coefficients with small trainable coefficient sets. “Expert Merging” constructs each layer of the merged model as

$\theta_{\mathrm{base}}$4

with only the $\theta_{\mathrm{base}}$5 scalars $\theta_{\mathrm{base}}$6 learned, while base and expert weights remain frozen. The coefficients are optimized using a few unlabeled calibration examples per task, typically $\theta_{\mathrm{base}}$7–$\theta_{\mathrm{base}}$8, to align the merged model’s hidden states and logits to those of the corresponding experts. The total objective combines hidden-state alignment, KL-based logit alignment, nonnegative task weights $\theta_{\mathrm{base}}$9, and an $\{\theta^{(t)}\}$0 regularizer that keeps coefficients close to Task Arithmetic initialization. Expert Merging++ extends this with importance-guided chunking, where a normalized layer-importance score

$\{\theta^{(t)}\}$1

allocates more chunk-wise coefficients to high-importance layers while keeping low-importance layers lightweight. On InternVL2.5, Expert Merging reaches 58.11 average and Expert Merging++ 58.45, exceeding both WUDI v2 at 56.96 and Mixture Training at 57.66; on Mistral-7B, Expert Merging++ reaches 48.71 versus WUDI v2 at 45.48 (Zhang et al., 30 Sep 2025).

Bayesian Model Merging formulates module-wise merging as anchor-regularized Bayesian regression. For module $\{\theta^{(t)}\}$2, the inner problem is

$\{\theta^{(t)}\}$3

with closed-form solution

$\{\theta^{(t)}\}$4

An outer Bayesian Optimization loop then searches block-wise scales and regularization strengths. The framework also provides a data-free variant by replacing the activation Gram matrix with a surrogate built from task-vector Grams. On the ViT-L/14 8-task benchmark, a single merged model reaches 95.1 with data-assisted BMM and 95.0 with data-free BMM, compared with average individual expert accuracy 95.8 and ISO-CTS anchor 94.8 (Li et al., 13 May 2026).

Gradient-free search offers a complementary path. PSO-Merging initializes a particle swarm with a pretrained model, expert models, and sparsified expert models, and then updates full parameter vectors directly in weight space via Particle Swarm Optimization. On Llama-3-8B with three experts, PSO-Merging reaches 61.12 average, above Task Arithmetic at 50.45, TIES-Merging at 58.89, DARE-TIES at 60.29, and the CMA-ES “Evo” baseline at 51.95 (Zhang et al., 27 Aug 2025).

MERGEvolve pushes this further by treating model merging as the initialization of an evolution strategy and then exploring outside the convex combination space of the experts. Its analysis shows that static merging remains in the convex hull, Model Swarms remains in the affine hull, whereas MERGEvolve adds Gaussian perturbations that move in arbitrary directions. The paper reports best average performance on 9 of 12 single-task datasets and attributes much of the gain to the quality of the merged initialization (Wang et al., 17 Jun 2026).

4. MoE-centered expert fusion and compression

MergeME studies how to combine several dense expert LLMs into a single MoE. In the homogeneous case, non-FFN layers are merged with interference-aware operators such as Dare or Ties, FFN layers are kept as separate experts, and a sparse MoE FFN is inserted: $\{\theta^{(t)}\}$5 In the heterogeneous case, MergeME introduces shared embeddings and heads, per-expert Proj-in and Proj-out layers, and sequence-level routing over average embeddings, enabling merging across different architectures after fine-tuning with cross-entropy (Zhou et al., 3 Feb 2025).

CAMEx addresses expert merging inside SMoE layers with a curvature-aware rule. With a reference expert $\{\theta^{(t)}\}$6 and domain vectors $\{\theta^{(t)}\}$7, the merged expert is

$\{\theta^{(t)}\}$8

where $\{\theta^{(t)}\}$9 are learnable curvature matrices that approximate natural-gradient preconditioning without explicit Fisher inversion. CAMEx also proposes a dynamic architecture in which a global expert is propagated across layers. On GPT-2-small with SMoE, Domain-specific-CA reaches WikiText-2 perplexity 21.06 versus 21.45 for Ties and 21.56 for Euclidean domain-specific merging (Nguyen et al., 26 Feb 2025).

NAMEx reinterprets expert merging as a bargaining game. With domain-vector matrix $\theta_{\mathrm{merged}}$0, it solves

$\theta_{\mathrm{merged}}$1

to obtain Nash bargaining weights, and updates the base expert by

$\theta_{\mathrm{merged}}$2

A complex-momentum extension updates a momentum buffer $\theta_{\mathrm{merged}}$3 and applies $\theta_{\mathrm{merged}}$4 to the base expert. The paper reports consistent gains over CAMEx and EP-CAMEx on language modelling, GLUE, ImageNet-1k, and large MoE systems including Qwen1.5-MoE and DeepSeek-MoE (Nguyen et al., 17 Oct 2025).

Sub-MoE addresses MoE compression by first clustering experts via K-means on cosine similarity of expert outputs, then performing joint SVD on concatenated expert weights to obtain a shared $\theta_{\mathrm{merged}}$5-matrix and expert-specific $\theta_{\mathrm{merged}}$6 components, and finally merging the $\theta_{\mathrm{merged}}$7 blocks with router-frequency weights. On Mixtral-8x7B, Sub-MoE maintains 96% of original performance with 25% expert reduction and 86% with 50% expert reduction on zero-shot benchmarks (Li et al., 29 Jun 2025).

A different internal-MoE perspective appears in “Learning More Generalized Experts by Merging Experts in Mixture-of-Experts,” where the two most frequently selected experts are averaged and the least frequently selected expert is overwritten: $\theta_{\mathrm{merged}}$8 The number of experts remains fixed, and the router subsequently adapts to the merged expert. The reported gains are modest but consistent in multi-domain task incremental learning (Park, 2024).

5. Dynamic, modular, continual, and online expert merging

MERGE generalizes expert merging from whole-model operations to component-wise modular recombination. A configuration matrix $\theta_{\mathrm{merged}}$9 assigns each task-specific component $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$0 to a group, and a modular expert is built as

$\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$1

where $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$2 is an intra-group merge such as weight averaging, Task Arithmetic, Ties-Merging, or Breadcrumbs, and $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$3 is conditional quantization. MERGE searches configurations by a Random-Forest-assisted NSGA-II over cross-task performance and storage cost, then assembles input-specific models from a reusable expert library via a lightweight routing network. On ViT-B/32, MERGE-TM at one Pareto point achieves $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$4 and $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$5, above EMR-Merging at $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$6 and $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$7 (Qiu et al., 6 Feb 2026).

DiDi-Merging addresses the accuracy–efficiency trade-off in dynamic model merging by decomposing task vectors into a shared component

$\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$8

and task-specific residuals $\tau^{(t)} = \theta^{(t)} - \theta_{\mathrm{pre}}$9, representing both as low-rank modules. Ranks are allocated through a differentiable tanh-based truncation on singular values and optimized under a global budget: $\theta_{\mathrm{pre}}$0 A data-free refinement stage then optimizes LoRA factors to reconstruct task vectors. DiDi-Merging matches prior dynamic baselines at 1.24x the parameters of a single fine-tuned model and surpasses them at 1.4x, whereas several prior dynamic baselines require more than 2x storage (Du et al., 17 May 2026).

MADE-IT studies continual model merging as a balance between backbone saturation and MoE-style expert redundancy. It compresses module updates into low-rank experts, represents them as Grassmannian subspaces, and uses a projection-based affinity

$\theta_{\mathrm{pre}}$1

with an adaptive threshold to decide whether to merge or create experts. Routing is data-free and training-free, based on feature–subspace alignment rather than a learned gate. On 20 tasks with CLIP ViT-B/32, MADE-IT reaches ACC 81.4% versus MINGLE at 77.1% and OPCM at 65.7% (Qiu et al., 24 Apr 2026).

Tanbr extends expert merging to online inference in SMoE. It estimates a task-distribution vector $\theta_{\mathrm{pre}}$2 for each time slot, chooses continuous merging weights $\theta_{\mathrm{pre}}$3 under simplex and sparsity constraints, and learns them with a tree-structured adaptive neural bandit. The method proves a regret bound of $\theta_{\mathrm{pre}}$4 and reports at least 45% lower inference latency and up to 25% lower memory usage while maintaining high accuracy (Han et al., 24 Sep 2025).

6. Empirical regularities, limitations, and open problems

Across otherwise different formulations, several recurring empirical regularities appear. First, expert merging is not uniformly local in depth or module type. Expert Merging++ reports that learned coefficient magnitudes concentrate in late layers and that MLP blocks are consistently more important than attention, especially in late stages; CAMEx likewise finds non-uniform benefits across modules; MADE-IT reports heavy pruning in generic modules and early layers, with more retained diversity in deeper modules (Zhang et al., 30 Sep 2025, Nguyen et al., 26 Feb 2025, Qiu et al., 24 Apr 2026).

Second, parameter interference remains a central obstacle. MergeME attributes this to conflicting task vectors and shows that removing coefficient regularization causes large average drops, including 58.11 $\theta_{\mathrm{pre}}$5 47.57 on InternVL and 47.43 $\theta_{\mathrm{pre}}$6 40.81 on Mistral. MergeME also shows that naive chunking of all layers degrades performance, whereas importance-guided chunking improves it. MergeME further reports that removing hidden alignment causes mild degradation while removing logit alignment causes much larger drops, which suggests that behavior-level matching matters more directly than parameter-centric objectives (Zhang et al., 30 Sep 2025).

Third, permutation alignment is not sufficient in the non-local regime. The non-local model merging paper argues that even after standard weight matching, non-local experts suffer from variance collapse in deep activations, and it proposes task-specific activation re-scaling and shifting to align output statistics. This challenges the misconception that permutation symmetries exhaust the non-local problem (Sharma et al., 2024).

The main limitations reported across the literature are also consistent. Expert Merging depends on small but representative calibration sets and assumes experts share a backbone; MergeME identifies scaling to dozens or hundreds of experts and heterogeneous architectures as open problems (Zhang et al., 30 Sep 2025). BMM explicitly limits experiments to models up to 8B parameters and notes that the alignment theorem may fail under extreme or non-standard fine-tuning (Li et al., 13 May 2026). MergeME and MADE-IT both identify routing or weighting under growing expert sets as increasingly delicate (Zhou et al., 3 Feb 2025, Qiu et al., 24 Apr 2026). MERGE and DiDi-Merging shift attention to storage budgets, reusable modular libraries, and the trade-off between static reuse and per-input customization (Qiu et al., 6 Feb 2026, Du et al., 17 May 2026).

A plausible synthesis is that expert merging is evolving from uniform global arithmetic toward structured alignment under explicit heterogeneity: heterogeneity across layers, across module types, across tasks, across expert architectures, across inference contexts, and across time. The technical frontier therefore lies less in a single universal merge operator than in principled mechanisms for deciding what should be shared, what should remain specialized, and how much adaptation can be afforded without reverting to full multi-task training.