Expert Merging++: Advanced Model Consolidation

• Expert Merging++ is a class of advanced model-merging methods that dynamically recombines task-specific parameter adjustments, improving efficiency and accuracy without joint training.
• It leverages dynamic routing, modular recombination, and unlabeled calibration data to optimize the balance between shared and expert-specific capacities.
• The approach supports multiple merge granularities—including channel, component, and subspace levels—to fine-tune the trade-offs between storage cost and performance.

“Expert Merging++” (Editor’s term) denotes a class of advanced model-merging methods that combine multiple task- or domain-specialized experts into a single deployable system while going beyond static weight averaging. In current literature, this umbrella includes dynamic merging of task vectors, component-wise modular recombination, sparse-MoE expert compression, curvature- and game-theoretic expert propagation, evolutionary exploration beyond convex combinations, and training-light coefficient learning from unlabeled calibration data. Across these variants, the central objective is to improve the accuracy–efficiency trade-off of expert consolidation without joint multi-task training and, in many cases, without access to original task data (Du et al., 17 May 2026, Qiu et al., 6 Feb 2026, Zhang et al., 30 Sep 2025).

1. Conceptual foundations

A common starting point is a pretrained backbone with parameters $\boldsymbol{\theta}_0$ and a set of fine-tuned expert models $\{\boldsymbol{\theta}_t\}_{t=1}^T$. The standard task-vector view defines

$$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$

Static model merging then forms a single input-agnostic checkpoint

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$

which covers Task Arithmetic, TIES, and related “soups”-style methods. Dynamic merging instead makes the merged parameters depend on the input or task identity, as in

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$

with a shared component $\boldsymbol{M}_s$, task-specific experts $\boldsymbol{M}_t$, and router weights $w_t(\mathbf{x})$ (Du et al., 17 May 2026).

This literature consistently distinguishes advanced expert merging from standard jointly trained MoE. In MoE, experts and router are typically trained together and capacity lives in the experts; in post-hoc dynamic merging, the inputs are already fine-tuned models or adapters, and routing selects or composes stored parameter deltas rather than full independently trained sub-networks. MERGE and related work further refine the notion of an “expert” by decomposing each model into homologous components, such as attention and MLP modules, and recombining them at inference rather than treating each task model as a monolithic whole (Qiu et al., 6 Feb 2026, Zhou et al., 3 Feb 2025).

A recurring theme is that static averaging is attractive for storage but suffers from interference, whereas fully storing many experts preserves specialization but scales poorly. “Expert Merging++” therefore emphasizes mechanisms that regulate where sharing should occur, how much capacity should remain expert-specific, and when routing or recombination should be reusable rather than instance-disposable (Du et al., 17 May 2026, Qiu et al., 6 Feb 2026).

2. Merge granularity and parameter organization

One major axis of variation is merge granularity. Early and simpler schemes merge at the whole-model or whole-layer level. More recent systems operate at finer resolution because homologous components have different mergeability. MERGE formulates a configuration matrix $\mathbf{G}=[g_{t,l}]$ over tasks and components, builds modular experts

$$\hat{\theta}_l^n = \mathcal{Q}\big(\mathcal{F}(\theta_{\text{pre}}^l,\Theta_l^n)\big),$$

and stores a reusable modular expert library $\{\boldsymbol{\theta}_t\}_{t=1}^T$0 assembled from Pareto-optimal configurations (Qiu et al., 6 Feb 2026).

A second line pushes granularity to the channel level. Channel Merging clusters output-channel deltas $\{\boldsymbol{\theta}_t\}_{t=1}^T$1 into $\{\boldsymbol{\theta}_t\}_{t=1}^T$2 groups per channel, merges each group by task arithmetic,

$\{\boldsymbol{\theta}_t\}_{t=1}^T$3

and reconstructs expert $\{\boldsymbol{\theta}_t\}_{t=1}^T$4 at inference by channel-wise lookup using an index map $\{\boldsymbol{\theta}_t\}_{t=1}^T$5. This preserves expert-specific channel patterns while reducing storage from $\{\boldsymbol{\theta}_t\}_{t=1}^T$6 toward $\{\boldsymbol{\theta}_t\}_{t=1}^T$7 for model size $\{\boldsymbol{\theta}_t\}_{t=1}^T$8 (Zhang et al., 2024).

A third line works in a shared subspace. Sub-MoE performs a joint SVD on concatenated expert weights,

$\{\boldsymbol{\theta}_t\}_{t=1}^T$9

treating $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$0 as a shared subspace and merging only the expert-specific $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$1 coordinates. MergeMoE gives a related but output-centric reinterpretation: if expert outputs are collected as $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$2 and routing weights as $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$3, a compressed MoE can be written as

$$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$4

with clustering matrix $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$5, output-combining matrix $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$6, and inserted compression matrices $$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$7 inside the expert computation. This suggests that expert merging can be understood as forward-graph matrix insertion rather than only parameter aggregation (Li et al., 29 Jun 2025, Miao et al., 16 Oct 2025).

These granularities are not interchangeable. A plausible implication is that whole-model merging is appropriate when experts are already well aligned, whereas channel-, component-, or subspace-level schemes are preferable when specialization is heterogeneous across modules.

3. Budgeting, alignment, and routing

A defining characteristic of the newer literature is that capacity allocation itself becomes an optimization problem. DiDi-Merging decomposes each task vector into a shared mean component and a per-task residual,

$$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$8

then applies SVD to each matrix and treats the effective rank as a continuous variable through smooth singular-value truncation,

$$\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.$$9

Ranks are optimized under a task-reconstruction term and a compression-ratio penalty,

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$0

followed by a data-free LoRA-style refinement step that reconstructs stored task vectors without any original data (Du et al., 17 May 2026).

MERGE frames configuration search as a bi-objective optimization balancing average cross-task performance and storage cost. It uses surrogate-assisted NSGA-II, initializes with K-means clustering over homologous components, and materializes a Pareto frontier of reusable modular expert libraries. Online recombination then uses a lightweight 3-layer fully connected routing network trained with at most 1000 instances per task to predict a task pattern and retrieve an expert chain (Qiu et al., 6 Feb 2026).

Expert Merging and Expert Merging++ take a different route: they learn a very small set of layer-wise or chunk-wise coefficients from unlabeled calibration data. For layer-wise coefficients,

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$1

the optimization explicitly aligns merged hidden states and logits with those of the corresponding experts through hidden-state MSE and temperature-scaled KL divergence, plus an L1 regularizer on coefficient deviation. Expert Merging++ then computes a normalized layer-importance score

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$2

and allocates chunk counts by

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$3

giving more degrees of freedom to high-importance layers while keeping the total number of learned parameters near the original $$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$4 scale (Zhang et al., 30 Sep 2025).

A persistent misconception is that all adaptive merging is “data-free.” The literature is more differentiated: DiDi performs fully data-free reconstruction from task vectors, MERGE requires a small labeled routing set, and Expert Merging++ uses unlabeled calibration examples to align behavior (Du et al., 17 May 2026, Qiu et al., 6 Feb 2026, Zhang et al., 30 Sep 2025).

4. Sparse-MoE, heterogeneous experts, and compression-oriented variants

Within sparse MoE, “Expert Merging++” often means replacing ad hoc expert averaging by routing-aware, geometry-aware, or compression-aware aggregation. MergeME addresses both homogeneous and heterogeneous expert sets. In the homogeneous case, it merges non-FFN layers with interference-aware task-vector methods such as TIES or Dare, uses top-$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$5 MoE routing, and proposes perplexity-based sequence routing that selects experts without router training. In the heterogeneous case, where experts may have different layer counts, it introduces a shared embedding/head, per-expert projectors, and sequence-level routing so that different architectures can still be composed in a single MoE (Zhou et al., 3 Feb 2025).

CAMEx reinterprets expert merging as a natural-gradient-style update on domain vectors

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$6

and replaces Euclidean merging with

$$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$7

where $$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$8 are learned curvature matrices in Kronecker-factorized form. Its dynamic variant propagates a global expert across layers, coupling expert consolidation with resource reduction in sparse MoE (Nguyen et al., 26 Feb 2025).

NAMEx pushes this further by casting expert merging as Nash bargaining. If the domain vectors are columns of $$\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,$$9, the bargaining solution yields

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$0

so merging weights depend on cooperative and competitive interactions among experts rather than simple averaging. The method additionally introduces complex momentum for expert propagation and reports convergence under spectral-radius conditions (Nguyen et al., 17 Oct 2025).

Compression-focused MoE merging provides another branch. Sub-MoE groups experts by functional similarity, merges them in a shared SVD subspace, and optionally adds activation-aware intra-expert compression. MergeMoE, by contrast, treats the compression of an MoE layer as an optimization problem over expert outputs and solves a least-squares subproblem for an inserted matrix

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$1

Both approaches aim to reduce memory overhead while retaining routed behavior, but they formalize the problem at different levels: Sub-MoE in shared parameter subspaces, MergeMoE in the forward computation of merged expert outputs (Li et al., 29 Jun 2025, Miao et al., 16 Oct 2025).

A separate training-time variant, motivated by continual multi-domain learning, periodically merges the two most frequently selected experts and overwrites the least frequently selected one, then continues router and expert training. This suggests that usage-frequency statistics can serve not only as merge weights but also as a capacity-recycling signal inside MoE training (Park, 2024).

5. Theory-driven extensions beyond heuristic averaging

Several recent works argue that advanced expert merging requires explicit theory about generalization, covariance, geometry, or search space.

MERGEvolve begins with a task-aware merged initialization

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$2

and then applies an evolution-strategy update

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$3

with Gaussian perturbations $$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$4. Its theoretical contribution is to show that while static merging is confined to the convex hull of expert parameters, the evolution phase explores regions outside both the convex and affine hulls (Wang et al., 17 Jun 2026).

A complementary theoretical line studies when merging should generalize. Under heterogeneous fine-tuning hyperparameters, the unified framework based on $$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$5-stability gives a merged model

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$6

and decomposes excess error into a stability term and a gradient-norm term. The resulting bound makes learning rate, number of steps, batch size, task heterogeneity, and expert weights explicit, and it predicts that large learning rates and strong heterogeneity harm merge-friendliness, whereas more data and suitably controlled weighting help (Li et al., 29 Jan 2026).

ACE-Merging targets the same interference problem from another angle. It argues that input covariance, a key factor in optimal merging, can be implicitly estimated from the parameter differences of fine-tuned models, even in a fully data-free setting. On that basis it proposes a closed-form adaptive covariance estimation framework rather than an iterative or heuristic solver (Xu et al., 3 Mar 2026).

ResMerge shows that for RL experts, the common assumption “leading singular directions contain the main signal and residuals can be compressed away” is unreliable. After decomposing each task vector into a leading head $$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$7 and residual $$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$8, it constructs a residual backbone by Spherical Residual Consensus Adaptation,

$$\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,$$9

and then adds a Lightweight Head Correction

$\boldsymbol{M}_s$0

This directly challenges a widespread spectral-merging heuristic by treating head and residual as distinct, both behavior-bearing components (Sun et al., 1 Jun 2026).

Taken together, these works suggest that “Expert Merging++” is increasingly defined by explicit modeling assumptions—about curvature, covariance, bargaining, or spectral decomposition—rather than by heuristic coefficient tuning alone.

6. Empirical patterns, misconceptions, and outlook

Representative systems illustrate the range of current designs and trade-offs.

Method	Core mechanism	Representative result or regime
DiDi-Merging	Differentiable shared/expert low-rank budgeting	1.24× params: 89.3%; 1.4×: 89.8 on 8 vision tasks (Du et al., 17 May 2026)
MERGE	Component-wise expert library + reusable routing	MERGE-TM (G2): $\boldsymbol{M}_s$1, $\boldsymbol{M}_s$2 on ViT-B/32 (Qiu et al., 6 Feb 2026)
Sub-MoE	Union-SVD subspace merging for MoE experts	Maintains 96%
Channel Merging	Channel-group parameter bank + per-expert indices	Comparable to model ensemble with just 53% parameters with a task-specific router (Zhang et al., 2024)
Expert Merging++	Hidden/logit alignment + importance-guided chunking	Best averages on InternVL, Qwen2-VL, and Mistral settings reported (Zhang et al., 30 Sep 2025)
MERGEvolve	Merged initialization + ES exploration	Best performance on 9/12 tasks in single-task experiments (Wang et al., 17 Jun 2026)

Several empirical regularities recur. First, balanced shared/expert decomposition is usually superior to both “heavy shared, tiny experts” and “heavy expert, no shared” extremes; DiDi-Merging explicitly targets this balanced regime and reports 89.3% average accuracy at 1.24× parameters and 89.8% at 1.4× on 8 vision tasks, compared with a 2.08× TSV-C baseline at 89.5% (Du et al., 17 May 2026). Second, modular or granular schemes often dominate monolithic ones under the same storage budget; MERGE reports that on ViT-B/32, EMR yields $\boldsymbol{M}_s$3, whereas MERGE-TM at a G2 operating point yields $\boldsymbol{M}_s$4 (Qiu et al., 6 Feb 2026). Third, subspace- and channel-level compression can preserve specialization at sharply reduced storage: Sub-MoE reports 96% and 86% of original Mixtral-8x7B performance at 25% and 50% expert reduction, and Channel Merging reports results comparable to model ensemble with just 53% parameters when paired with a task-specific router (Li et al., 29 Jun 2025, Zhang et al., 2024). Fourth, search beyond ordinary convex merging can discover genuinely new behaviors: MERGEvolve reports best performance on 9 of 12 tasks and an emergent capability rate of 13.95% on ARC-C (Wang et al., 17 Jun 2026).

Common misconceptions are addressed directly by the literature. Expert merging is not identical to standard MoE: many systems operate post hoc on fine-tuned checkpoints rather than jointly learned experts and routers (Du et al., 17 May 2026, Zhou et al., 3 Feb 2025). Not all residual spectral mass is noise: ResMerge shows that for RL task vectors, both leading heads and residuals can independently recover substantial behavior knowledge (Sun et al., 1 Jun 2026). More experts do not automatically improve a merge: large models and strong bases make merging easier, but the scale study finds that weak bases or smaller models suffer significant held-in degradation as the number of merged experts rises (Yadav et al., 2024). Data-free is not universal: some of the strongest methods use unlabeled or lightly labeled calibration data rather than none at all (Zhang et al., 30 Sep 2025, Qiu et al., 6 Feb 2026).

A plausible near-term direction is the combination of these strands: differentiable capacity allocation, reusable modular libraries, theory-guided weighting, and lightweight input-aware routing within a single expert-merging pipeline. The literature already suggests the ingredients—data-free delta reconstruction, output-level compression, chunk-wise importance allocation, bargaining or curvature-aware propagation, and subspace alignment across heterogeneous experts—but their unification remains an open systems problem rather than a settled recipe (Du et al., 17 May 2026, Miao et al., 16 Oct 2025, Nguyen et al., 17 Oct 2025).