---
title: 'Expert Merging++: Advanced Model Consolidation'
url: https://www.emergentmind.com/topics/expert-merging-8ebffe4f-63d6-404d-bf2a-1a3ce7929494
type: topic
---

# Expert Merging++: Advanced Model Consolidation

**“Expert Merging++” (Editor’s term)** denotes a class of advanced model-merging methods that combine multiple task- or domain-specialized experts into a single deployable system while going beyond static weight averaging. In current literature, this umbrella includes dynamic merging of task vectors, component-wise modular recombination, sparse-MoE expert compression, curvature- and game-theoretic expert propagation, evolutionary exploration beyond convex combinations, and training-light coefficient learning from unlabeled calibration data. Across these variants, the central objective is to improve the accuracy–efficiency trade-off of expert consolidation without joint multi-task training and, in many cases, without access to original task data [2605.18904] [2602.06552] [2509.25712].

## 1. Conceptual foundations

A common starting point is a pretrained backbone with parameters $\boldsymbol{\theta}_0$ and a set of fine-tuned expert models $\{\boldsymbol{\theta}_t\}_{t=1}^T$. The standard task-vector view defines
\[
\boldsymbol{\tau}_t = \boldsymbol{\theta}_t - \boldsymbol{\theta}_0.
\]
Static model merging then forms a single input-agnostic checkpoint
\[
\boldsymbol{\theta}^* = \boldsymbol{\theta}_0 + \sum_{t=1}^T \lambda_t \boldsymbol{\tau}_t,
\]
which covers Task Arithmetic, TIES, and related “soups”-style methods. Dynamic merging instead makes the merged parameters depend on the input or task identity, as in
\[
\boldsymbol{\theta}^*(\mathbf{x}) = \boldsymbol{\theta}_0 + \boldsymbol{M}_s + \sum_{t=1}^T w_t(\mathbf{x}) \boldsymbol{M}_t,
\]
with a shared component $\boldsymbol{M}_s$, task-specific experts $\boldsymbol{M}_t$, and router weights $w_t(\mathbf{x})$ [2605.18904].

This literature consistently distinguishes advanced expert merging from standard jointly trained MoE. In MoE, experts and router are typically trained together and capacity lives in the experts; in post-hoc dynamic merging, the inputs are already fine-tuned models or adapters, and routing selects or composes stored parameter deltas rather than full independently trained sub-networks. MERGE and related work further refine the notion of an “expert” by decomposing each model into homologous components, such as attention and MLP modules, and recombining them at inference rather than treating each task model as a monolithic whole [2602.06552] [2502.00997].

A recurring theme is that static averaging is attractive for storage but suffers from interference, whereas fully storing many experts preserves specialization but scales poorly. “Expert Merging++” therefore emphasizes mechanisms that regulate where sharing should occur, how much capacity should remain expert-specific, and when routing or recombination should be reusable rather than instance-disposable [2605.18904] [2602.06552].

## 2. Merge granularity and parameter organization

One major axis of variation is **merge granularity**. Early and simpler schemes merge at the whole-model or whole-layer level. More recent systems operate at finer resolution because homologous components have different mergeability. MERGE formulates a configuration matrix $\mathbf{G}=[g_{t,l}]$ over tasks and components, builds modular experts
\[
\hat{\theta}_l^n = \mathcal{Q}\big(\mathcal{F}(\theta_{\text{pre}}^l,\Theta_l^n)\big),
\]
and stores a reusable modular expert library $\mathbb{L}$ assembled from Pareto-optimal configurations [2602.06552].

A second line pushes granularity to the **channel level**. Channel Merging clusters output-channel deltas $\boldsymbol{\delta}_i^{t_n}$ into $K$ groups per channel, merges each group by task arithmetic,
\[
\boldsymbol{\Theta}_i^k = \mathbf{P}_i + \lambda \sum_{\boldsymbol{\delta}_i^{t_n}\in C_i^k}\boldsymbol{\delta}_i^{t_n},
\]
and reconstructs expert $t_n$ at inference by channel-wise lookup using an index map $S^{t_n}$. This preserves expert-specific channel patterns while reducing storage from $N\Psi$ toward $K\Psi$ for model size $\Psi$ [2412.15283].

A third line works in a **shared subspace**. Sub-MoE performs a joint SVD on concatenated expert weights,
\[
\text{SVD}\big([W^{(1)}; \dots; W^{(n)}]\big) = U\Sigma [V^{(1)}; \dots; V^{(n)}]^T,
\]
treating $U\Sigma$ as a shared subspace and merging only the expert-specific $V$ coordinates. MergeMoE gives a related but output-centric reinterpretation: if expert outputs are collected as $Y(X)$ and routing weights as $g(X)$, a compressed MoE can be written as
\[
\text{MoE}'(X)=Y(X)BA\,g(X)^\top,
\]
with clustering matrix $A$, output-combining matrix $B$, and inserted compression matrices $T_1,T_2,T_3$ inside the expert computation. This suggests that expert merging can be understood as forward-graph matrix insertion rather than only parameter aggregation [2506.23266] [2510.14436].

These granularities are not interchangeable. A plausible implication is that whole-model merging is appropriate when experts are already well aligned, whereas channel-, component-, or subspace-level schemes are preferable when specialization is heterogeneous across modules.

## 3. Budgeting, alignment, and routing

A defining characteristic of the newer literature is that **capacity allocation itself becomes an optimization problem**. DiDi-Merging decomposes each task vector into a shared mean component and a per-task residual,
\[
\boldsymbol{M}_s = \mathrm{avg}(\boldsymbol{\tau}_1,\dots,\boldsymbol{\tau}_T),\qquad
\boldsymbol{M}_t = \boldsymbol{\tau}_t - \boldsymbol{M}_s,
\]
then applies SVD to each matrix and treats the effective rank as a continuous variable through smooth singular-value truncation,
\[
\tilde{\sigma}_i = \sigma_i \left[0.5\cdot\tanh\big(\beta(r-i)\big)+0.5\right].
\]
Ranks are optimized under a task-reconstruction term and a compression-ratio penalty,
\[
L = L_{\text{task}} + \gamma |R_{\text{now}} - R_{\text{tar}}|,
\]
followed by a data-free LoRA-style refinement step that reconstructs stored task vectors without any original data [2605.18904].

MERGE frames configuration search as a **bi-objective optimization** balancing average cross-task performance and storage cost. It uses surrogate-assisted NSGA-II, initializes with K-means clustering over homologous components, and materializes a Pareto frontier of reusable modular expert libraries. Online recombination then uses a lightweight 3-layer fully connected routing network trained with at most 1000 instances per task to predict a task pattern and retrieve an expert chain [2602.06552].

Expert Merging and Expert Merging++ take a different route: they learn a very small set of layer-wise or chunk-wise coefficients from **unlabeled calibration data**. For layer-wise coefficients,
\[
\theta_{\text{merged}}^\ell = \theta_{\text{base}}^\ell + \sum_{k=1}^K \alpha_k^\ell \tau_k^\ell,
\]
the optimization explicitly aligns merged hidden states and logits with those of the corresponding experts through hidden-state MSE and temperature-scaled KL divergence, plus an L1 regularizer on coefficient deviation. Expert Merging++ then computes a normalized layer-importance score
\[
\tilde I_\ell = \sum_{k=1}^K |\alpha_k^\ell| s_k^\ell n_\ell,\qquad
I_\ell = \frac{\tilde I_\ell}{\sum_j \tilde I_j},
\]
and allocates chunk counts by
\[
m_\ell = \left\lfloor B\cdot \frac{I_\ell^\kappa}{\sum_{j=1}^L I_j^\kappa}\right\rfloor,
\]
giving more degrees of freedom to high-importance layers while keeping the total number of learned parameters near the original $K\times L$ scale [2509.25712].

A persistent misconception is that all adaptive merging is “data-free.” The literature is more differentiated: DiDi performs fully data-free reconstruction from task vectors, MERGE requires a small labeled routing set, and Expert Merging++ uses unlabeled calibration examples to align behavior [2605.18904] [2602.06552] [2509.25712].

## 4. Sparse-MoE, heterogeneous experts, and compression-oriented variants

Within sparse MoE, “Expert Merging++” often means replacing ad hoc expert averaging by routing-aware, geometry-aware, or compression-aware aggregation. MergeME addresses both **homogeneous** and **heterogeneous** expert sets. In the homogeneous case, it merges non-FFN layers with interference-aware task-vector methods such as TIES or Dare, uses top-$K$ MoE routing, and proposes perplexity-based sequence routing that selects experts without router training. In the heterogeneous case, where experts may have different layer counts, it introduces a shared embedding/head, per-expert projectors, and sequence-level routing so that different architectures can still be composed in a single MoE [2502.00997].

CAMEx reinterprets expert merging as a natural-gradient-style update on domain vectors
\[
\tau_i=\mathbf{E}_i-\mathbf{E}_m,
\]
and replaces Euclidean merging with
\[
\hat{\mathbf{E}}_m^l=\mathbf{E}_m^l+\alpha\sum_{i=1}^{N-1}\mathbf{M}_i\cdot (s_i^l * \tau_i^l),
\]
where $\mathbf{M}_i$ are learned curvature matrices in Kronecker-factorized form. Its dynamic variant propagates a global expert across layers, coupling expert consolidation with resource reduction in sparse MoE [2502.18821].

NAMEx pushes this further by casting expert merging as **Nash bargaining**. If the domain vectors are columns of $\mathbf{G}$, the bargaining solution yields
\[
\Delta^*=\sum_{i=1}^N \alpha_i \tau_i,\qquad
\mathbf{G}^\top \mathbf{G}\,\boldsymbol{\alpha}=1/\boldsymbol{\alpha},
\]
so merging weights depend on cooperative and competitive interactions among experts rather than simple averaging. The method additionally introduces complex momentum for expert propagation and reports convergence under spectral-radius conditions [2510.16138].

Compression-focused MoE merging provides another branch. Sub-MoE groups experts by functional similarity, merges them in a shared SVD subspace, and optionally adds activation-aware intra-expert compression. MergeMoE, by contrast, treats the compression of an MoE layer as an optimization problem over expert outputs and solves a least-squares subproblem for an inserted matrix
\[
T_1 = QP^\dagger.
\]
Both approaches aim to reduce memory overhead while retaining routed behavior, but they formalize the problem at different levels: Sub-MoE in shared parameter subspaces, MergeMoE in the forward computation of merged expert outputs [2506.23266] [2510.14436].

A separate training-time variant, motivated by continual multi-domain learning, periodically merges the two most frequently selected experts and overwrites the least frequently selected one, then continues router and expert training. This suggests that usage-frequency statistics can serve not only as merge weights but also as a capacity-recycling signal inside MoE training [2405.11530].

## 5. Theory-driven extensions beyond heuristic averaging

Several recent works argue that advanced expert merging requires explicit theory about **generalization, covariance, geometry, or search space**.

MERGEvolve begins with a task-aware merged initialization
\[
\theta^{(0)}=\theta_{\text{pre}}+\frac{\alpha}{M}\sum_{i=1}^M w_i(\theta_i-\theta_{\text{pre}})
\]
and then applies an evolution-strategy update
\[
\theta^{(t+1)}=\theta^{(t)}+\frac{\alpha}{N\sigma}\sum_{i=1}^N w_i \epsilon_i^{(t)},
\]
with Gaussian perturbations $\epsilon_i^{(t)}$. Its theoretical contribution is to show that while static merging is confined to the convex hull of expert parameters, the evolution phase explores regions outside both the convex and affine hulls [2606.28373].

A complementary theoretical line studies when merging should generalize. Under heterogeneous fine-tuning hyperparameters, the unified framework based on $L_2$-stability gives a merged model
\[
\boldsymbol{x}_{avg}=\sum_{i=1}^N \lambda_i \boldsymbol{x}_i
\]
and decomposes excess error into a stability term and a gradient-norm term. The resulting bound makes learning rate, number of steps, batch size, task heterogeneity, and expert weights explicit, and it predicts that large learning rates and strong heterogeneity harm merge-friendliness, whereas more data and suitably controlled weighting help [2601.21690].

ACE-Merging targets the same interference problem from another angle. It argues that input covariance, a key factor in optimal merging, can be implicitly estimated from the parameter differences of fine-tuned models, even in a fully data-free setting. On that basis it proposes a closed-form adaptive covariance estimation framework rather than an iterative or heuristic solver [2603.02945].

ResMerge shows that for **RL experts**, the common assumption “leading singular directions contain the main signal and residuals can be compressed away” is unreliable. After decomposing each task vector into a leading head $H_i$ and residual $R_i$, it constructs a residual backbone by Spherical Residual Consensus Adaptation,
\[
q_{\text{src}}=\frac{\sum_{i=1}^N q_i}{\left\|\sum_{i=1}^N q_i\right\|_F+\epsilon},\qquad
A=\alpha_R n_R q_{\text{src}},
\]
and then adds a Lightweight Head Correction
\[
C=\rho \cdot S_H \cdot \|A\|_F \cdot \frac{H_{\text{mean}}}{\|H_{\text{mean}}\|_F+\epsilon}.
\]
This directly challenges a widespread spectral-merging heuristic by treating head and residual as distinct, both behavior-bearing components [2606.02252].

Taken together, these works suggest that “Expert Merging++” is increasingly defined by explicit modeling assumptions—about curvature, covariance, bargaining, or spectral decomposition—rather than by heuristic coefficient tuning alone.

## 6. Empirical patterns, misconceptions, and outlook

Representative systems illustrate the range of current designs and trade-offs.

| Method | Core mechanism | Representative result or regime |
|---|---|---|
| DiDi-Merging | Differentiable shared/expert low-rank budgeting | 1.24× params: 89.3%; 1.4×: 89.8 on 8 vision tasks [2605.18904] |
| MERGE | Component-wise expert library + reusable routing | MERGE-TM (G2): $\mathcal{C}=1.0162$, $\mathcal{P}=89.2$ on ViT-B/32 [2602.06552] |
| Sub-MoE | Union-SVD subspace merging for MoE experts | Maintains 96%|86% of original performance with 25%|50% expert reduction on Mixtral-8x7B [2506.23266] |
| Channel Merging | Channel-group parameter bank + per-expert indices | Comparable to model ensemble with just 53% parameters with a task-specific router [2412.15283] |
| Expert Merging++ | Hidden/logit alignment + importance-guided chunking | Best averages on InternVL, Qwen2-VL, and Mistral settings reported [2509.25712] |
| MERGEvolve | Merged initialization + ES exploration | Best performance on 9/12 tasks in single-task experiments [2606.28373] |

Several empirical regularities recur. First, balanced shared/expert decomposition is usually superior to both “heavy shared, tiny experts” and “heavy expert, no shared” extremes; DiDi-Merging explicitly targets this balanced regime and reports 89.3% average accuracy at 1.24× parameters and 89.8% at 1.4× on 8 vision tasks, compared with a 2.08× TSV-C baseline at 89.5% [2605.18904]. Second, modular or granular schemes often dominate monolithic ones under the same storage budget; MERGE reports that on ViT-B/32, EMR yields $\mathcal{C}=1.25,\mathcal{P}=88.8$, whereas MERGE-TM at a G2 operating point yields $\mathcal{C}=1.0162,\mathcal{P}=89.2$ [2602.06552]. Third, subspace- and channel-level compression can preserve specialization at sharply reduced storage: Sub-MoE reports 96% and 86% of original Mixtral-8x7B performance at 25% and 50% expert reduction, and Channel Merging reports results comparable to model ensemble with just 53% parameters when paired with a task-specific router [2506.23266] [2412.15283]. Fourth, search beyond ordinary convex merging can discover genuinely new behaviors: MERGEvolve reports best performance on 9 of 12 tasks and an emergent capability rate of 13.95% on ARC-C [2606.28373].

Common misconceptions are addressed directly by the literature. **Expert merging is not identical to standard MoE**: many systems operate post hoc on fine-tuned checkpoints rather than jointly learned experts and routers [2605.18904] [2502.00997]. **Not all residual spectral mass is noise**: ResMerge shows that for RL task vectors, both leading heads and residuals can independently recover substantial behavior knowledge [2606.02252]. **More experts do not automatically improve a merge**: large models and strong bases make merging easier, but the scale study finds that weak bases or smaller models suffer significant held-in degradation as the number of merged experts rises [2410.03617]. **Data-free is not universal**: some of the strongest methods use unlabeled or lightly labeled calibration data rather than none at all [2509.25712] [2602.06552].

A plausible near-term direction is the combination of these strands: differentiable capacity allocation, reusable modular libraries, theory-guided weighting, and lightweight input-aware routing within a single expert-merging pipeline. The literature already suggests the ingredients—data-free delta reconstruction, output-level compression, chunk-wise importance allocation, bargaining or curvature-aware propagation, and subspace alignment across heterogeneous experts—but their unification remains an open systems problem rather than a settled recipe [2605.18904] [2510.14436] [2510.16138].

Source: https://www.emergentmind.com/topics/expert-merging-8ebffe4f-63d6-404d-bf2a-1a3ce7929494