Model MoErging: Dynamic Expert Integration

Updated 2 July 2026

Model MoErging is a research paradigm that combines independently trained expert models using dynamic routing and merging strategies to achieve modular and decentralized generalization.
It leverages parameter-space techniques like permutation alignment and low-rank decomposition to mitigate conflicting updates and enhance rapid integration of new capabilities.
Hybrid approaches, including semantic global routers and black-box merging, extend MoErging to heterogeneous and multimodal systems, optimizing performance under varied operational settings.

Model MoErging is a research paradigm and practical family of methods that create aggregate systems by recycling and combining independently trained “expert” models—commonly parameter-efficient adapters or fine-tuned checkpoints—via dynamic selection and/or merging strategies. Distinct from classic model ensembling or multitask training, MoErging constructs a routed or merged model that enables modular, decentralized generalization, efficient reuse of previous computation, and rapid inclusion of new capabilities without retraining an entire monolithic model (Yadav et al., 2024).

1. Foundational Concepts and Taxonomy

Model MoErging operates on the principle that a pool of expert models $E_1,\dots, E_K$ , each trained for a domain or task, can be leveraged through a “router” function $r(x;\phi)$ that selects or weighs experts for a given input $x$ . The “merger” then aggregates the selected experts’ representations, either in prediction space (output ensembling) or in parameter space (weight merging), often modulated by downstream fine-tuning or regularization (Yadav et al., 2024).

Key system components:

Expert pool: Specialist models or adapters.
Router: Function producing a distribution or selection over experts conditioned on input.
Merger: Mechanism for aggregating selected experts’ outputs or parameters.
Fine-tuning (optional): Refines router/merger parameters post-merging.

The design taxonomy spans three levels:

Expert Model Design: Training scheme, data privacy, and reuse.
Routing Design: Routing dataset (none/target/expert/general), input and depth granularity (task-/example-/token-level; per-model/per-module), expert selection (sparse/dense), and aggregation method (output/parameter/none).
Application Design: Generalization regime (ID/OOD), data regime (zero-shot/few-shot/full), and specific adaptation requirements (Yadav et al., 2024).

2. Parameter-Space Model MoErging: Merging Algorithms and Theoretical Foundations

Most parameter-space MoErging methods operate by directly combining model weights. Recent work extends naïve averaging (e.g., Model Soups) to address permutations, conflicting updates, interference, and low-rank structure:

Permutation Alignment: Cycle-consistent merging (C $^2$ M $^3$ ) applies Frank-Wolfe optimization over the convex hull of permutation matrices, aligning independently seeded networks so that averaging is semantically consistent (Crisostomi, 2 May 2026).
Low-rank Task Vector Decompositions: Task vectors $\tau_t=\theta_t-\theta_{\text{pre}}$ (the difference between specialist and base models) inherit the low-rank structure of neural gradients. Task Singular Vectors (TSV) capture the major modes of variation, and merged models can be built by projecting expert deviations onto the TSV subspace for compression and interference reduction (Crisostomi, 2 May 2026).
Subspace and SVD Routing: Approaches like MASS build adaptive, input-dependent routers that select task-relevant subspaces based on geometric proximity in feature space; these are constructed via SVD on parameter updates, enabling near-oracle multitask performance with moderate storage/inference overhead (Crisostomi et al., 6 Apr 2025, Crisostomi, 2 May 2026).

A family of algorithms applies evolutionary search to identify optimal convex (or non-convex) combinations of expert parameters, including MERGE³ (with IRT-based evaluation minimization) and MERGEvolve, which demonstrate both theoretical guarantees (escape from the convex hull of initial experts) and strong empirical multitask generalization (Wang et al., 17 Jun 2026, Crisostomi, 2 May 2026).

In MoE models, a small router network determines sparse token-level expert selection (e.g., top- $k$ out of $K$ experts via softmax + top- $k$ ). Traditional linear merging or arithmetic on router parameters catastrophically fails, a failure mode termed “routing breakdown.” This occurs because:

The softmax and top- $k$ operations are highly sensitive to small parameter perturbations—softmax’s Hessian amplifies errors especially when pretraining encourages uniform routing.
Top- $r(x;\phi)$ 0 selection is discrete; even minute changes to logits can alter expert assignation for the majority of tokens.
Empirically, over 50% of expert selections differ after naïve merging at early layers, and >99% at deep layers, leading to severe downstream performance loss (Huang et al., 2 Jun 2026).

Hessian-Aware Router Calibration (HARC) addresses this failure by performing a second-order (curvature-aware) calibration of merged router parameters. Rather than retraining, HARC fits the merged router to minimize the KL divergence between the merged and each source router’s routing distributions, weighted by the Hessian of the softmax at the local logit configuration. This admits a closed-form, efficiently solvable via matrix-free conjugate gradient, and restores expert selection fidelity without backpropagation (Huang et al., 2 Jun 2026).

4. Adaptive Routing, Semantic Routers, and Hybrid MoErging

Token-level MoE routing and static parameter merging are sometimes insufficient for practical generalization and held-in task performance. Recent advances introduce multi-scale and hybrid routing mechanisms:

Semantic Global Routers: GLIDER utilizes LLM-generated task instructions as global embeddings, allowing retrieval-based expert selection by semantic context. The global signal is combined with local, token-level routing based on cosine similarities between activation vectors and expert module-specific routing vectors (Li et al., 2024).
Input-Adaptive, Data-Free Routers: Methods like MASS and MergeME use geometric or perplexity-based scores to select among task subspaces or experts without additional data or fine-tuning. MergeME further expands to heterogeneous architectures using learned projectors to unify feature spaces and routing at the sequence-level (Crisostomi et al., 6 Apr 2025, Zhou et al., 3 Feb 2025).
Representation-Aware Modality Merging: ES-Merging for MLLMs fuses specialized adapters by evaluating layer-wise and element-wise modality-specific activation differences, estimating merging coefficients from embedding space rather than weight magnitude. This explicitly preserves both specialization and cross-modal generalization (Lee et al., 15 Mar 2026).

These hybrid formation and routing techniques are validated across T5, LLaMA, and CLIP-based systems, demonstrating gains on both held-in and OOD benchmarks (Li et al., 2024, Crisostomi et al., 6 Apr 2025, Zhou et al., 3 Feb 2025).

5. MoErging in Black-Box and Heterogeneous Settings

Recent research addresses scenarios where model weights are inaccessible or expert architectures diverge:

Black-Box Merging (BMM): When models are only accessible via API inference (LMaaS), Evo-Merging applies derivative-free evolutionary algorithms (CMA-ES) to jointly select sparsity masks and scaling weights, using validation performance as the only signal. Sparsity-based denoising and sign-aware scaling allow effective combination of a large pool of adapters, with rigorous bounds on performance preservation and convergence guarantees (Chen et al., 16 Sep 2025).
Merging Heterogeneous Experts: MergeME utilizes input/output projectors and a shared vocabulary embedding/head to unify experts of varying hidden sizes and depths, performing sequence-level routing to combine features (Zhou et al., 3 Feb 2025).

These strategies expand MoErging beyond the original context of homogeneous, white-box weights.

6. Compression-Oriented MoErging for MoE Models

Model MoErging is also deployed for efficient compression of immense MoE models:

Expert Output Merging: MergeMoE interprets compression as learning low-rank matrices $r(x;\phi)$ 1 to project the high-dimensional output combination of experts to a lower-dimensional merged block, optimized via least-squares fitting to preserve routed output geometry (Miao et al., 16 Oct 2025).
Subspace Expert Merging: Sub-MoE clusters functionally similar experts and applies joint SVD to extract shared subspaces, merging only right-singular (output) factors using router-inferred activation frequencies as weights. Intra-expert compression via activation-aware subspace truncation yields further gains, with empirical retention of up to 96% performance at high compression ratios (Li et al., 29 Jun 2025).

Both approaches target scalable deployment of massive MoEs, with clear superiority over naïve averaging, usage-based, or prior pruning methods.

7. Challenges, Evaluation, and Future Directions

Open challenges in MoErging include:

Systematic Benchmarking: Lack of unified experimental protocols hampers cross-method comparison; multi-domain, multi-task, and OOD evaluation suites are needed (Yadav et al., 2024).
Robustness and Security: Mitigating parameter interference, expert redundancy, and potential adversarial or poisoned experts is an unsolved research problem (Zhou et al., 3 Feb 2025, Yadav et al., 2024).
Continual and Federated MoErging: Methods must support dynamic expert addition/removal and privacy-aware routing, possibly integrating federated and personalized MoE techniques (Yadav et al., 2024).
Theory of Curvature-Aware Routing: Better understanding of the non-Euclidean structure induced by softmax+top- $r(x;\phi)$ 2 and other nonlinearities in sparse architectures may yield improved merging and calibration frameworks (Huang et al., 2 Jun 2026).
Scaling to Arbitrary Architectures and Multi-modal Fusions: Strong demand exists for methods merging diverse modalities, architectures, and black-box services (Lee et al., 15 Mar 2026, Chen et al., 16 Sep 2025).

Recent empirical results demonstrate that state-of-the-art MoErging methods recover up to 98% of the per-task performance of individually fine-tuned models, surpass heuristic- and parameter-magnitude-based merging by significant margins across image, text, and multimodal tasks (Crisostomi et al., 6 Apr 2025, Lee et al., 15 Mar 2026).

The MoErging paradigm thus advances weight-space model composition, modular upgradeability, scalable deployment, and federated collaboration in deep learning, by unifying advances in routing, parameter alignment, subspace projections, black-box optimization, and hybrid signal integration across architectures and modalities (Yadav et al., 2024, Crisostomi, 2 May 2026, Miao et al., 16 Oct 2025, Huang et al., 2 Jun 2026, Li et al., 29 Jun 2025, Li et al., 2024, Lee et al., 15 Mar 2026).