Expert-Merging Procedures

Updated 17 March 2026

Expert-merging procedures are techniques that integrate multiple task-specialized models into a unified system, enhancing multi-task performance without extensive retraining.
They employ methods ranging from simple arithmetic averaging and pruning to advanced data-driven, game-theoretic, and modular optimization to balance task-specific biases and resource efficiency.
Practical applications include scalable multi-task inference in language, vision, and multimodal systems, achieving improved accuracy and compressed memory usage.

Expert-merging procedures integrate the knowledge and inductive biases of multiple task-specialized models (“experts”) into a single parameter vector or deployable model, with the goal of achieving strong multi-task generalization while minimizing retraining, data access, or resource overhead. This area spans data-independent merging (simple arithmetic), data-driven and gradient-based consolidation, meta-learned coefficient search, curvature-aware geometry, game-theoretic and evolutionary methods, structure-aware merging (e.g., channel or modular), and scalable or compressed merging for extreme settings. Below, expert-merging is investigated through key algorithmic classes, mathematical underpinnings, scaling and memory considerations, representative algorithms, and comparative analysis, drawing on recent results from both large-scale and fine-grained regime studies.

1. Classical Weight-Space and Task-Vector Merging

The foundational approach involves algebraic layer-wise operations on expert model parameters, nearly always fine-tuned from a shared base $\theta_0$ . Let $\{\theta_i\}_{i=1}^N$ denote $N$ expert models, with task vectors $\tau_i = \theta_i - \theta_0$ .

Parameter Averaging: $\theta_m = \frac{1}{N}\sum_{i=1}^N \theta_i$ ; ignores base and task differences (Yadav et al., 2024).
Task Arithmetic: $\theta_m = \theta_0 + \alpha \sum_{i=1}^N (\theta_i - \theta_0)$ , $\alpha$ tunable; provides direct control over total update magnitude (Yadav et al., 2024, Sun et al., 29 May 2025).
Support-Pruned and Masked Merging:
- TIES: Prune each $\tau_i$ by thresholding; average signs for each parameter and aggregate only those with majority agreement, mitigating destructive interference from conflicting updates (Yadav et al., 2024, Wei et al., 26 May 2025).
- DARE-TIES: Combine TIES with random coordinate drop and rescale (“Drop-And-REscale”) for noise robustness (Yadav et al., 2024, Skorobogat et al., 19 Jun 2025).

Operationally, these methods are stateless, data-free, and computationally trivial. While effective in “local” regimes (experts closely related in weight space), performance degrades with increased task divergence or number of experts due to interference and unmodeled parameter conflicts (Sharma et al., 2024, Skorobogat et al., 19 Jun 2025). They remain the baseline of reference for benchmarking more advanced merging procedures.

2. Data-Driven and Feature-Based Merging

Data-driven approaches utilize calibration or validation sets (typically small), introducing data-dependent criteria into the merging process for improved multi-task alignment.

PSO-Merging: Leverages Particle Swarm Optimization over the full parameter space. The population is initialized with the base, full experts, and sparsified deltas. The merger proceeds via PSO steps with fitness evaluated as the average task score on a validation set. This enables gradient-free but data-informed optimization, efficiently traversing the high-dimensional space to identify parameter configurations that balance multi-task performance (Zhang et al., 27 Aug 2025).
LOT Merging (Layer-wise Optimal Task Vector Fusion): Recognizes that layerwise “feature drift” — the difference in intermediate representations between experts and the merged model — tightly correlates with downstream loss. For each layer, a closed-form convex quadratic program minimizes the sum of squared layer-feature deviations over all tasks; for linear or normalization layers, this reduces to simple matrix operations. Empirically, this layerwise approach outperforms parameter-aligned or loss-minimizing baselines, particularly in vision and vision-language settings (Sun et al., 29 May 2025).

These methods generally require only a handful of examples per task (often $N \le 64$ ), perform no backpropagation, and are practical for models with $10^7$ – $\{\theta_i\}_{i=1}^N$ 0 parameters.

3. Advanced Conflict and Interference Mitigation

Several procedures have been devised to specifically resolve interference and parameter conflicts that arise when naively aggregating divergent expert weights.

Channel Merging: Instead of merging entire layers or modules, channels (rows) across experts are clustered using a similarity metric (e.g., cosine). Only channels deemed “similar” are merged (by averaging), while distinct channels are stored separately. An index maps each expert to a specific subset of merged groups. A lightweight router determines at inference which expert chain to reconstruct via channel lookup, ensuring storage cost is $\{\theta_i\}_{i=1}^N$ 1 (with $\{\theta_i\}_{i=1}^N$ 2 clusters) relative to a full ensemble and preserving per-expert specialization even under high compression (Zhang et al., 2024).
Subspace-Boosted Merging: Addresses “rank collapse” as the number of merged experts increases, where the spectrum of the summed task vectors becomes low-rank, limiting multi-task expressivity. After SVD decomposition of layerwise deltas, tail singular values are elevated to a minimum threshold (“boosted”), preserving signal in otherwise vanished directions. Higher-order GSVD further enables interpretable quantification of expert alignment for subset selection (Skorobogat et al., 19 Jun 2025).
ACE-Merging: Derives a closed-form data-free solution for merging by adaptively estimating the input covariance (implicitly, via the empirical second moment of each task delta) and using its inverse as a precision matrix for weighted averaging. This maximizes the likelihood under a local Gaussian approximation and provably reduces inter-task interference, subject to Tikhonov/spectral regularization for numerical stability. Typical gains over previous iterative or heuristic methods exceed 4–12% on both vision and NLP tasks (Xu et al., 3 Mar 2026).
SurgeryV2: Observes the problem of “representation bias” — a persistent distribution gap between expert and merged intermediate activations at every layer that is not resolved by simple final-layer alignment. It introduces per-task, per-layer adapters $\{\theta_i\}_{i=1}^N$ 3 optimized (unsupervised, on calibration data) to closely match expert and merged representations throughout the network, eliminating cumulative bias and closing the gap to joint multi-task learning (Yang et al., 2024).

These approaches address the central challenge of “catastrophic interference” and loss of task specificity in deeply specialized regimes.

4. Structure- and Modularity-Aware Merging Algorithms

Recent work recognizes that functional modularity exists within deep architectures.

MERGE (Modular Expert Recombination): Decomposes every expert model into homologous components (e.g., attention heads, MLPs) and formulates a bi-objective optimization (utility vs. compression) over how to share or merge components across tasks. A surrogate-assisted evolutionary algorithm finds Pareto-optimal merging configurations. These modular experts are stored in a library; at inference, a lightweight router assembles task- or input-specific models by chaining appropriate modules, supporting dynamic recombination under memory constraints (Qiu et al., 6 Feb 2026).
Expert Merging++: Builds on previous layerwise coefficient optimization by introducing importance-guided chunking, assigning more fine-grained merging coefficients to high-importance layers determined via learned weights, delta magnitudes, and parameter counts (Zhang et al., 30 Sep 2025).
PuzzleMoE/Bit-Packed Sparse Merging: For compression, introduces element-wise dual masks (similarity + saliency) that select, for each parameter across experts, whether to share, select, or separately retain. Merged weights, masks, and sign bits are packed into underused exponent bits (in bfloat16) for jointly compressed storage and rapid inference with custom kernels (Zhao et al., 6 Nov 2025).

Such fine-grained and modular procedures maximize the utility of partial sharing, enable task decomposition, and reduce both memory and inference costs far beyond naïve block-level approaches.

5. Evolutionary, Game-Theoretic, and Optimization-Based Merging

Population-based optimization and game-theoretic criteria have recently been leveraged for robust, interference-aware expert merging.

PSO-Merging: Uses classical Particle Swarm Optimization over the entire parameter space, where each “particle” is a candidate model (base, experts, sparsified versions). Fitness is evaluated over multiple tasks; velocity and position updates combine personal and social bests. This method is highly scalable, requires only forward passes, and consistently outperforms purely gradient-based or evolutionary baselines on large LMs and multitask benchmarks (Zhang et al., 27 Aug 2025).
Nash Bargaining Merging (NAMEx): Frames merging as a Nash bargaining game: optimizing an update within the span of experts’ domain vectors $\{\theta_i\}_{i=1}^N$ 4 to maximize the geometric mean of utilities (aligned “gain”) on each task. The resulting closed-form nonlinear system yields balanced merging weights $\{\theta_i\}_{i=1}^N$ 5, with extensions to complex momentum and curvature-aware schemes that guarantee convergence and performance gains in sparse MoEs (Nguyen et al., 17 Oct 2025).
MergeMoE/Output-Merging: Treats merging as minimizing the difference in expert outputs, not parameters. The framework introduces linear mixing and compression (with matrices $\{\theta_i\}_{i=1}^N$ 6) directly into the model’s forward computation. The optimization (over group assignments and least-squares fits for output composition) achieves state-of-the-art results for high-ratio MoE compression (Miao et al., 16 Oct 2025).
Tanbr/Adaptive Neural Bandit Routers: In the online inference regime, a tree-structured bandit partitions the continuous space of expert merging weights; a neural network estimates rewards, balancing memory/performance/latency tradeoffs and dynamically adapting to unknown, time-varying task distributions, with $\{\theta_i\}_{i=1}^N$ 7 regret (Han et al., 24 Sep 2025).

These methods are particularly effective when model structure, scalability, and task heterogeneity pose nontrivial optimization or alignment challenges.

6. Scaling Properties, Memory, and Practical Considerations

Scaling behavior, memory use, and computational efficiency are central in evaluating merging procedures for modern LLMs and vision models.

Scaling Behavior: Larger models (e.g., 64B) are consistently easier to merge, with performance approaching oracle expert quality even under naïve averaging. Advanced methods (e.g., TIES, Dare, output-merging) yield small or negligible further improvements at very large scales (Yadav et al., 2024).
Memory and Inference: Merging methods such as PSO-Merging and Channel Merging sidestep backpropagation and store only a minimal number of intermediate models or channel groups, enabling multi-expert schemes to fit within 14–20 GB even for 7B–13B models (Zhang et al., 27 Aug 2025, Zhang et al., 2024). Advanced methods (e.g., PuzzleMoE) compress experts and metadata to fit within single-GPU DRAM even at 50% parameter loads without degradation (Zhao et al., 6 Nov 2025).
Empirical Outcomes: State-of-the-art procedures (LOT Merging, ACE-Merging, Expert Merging++, Channel Merging) yield gains between 2–5% absolute over prior baselines on vision, language, and multimodal benchmarks (e.g., +4.4% on ViT-B/32, +2.5% on MLLMs). Modular recombination and fine-grained routing match or exceed oracle multitask models for similar storage (Sun et al., 29 May 2025, Qiu et al., 6 Feb 2026, Zhang et al., 30 Sep 2025, Zhang et al., 2024, Xu et al., 3 Mar 2026).

A summary of empirical findings is presented below.

Method	Backbone	# Experts	Δ Performance vs. Baseline	Storage Reduction (%)
PSO-Merging	Flan-T5, Llama	3–8	+1.7–4.6 points	N/A (single merged model)
LOT Merging	CLIP-ViT	8	+4.4% absolute	N/A
Channel Merging	Mistral-7B	4	≈ equal to full ensemble	53%
Expert Merging++	Mistral-7B, Qwen	3–5	+1–3% over baselines	N/A (single merged model)
Subspace Boosting	ViT-B/16	8–20	+10–15% over TA/TIES	N/A
PuzzleMoE	Mixtral-8x7B	8→4	+16.7% on MMLU	50%

All methods above have been validated on high-capacity LLMs/MLLMs or vision models in large-scale settings (Zhang et al., 27 Aug 2025, Sun et al., 29 May 2025, Zhang et al., 2024, Qiu et al., 6 Feb 2026, Zhang et al., 30 Sep 2025, Xu et al., 3 Mar 2026, Skorobogat et al., 19 Jun 2025, Zhao et al., 6 Nov 2025).

7. Limitations, Open Problems, and Future Directions

Despite progress, several fundamental challenges and limitations persist.

Interference Persistence: All methods incur some “tax” relative to joint multitask models or best-in-domain experts, particularly as the number/diversity of tasks increases or when merging models from highly non-convex, non-local basins (Sharma et al., 2024, Skorobogat et al., 19 Jun 2025).
Hyperparameter Tuning: Many methods depend on tuning merging coefficients ( $\{\theta_i\}_{i=1}^N$ 8 in TA, $\{\theta_i\}_{i=1}^N$ 9 in TIES/DARE), thresholds (e.g., SVD rank, channel clustering $N$ 0), or momentum terms. Current practice remains largely empirical.
Task/Layer Heterogeneity: Uniform or per-layer merging is sub-optimal. Importance-aware chunking, modular recombination, and integrated downstream alignment address this, but optimal division of capacity remains an open problem (Qiu et al., 6 Feb 2026, Zhang et al., 30 Sep 2025).
Scalability and Explainability: While merging hundreds of experts remains computationally tractable for certain methods, expressive capacity, interpretability, and optimal expert selection (via e.g. HO-GSVD) are areas of active research (Skorobogat et al., 19 Jun 2025).
Theoretical Guarantees: For certain regimes (ACE, Nash), provable optimality in a local Gaussian or game-theory sense is available; fully general, non-linear guarantees for network merging are lacking (Xu et al., 3 Mar 2026, Nguyen et al., 17 Oct 2025).
Non-Local and Cross-Modal Merging: Handling models diverged in both architecture and initialization, or merging across modalities, poses unique challenges. Some procedures handle this (e.g., MergeME, MERGE, OptMerge (Zhou et al., 3 Feb 2025, Qiu et al., 6 Feb 2026, Wei et al., 26 May 2025)), but scalability and universality must be further evaluated.

Continued work is anticipated on automating hyperparameter selection, hierarchical or compositional merges across modalities and tasks, and deeper integration with online and adaptive deployment platforms.

Summary:

Expert-merging procedures range from algebraic and pruning-based techniques to sophisticated data-driven, modular, subspace-boosted, and game-theoretic approaches, each addressing specific statistical, geometric, and operational challenges of model aggregation. As models and expert pools scale, practical and theoretical insights from these methods drive advances in multi-task generalization, efficient inference, storage compression, and dynamic deployment across domains, languages, and modalities.