Evo-Merging: Evolutionary Model Merging

• Evo-Merging is a family of methods that leverages evolutionary search to optimize model merging recipes, combining independently trained models without full retraining.
• It employs genetic operators such as mutation, crossover, and recombination over both parameter and data flow spaces to explore non-intuitive, task-specific merge strategies.
• The approach enables efficient multi-objective optimization, yielding enhanced cross-domain, multilingual, and storage-efficient model performance.

Searching arXiv for papers on Evo-Merging and related evolutionary model merging frameworks. arXiv search query: Evo-Merging model merging evolution evolutionary model merging MERGE3 EvoGM MERGEvolve Evo-Merging denotes a family of methods that formulates model merging as an evolutionary search problem over merge recipes rather than as a fixed arithmetic rule. In this literature, candidate merged models are treated as individuals; fitness is measured on validation tasks; and new candidates are produced by operations such as coefficient mutation, crossover in weight space, layer or module recombination, sparsification, or learned proposal generation. The core objective is to combine independently trained or fine-tuned models into stronger composite systems without full retraining, while expanding the search beyond hand-designed convex interpolations and manual recipe selection (Akiba et al., 2024).

1. Historical emergence and conceptual core

Early work framed the idea in explicitly neuroevolutionary terms. "EvoMerge: Neuroevolution for LLMs" treats weight crossover as model merging and weight mutation as further fine-tuning, yielding a six-step cycle of initialization, evaluation, selection, crossover, mutation, and repetition (Jiang, 2024). In that formulation, population-based search replaces a single sequential fine-tuning trajectory, and merging becomes the analogue of biological recombination.

A more general formulation appeared in "Evolutionary Optimization of Model Merging Recipes" (Akiba et al., 2024). That work treats model composition as an optimization problem over two orthogonal axes: Parameter Space (PS), which searches how weights are combined, and Data Flow Space (DFS), which searches how layers from different source models are selected and ordered during inference. This establishes Evo-Merging not merely as weighted averaging, but as an automated search procedure over model composition rules.

The conceptual shift is from manual to evolved composition. Fixed arithmetic methods such as uniform interpolation, task arithmetic, or layerwise heuristics remain within a narrow design space. Evo-Merging instead assumes that high-performing merge recipes may be non-intuitive, task-dependent, and only discoverable through repeated evaluation and adaptation. This suggests that the principal object of optimization is not the source models themselves, but the recipe by which their capabilities are composed.

2. Search spaces, representations, and formal problem structure

In parameter-space merging, the genotype typically consists of layerwise or modelwise coefficients that determine how expert checkpoints are blended. A canonical example is the layerwise interpolation rule

$$\mathbf{W}_{\text{merged}}^{(l)} = \alpha^{(l)} \mathbf{W}_A^{(l)} + (1 - \alpha^{(l)}) \mathbf{W}_B^{(l)},$$

with one coefficient per layer or component (Akiba et al., 2024). This regime is natural when models share a common architecture and initialization, and it subsumes recipes based on linear interpolation, model soups, TIES-Merging, and DARE.

In data flow space merging, the genotype no longer specifies only coefficients. Instead, it encodes which layers from which source models are active, and how transitions between heterologous layers are scaled. In the formulation of (Akiba et al., 2024), DFS uses an indicator array $\mathcal{I}$ and a scaling matrix $\mathbf{W}$, or alternatively a neural parameterization $\pi_\theta(i,j,t)$, with inter-layer transfer written as

$$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$

This allows "Frankenmerge" constructions in which input/output embeddings and some boundary layers remain fixed while internal layers are selected, skipped, or reordered.

Subsequent work formalized DFS as a mixed binary-continuous black-box optimization problem. "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging" writes the problem as

$$\min_{\mathbf{x}, \mathbf{z}} f(\mathbf{x}, \mathbf{z}) \quad \text{subject to} \quad \mathbf{x} \in \mathbb{R}^n, \; \mathbf{z} \in \{0,1\}^m,$$

where $\mathbf{z}$ selects layers and $\mathbf{x}$ controls scaling weights (Niloy, 12 May 2026). The paper emphasizes conditional dependencies: only the continuous variables associated with active binary selections matter. In its preliminary empirical study, a structured DFS optimizer that respected this dependency outperformed an unstructured one by 6.7% accuracy while reducing the effective search space by 51.4% (Niloy, 12 May 2026). This result suggests that DFS is not merely high-dimensional; it is structurally mismatched to optimizers designed for unconditional continuous domains.

A standard taxonomy now distinguishes three categories: PS merging, DFS merging, and hybrid approaches that optimize both (Niloy, 12 May 2026). Hybrid strategies often first evolve a weight-space merge and then reuse the resulting merged models as building blocks for DFS search, or vice versa. A plausible implication is that Evo-Merging should be understood less as a single algorithm than as a design space of evolutionary search problems over multiple compositional representations.

3. Evolutionary operators and objective functions

The basic evolutionary loop remains close to classical evolutionary computation: initialize a population, evaluate fitness, select parents, generate offspring, and iterate (Jiang, 2024). What varies across Evo-Merging methods is the definition of crossover, mutation, and fitness.

For crossover, one recurrent operator is Spherical Linear Interpolation (SLERP): $$\text{SLERP}(\mathbf{w}_1, \mathbf{w}_2; t) = \frac{\sin((1-t)\theta)}{\sin \theta} \mathbf{w}_1 + \frac{\sin(t\theta)}{\sin \theta} \mathbf{w}_2,$$ with

$$\theta = \arccos{\left( \frac{\langle \mathbf{w}_1, \mathbf{w}_2 \rangle}{\| \mathbf{w}_1 \| \| \mathbf{w}_2 \|} \right)}.$$

In EvoMerge, SLERP is recommended for weight crossover, though TIES and DARE are also identified as viable alternatives (Jiang, 2024). For mutation, EvoMerge uses further fine-tuning, including DPO fine-tuning, as the source of post-crossover variation (Jiang, 2024).

Later systems broadened the objective beyond a single scalar benchmark. MERGE formulates component-wise merging as a bi-objective optimization over cross-task performance $\mathcal{I}$0 and storage cost $\mathcal{I}$1, and uses NSGA-II with a Random Forest surrogate model to identify Pareto-optimal merging configurations (Qiu et al., 6 Feb 2026). Evo-L2S likewise uses a bi-objective formulation, but with the axes Pass@1 accuracy and output length, defining fitness as

$\mathcal{I}$2

It uses NSGA-II, SBX, and Polynomial Mutation to construct a Pareto front of models trading off accuracy and reasoning length (Iacobelli et al., 7 Apr 2026).

Fitness shaping has also incorporated sparsity and conflict handling. "Sparsity-Aware Evolution for Model Merging" defines a layerwise mixing ratio

$\mathcal{I}$3

with merged weights

$\mathcal{I}$4

Here, $\mathcal{I}$5 are performance scores and $\mathcal{I}$6 are sparsity-induced weights (Zhang et al., 9 Feb 2026). The method adds iterative pruning-merging cycles and argues that competition for sparsity creates a local "attraction" effect, because zero positions in one parent can be occupied by non-zero parameters from the other (Zhang et al., 9 Feb 2026).

These formulations show that Evo-Merging objectives now routinely combine task performance with secondary criteria such as storage, sparsity, or inference length. This suggests a move away from one-dimensional "best merge" thinking toward Pareto optimization over deployment-relevant trade-offs.

4. Efficient fitness evaluation and search tractability

A central practical obstacle is the cost of repeatedly evaluating merged candidates. MERGE$\mathcal{I}$7 addresses this by reducing evaluation to a small subset of examples and using Item Response Theory (IRT) to estimate performance on the full dataset (Mencattini et al., 9 Feb 2025). Its three stages are summarized as Extract, Estimate, and Evolve: extract a reduced dataset $\mathcal{I}$8, estimate endpoint model abilities with multidimensional IRT, and evaluate evolutionary candidates with IRT-based performance estimators such as mp-IRT and gmp-IRT (Mencattini et al., 9 Feb 2025).

The IRT model is written as

$\mathcal{I}$9

and merged-model ability is assumed to be a convex combination of endpoint abilities: $\mathbf{W}$0 MERGE$\mathbf{W}$1 reports a 50$\mathbf{W}$2 reduction in fitness computation cost while preserving performance, making evolutionary merging feasible on a single consumer GPU (Mencattini et al., 9 Feb 2025). The paper also provides $\mathbf{W}$3-stability and asymptotic-unbiasedness guarantees for subset-based estimation (Mencattini et al., 9 Feb 2025).

Evo-L2S adopts a different tractability strategy. Rather than IRT, it constructs a fixed evaluation subset of $\mathbf{W}$4 high-entropy items, using correctness statistics from $\mathbf{W}$5 calibration merges and Bernoulli entropy

$\mathbf{W}$6

to identify the most discriminative problems (Iacobelli et al., 7 Apr 2026). The paper reports that entropy-based subsets achieve higher Spearman rank correlation with full-benchmark rankings than random or disagreement-based sampling (Iacobelli et al., 7 Apr 2026).

These methods indicate that evolutionary search is often limited less by the merge operator than by the evaluation loop. A plausible implication is that future progress in Evo-Merging may depend as much on fitness surrogates and evaluation subset design as on new genetic operators.

5. Major variants and specialized frameworks

The diversification of Evo-Merging has produced several distinct frameworks.

Framework	Distinctive mechanism	Reference
Evo-Merging	PS and DFS recipe evolution with CMA-ES	(Akiba et al., 2024)
MERGE$\mathbf{W}$7	IRT-accelerated evolutionary search	(Mencattini et al., 9 Feb 2025)
MERGEvolve	merge initialization followed by ES exploration	(Wang et al., 17 Jun 2026)
MERGE	component-wise modular expert recombination	(Qiu et al., 6 Feb 2026)
Evo-L2S	multi-objective merging for accuracy vs. output length	(Iacobelli et al., 7 Apr 2026)
EvoGM	dual-generator, cycle-consistent generative search	(Jiang et al., 28 May 2026)
Black-box Evo-Merging	API-only merging with sparsity denoising and sign-aware scaling	(Chen et al., 16 Sep 2025)
SAE	sparsity-aware evolutionary pruning-merging cycles	(Zhang et al., 9 Feb 2026)

MERGEvolve unifies static model merging and evolution within a single evolution strategy. Its initialization is

$\mathbf{W}$8

followed by iterative ES updates

$\mathbf{W}$9

The paper’s theoretical analysis states that the random perturbation term allows exploration outside both the convex and affine hulls spanned by the experts, and its ablations show that removing the merging-based initialization causes drastic drops, including -69.48% on Flores-101 (Wang et al., 17 Jun 2026).

MERGE moves from whole-model recipes to component-wise merging. It represents a configuration as a matrix $\pi_\theta(i,j,t)$0, where $\pi_\theta(i,j,t)$1 specifies the group assignment of the $\pi_\theta(i,j,t)$2-th component of task $\pi_\theta(i,j,t)$3's model (Qiu et al., 6 Feb 2026). Pareto-optimal configurations are converted into a reusable modular expert library

$\pi_\theta(i,j,t)$4

and a lightweight routing network dynamically recombines modular experts at inference time (Qiu et al., 6 Feb 2026). The paper reports that MERGE consistently outperforms strong baselines while providing diverse performance-storage trade-offs (Qiu et al., 6 Feb 2026).

EvoGM replaces hand-crafted mutation in coefficient space with a learned proposal mechanism. It defines task vectors $\pi_\theta(i,j,t)$5 and merged parameters

$\pi_\theta(i,j,t)$6

Its search model uses dual generators $\pi_\theta(i,j,t)$7 and $\pi_\theta(i,j,t)$8, trained on winner-loser pairs from historical trajectories with cycle-consistency and optimization-guided losses (Jiang et al., 28 May 2026). The resulting multi-round pipeline periodically refreshes the expert basis with elite merged models, so the search space itself evolves (Jiang et al., 28 May 2026).

Black-box Evo-Merging addresses Language-Model-as-a-Service settings where parameters are inaccessible. It performs two CMA-ES stages over LoRA adapters: sparsity-based denoising, which sparsifies $\pi_\theta(i,j,t)$9 to $$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$0, and sign-aware scaling, which assigns real-valued, possibly negative coefficients $$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$1 (Chen et al., 16 Sep 2025). The merged adapter is

$$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$2

Theoretical analysis in that paper focuses on asymmetric sparsification, arguing for sparsifying the LoRA-A matrix rather than the LoRA-B matrix (Chen et al., 16 Sep 2025).

SAE is best read as a mutation and scoring extension rather than a separate representation regime. By making sparsity a first-class part of selection, it seeks more reliable model merging and smoother merge landscapes (Zhang et al., 9 Feb 2026).

6. Empirical profile, capabilities, and unresolved issues

Empirically, Evo-Merging has been used to demonstrate cross-domain transfer, multilingual composition, storage-efficient reuse, and reasoning-cost reduction. In the Japanese Math LLM experiment of (Akiba et al., 2024), source models showed MGSM-JA accuracy $$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$3, whereas evolved merges reached 52.0% in PS, 36.4% in DFS, and 55.2% in PS+DFS. The same study reports that the resulting 7B merged models exceeded larger Japanese baselines such as Japanese StableLM 70B: 17.2% and Swallow 70B: 13.6% on MGSM-JA (Akiba et al., 2024). The paper further states that only the merged models answered questions requiring both Japanese cultural knowledge and mathematical reasoning (Akiba et al., 2024).

MERGE$$\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).$$4 emphasizes accessibility rather than new search spaces. It reports that standard full-dataset evolutionary merging would require over a month on a single NVIDIA 4090 for 1,000 trials, whereas its reduced-evaluation pipeline runs in under a day while preserving merge quality (Mencattini et al., 9 Feb 2025). It also reports 10–20% higher accuracy than TIES, Task Arithmetic, and DARE in cross-lingual math transfer, and up to 19% improvement over individual language-specific fine-tunes on multilingual ARC QA (Mencattini et al., 9 Feb 2025).

Evo-L2S demonstrates a different capability profile: explicit control of reasoning length. On the 1.5B scale, the System 2 baseline achieved 45.5% accuracy with 3706 tokens, while the best Evo-L2S point achieved 50.1% accuracy and 52.5% shorter output. On the 7B scale, the baseline achieved 59.2% accuracy with 4548 tokens, and the best Evo-L2S point reached 59.6% accuracy with 39% shorter output (Iacobelli et al., 7 Apr 2026). In that setting, evolutionary multi-objective merging is presented as less brittle than fixed-hyperparameter arithmetic methods (Iacobelli et al., 7 Apr 2026).

Several papers also emphasize robustness to noisy or heterogeneous expert pools. MERGEvolve reports state-of-the-art results on 9 out of 12 single-task benchmarks and defines an Emergent Capability Rate (ECR) in which evolved models solve a non-trivial fraction of problems that none of the initial experts solved, with values up to 14% (Wang et al., 17 Jun 2026). Black-box Evo-Merging reports state-of-the-art results on information extraction tasks with over 100 LoRA adapters and, on its out-of-domain setting, reports 52.1 F1 versus 40.9 F1 for the best baseline (Chen et al., 16 Sep 2025). SAE reports 0.798 on GSM8K and 0.484 average score for its global variant, compared with 0.7801 and 0.472 for PSO (Zhang et al., 9 Feb 2026).

The main unresolved issues are now well defined. DFS merging remains a high-dimensional mixed-variable problem with conditional dependencies that standard optimizers were not designed to handle (Niloy, 12 May 2026). Some frameworks still assume homogeneous architectures or shared initializations; EvoGM explicitly notes this as a current limitation (Jiang et al., 28 May 2026). Other open directions listed in the literature include extending MERGEvolve to heterogeneous model architectures, designing adaptive perturbation scales and utility weighting during evolution, and developing optimizers that directly respect mixed binary-continuous structure (Wang et al., 17 Jun 2026). This suggests that the next phase of Evo-Merging research is likely to be shaped by optimizer design, evaluation efficiency, and broader architectural compatibility rather than by simple increases in population size or trial count.