---
title: 'Evo-Merging: Evolutionary Model Merging'
url: https://www.emergentmind.com/topics/evo-merging
type: topic
---

# Evo-Merging: Evolutionary Model Merging

Searching arXiv for papers on Evo-Merging and related evolutionary model merging frameworks.
arXiv search query: Evo-Merging model merging evolution evolutionary model merging MERGE3 EvoGM MERGEvolve
Evo-Merging denotes a family of methods that formulates model merging as an evolutionary search problem over merge recipes rather than as a fixed arithmetic rule. In this literature, candidate merged models are treated as individuals; fitness is measured on validation tasks; and new candidates are produced by operations such as coefficient mutation, crossover in weight space, layer or module recombination, sparsification, or learned proposal generation. The core objective is to combine independently trained or fine-tuned models into stronger composite systems without full retraining, while expanding the search beyond hand-designed convex interpolations and manual recipe selection [2403.13187].

## 1. Historical emergence and conceptual core

Early work framed the idea in explicitly neuroevolutionary terms. "EvoMerge: Neuroevolution for Large Language Models" treats **weight crossover** as model merging and **weight mutation** as further fine-tuning, yielding a six-step cycle of initialization, evaluation, selection, crossover, mutation, and repetition [2402.00070]. In that formulation, population-based search replaces a single sequential fine-tuning trajectory, and merging becomes the analogue of biological recombination.

A more general formulation appeared in "Evolutionary Optimization of Model Merging Recipes" [2403.13187]. That work treats model composition as an optimization problem over two orthogonal axes: **Parameter Space (PS)**, which searches how weights are combined, and **Data Flow Space (DFS)**, which searches how layers from different source models are selected and ordered during inference. This establishes Evo-Merging not merely as weighted averaging, but as an automated search procedure over model composition rules.

The conceptual shift is from *manual* to *evolved* composition. Fixed arithmetic methods such as uniform interpolation, task arithmetic, or layerwise heuristics remain within a narrow design space. Evo-Merging instead assumes that high-performing merge recipes may be non-intuitive, task-dependent, and only discoverable through repeated evaluation and adaptation. This suggests that the principal object of optimization is not the source models themselves, but the **recipe** by which their capabilities are composed.

## 2. Search spaces, representations, and formal problem structure

In **parameter-space merging**, the genotype typically consists of layerwise or modelwise coefficients that determine how expert checkpoints are blended. A canonical example is the layerwise interpolation rule
\[
\mathbf{W}_{\text{merged}}^{(l)} = \alpha^{(l)} \mathbf{W}_A^{(l)} + (1 - \alpha^{(l)}) \mathbf{W}_B^{(l)},
\]
with one coefficient per layer or component [2403.13187]. This regime is natural when models share a common architecture and initialization, and it subsumes recipes based on linear interpolation, model soups, TIES-Merging, and DARE.

In **data flow space merging**, the genotype no longer specifies only coefficients. Instead, it encodes which layers from which source models are active, and how transitions between heterologous layers are scaled. In the formulation of [2403.13187], DFS uses an indicator array \(\mathcal{I}\) and a scaling matrix \(\mathbf{W}\), or alternatively a neural parameterization \(\pi_\theta(i,j,t)\), with inter-layer transfer written as
\[
\mathbf{h}_{j} = W_{ij} \cdot f_{j}(\mathbf{h}_{i}).
\]
This allows "Frankenmerge" constructions in which input/output embeddings and some boundary layers remain fixed while internal layers are selected, skipped, or reordered.

Subsequent work formalized DFS as a mixed binary-continuous black-box optimization problem. "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging" writes the problem as
\[
\min_{\mathbf{x}, \mathbf{z}} f(\mathbf{x}, \mathbf{z}) \quad \text{subject to} \quad \mathbf{x} \in \mathbb{R}^n, \; \mathbf{z} \in \{0,1\}^m,
\]
where \(\mathbf{z}\) selects layers and \(\mathbf{x}\) controls scaling weights [2605.12326]. The paper emphasizes **conditional dependencies**: only the continuous variables associated with active binary selections matter. In its preliminary empirical study, a structured DFS optimizer that respected this dependency outperformed an unstructured one by **6.7% accuracy** while reducing the effective search space by **51.4%** [2605.12326]. This result suggests that DFS is not merely high-dimensional; it is structurally mismatched to optimizers designed for unconditional continuous domains.

A standard taxonomy now distinguishes three categories: **PS merging**, **DFS merging**, and **hybrid approaches** that optimize both [2605.12326]. Hybrid strategies often first evolve a weight-space merge and then reuse the resulting merged models as building blocks for DFS search, or vice versa. A plausible implication is that Evo-Merging should be understood less as a single algorithm than as a design space of evolutionary search problems over multiple compositional representations.

## 3. Evolutionary operators and objective functions

The basic evolutionary loop remains close to classical evolutionary computation: initialize a population, evaluate fitness, select parents, generate offspring, and iterate [2402.00070]. What varies across Evo-Merging methods is the definition of crossover, mutation, and fitness.

For **crossover**, one recurrent operator is **Spherical Linear Interpolation (SLERP)**:
\[
\text{SLERP}(\mathbf{w}_1, \mathbf{w}_2; t) = \frac{\sin((1-t)\theta)}{\sin \theta} \mathbf{w}_1 + \frac{\sin(t\theta)}{\sin \theta} \mathbf{w}_2,
\]
with
\[
\theta = \arccos{\left( \frac{\langle \mathbf{w}_1, \mathbf{w}_2 \rangle}{\| \mathbf{w}_1 \| \| \mathbf{w}_2 \|} \right)}.
\]
In EvoMerge, SLERP is recommended for weight crossover, though TIES and DARE are also identified as viable alternatives [2402.00070]. For **mutation**, EvoMerge uses further fine-tuning, including DPO fine-tuning, as the source of post-crossover variation [2402.00070].

Later systems broadened the objective beyond a single scalar benchmark. MERGE formulates component-wise merging as a **bi-objective optimization** over cross-task performance \(\mathcal{P}_{\mathbf{G}}\) and storage cost \(\mathcal{C}_{\mathbf{G}}\), and uses **NSGA-II** with a **Random Forest surrogate model** to identify Pareto-optimal merging configurations [2602.06552]. Evo-L2S likewise uses a bi-objective formulation, but with the axes **Pass@1 accuracy** and **output length**, defining fitness as
\[
F(M) = [-\text{Acc}(M),\ \text{Len}(M)].
\]
It uses **NSGA-II**, **SBX**, and **Polynomial Mutation** to construct a Pareto front of models trading off accuracy and reasoning length [2604.06465].

Fitness shaping has also incorporated sparsity and conflict handling. "Sparsity-Aware Evolution for Model Merging" defines a layerwise mixing ratio
\[
\lambda_r^{(l)} =
\frac{s_A + \omega_A^{(l)}}{(s_A + \omega_A^{(l)}) + (s_B + \omega_B^{(l)})},
\]
with merged weights
\[
\theta_{\mathcal{M}}^{(l)} =
\lambda_r^{(l)} \cdot \theta_A^{(l)} + (1 - \lambda_r^{(l)}) \cdot \theta_B^{(l)}.
\]
Here, \(s_A,s_B\) are performance scores and \(\omega_A^{(l)},\omega_B^{(l)}\) are sparsity-induced weights [2602.08218]. The method adds iterative pruning-merging cycles and argues that competition for sparsity creates a local "attraction" effect, because zero positions in one parent can be occupied by non-zero parameters from the other [2602.08218].

These formulations show that Evo-Merging objectives now routinely combine task performance with secondary criteria such as storage, sparsity, or inference length. This suggests a move away from one-dimensional "best merge" thinking toward Pareto optimization over deployment-relevant trade-offs.

## 4. Efficient fitness evaluation and search tractability

A central practical obstacle is the cost of repeatedly evaluating merged candidates. MERGE\(^3\) addresses this by reducing evaluation to a small subset of examples and using **Item Response Theory (IRT)** to estimate performance on the full dataset [2502.10436]. Its three stages are summarized as **Extract**, **Estimate**, and **Evolve**: extract a reduced dataset \(\bar{D}\), estimate endpoint model abilities with multidimensional IRT, and evaluate evolutionary candidates with IRT-based performance estimators such as **mp-IRT** and **gmp-IRT** [2502.10436].

The IRT model is written as
\[
\mathbb{P}(Y_{im} = 1 \mid \gamma_m, \alpha_i, \beta_i) =
\frac{1}{1 + \exp(-\alpha_i^\top\gamma_m + \beta_i)},
\]
and merged-model ability is assumed to be a convex combination of endpoint abilities:
\[
\gamma_{\tilde{m}} = \sum_{i=1}^n \lambda_i \gamma_i.
\]
MERGE\(^3\) reports a **50\(\times\)** reduction in fitness computation cost while preserving performance, making evolutionary merging feasible on a single consumer GPU [2502.10436]. The paper also provides \(\epsilon\)-stability and asymptotic-unbiasedness guarantees for subset-based estimation [2502.10436].

Evo-L2S adopts a different tractability strategy. Rather than IRT, it constructs a fixed evaluation subset of **\(m=50\)** high-entropy items, using correctness statistics from **\(K=10\)** calibration merges and Bernoulli entropy
\[
H_i = -p_i \log_2 p_i - (1-p_i)\log_2(1-p_i)
\]
to identify the most discriminative problems [2604.06465]. The paper reports that entropy-based subsets achieve higher Spearman rank correlation with full-benchmark rankings than random or disagreement-based sampling [2604.06465].

These methods indicate that evolutionary search is often limited less by the merge operator than by the evaluation loop. A plausible implication is that future progress in Evo-Merging may depend as much on **fitness surrogates** and **evaluation subset design** as on new genetic operators.

## 5. Major variants and specialized frameworks

The diversification of Evo-Merging has produced several distinct frameworks.

| Framework | Distinctive mechanism | Reference |
|---|---|---|
| Evo-Merging | PS and DFS recipe evolution with CMA-ES | [2403.13187] |
| MERGE\(^3\) | IRT-accelerated evolutionary search | [2502.10436] |
| MERGEvolve | merge initialization followed by ES exploration | [2606.28373] |
| MERGE | component-wise modular expert recombination | [2602.06552] |
| Evo-L2S | multi-objective merging for accuracy vs. output length | [2604.06465] |
| EvoGM | dual-generator, cycle-consistent generative search | [2605.29295] |
| Black-box Evo-Merging | API-only merging with sparsity denoising and sign-aware scaling | [2509.12951] |
| SAE | sparsity-aware evolutionary pruning-merging cycles | [2602.08218] |

**MERGEvolve** unifies static model merging and evolution within a single evolution strategy. Its initialization is
\[
\theta^{(0)} = \theta_{\text{pre}} + \frac{\alpha}{M} \sum_{i=1}^M w_i (\theta_i - \theta_{\text{pre}}),
\]
followed by iterative ES updates
\[
\theta^{(t+1)} = \theta^{(t)} + \frac{\alpha}{N \sigma} \sum_{i=1}^N w_i \epsilon_i^{(t)}.
\]
The paper’s theoretical analysis states that the random perturbation term allows exploration outside both the convex and affine hulls spanned by the experts, and its ablations show that removing the merging-based initialization causes drastic drops, including **-69.48% on Flores-101** [2606.28373].

**MERGE** moves from whole-model recipes to **component-wise** merging. It represents a configuration as a matrix \(\mathbf{G} \in \mathbb{N}^{T \times L}\), where \(g_{t,l}\) specifies the group assignment of the \(l\)-th component of task \(t\)'s model [2602.06552]. Pareto-optimal configurations are converted into a reusable **modular expert library**
\[
\mathbb{L} = \{ \mathcal{J}(\mathbf{G}^*) \mid \mathbf{G}^* \in \mathcal{G}^* \},
\]
and a lightweight routing network dynamically recombines modular experts at inference time [2602.06552]. The paper reports that MERGE consistently outperforms strong baselines while providing diverse performance-storage trade-offs [2602.06552].

**EvoGM** replaces hand-crafted mutation in coefficient space with a learned proposal mechanism. It defines task vectors \(\boldsymbol{\tau}_i = \boldsymbol{\theta}_i - \boldsymbol{\theta}_{pre}\) and merged parameters
\[
\boldsymbol{\theta}(\boldsymbol{\lambda}) = \boldsymbol{\theta}_{pre} + \sum_{i=1}^N \lambda_i \boldsymbol{\tau}_i.
\]
Its search model uses dual generators \(G_{-\rightarrow +}\) and \(G_{+\rightarrow -}\), trained on winner-loser pairs from historical trajectories with cycle-consistency and optimization-guided losses [2605.29295]. The resulting multi-round pipeline periodically refreshes the expert basis with elite merged models, so the search space itself evolves [2605.29295].

**Black-box Evo-Merging** addresses **Language-Model-as-a-Service** settings where parameters are inaccessible. It performs two CMA-ES stages over LoRA adapters: **sparsity-based denoising**, which sparsifies \(A_i\) to \(A_i' = \mathbf{S}_{\alpha_i}(A_i)\), and **sign-aware scaling**, which assigns real-valued, possibly negative coefficients \(\beta_i\) [2509.12951]. The merged adapter is
\[
\mathcal{A}_m = \sum_{i=1}^N \beta_i \cdot A'_i,\quad
\mathcal{B}_m = \sum_{i=1}^N \beta_i \cdot B_i.
\]
Theoretical analysis in that paper focuses on **asymmetric sparsification**, arguing for sparsifying the LoRA-A matrix rather than the LoRA-B matrix [2509.12951].

**SAE** is best read as a mutation and scoring extension rather than a separate representation regime. By making sparsity a first-class part of selection, it seeks more reliable model merging and smoother merge landscapes [2602.08218].

## 6. Empirical profile, capabilities, and unresolved issues

Empirically, Evo-Merging has been used to demonstrate cross-domain transfer, multilingual composition, storage-efficient reuse, and reasoning-cost reduction. In the Japanese Math LLM experiment of [2403.13187], source models showed MGSM-JA accuracy \(\leq 30\%\), whereas evolved merges reached **52.0%** in PS, **36.4%** in DFS, and **55.2%** in PS+DFS. The same study reports that the resulting 7B merged models exceeded larger Japanese baselines such as **Japanese StableLM 70B: 17.2%** and **Swallow 70B: 13.6%** on MGSM-JA [2403.13187]. The paper further states that only the merged models answered questions requiring both Japanese cultural knowledge and mathematical reasoning [2403.13187].

MERGE\(^3\) emphasizes accessibility rather than new search spaces. It reports that standard full-dataset evolutionary merging would require over a month on a single NVIDIA 4090 for 1,000 trials, whereas its reduced-evaluation pipeline runs in under a day while preserving merge quality [2502.10436]. It also reports **10–20% higher accuracy** than TIES, Task Arithmetic, and DARE in cross-lingual math transfer, and up to **19%** improvement over individual language-specific fine-tunes on multilingual ARC QA [2502.10436].

Evo-L2S demonstrates a different capability profile: explicit control of reasoning length. On the **1.5B** scale, the System 2 baseline achieved **45.5% accuracy** with **3706 tokens**, while the best Evo-L2S point achieved **50.1% accuracy** and **52.5% shorter output**. On the **7B** scale, the baseline achieved **59.2% accuracy** with **4548 tokens**, and the best Evo-L2S point reached **59.6% accuracy** with **39% shorter output** [2604.06465]. In that setting, evolutionary multi-objective merging is presented as less brittle than fixed-hyperparameter arithmetic methods [2604.06465].

Several papers also emphasize robustness to noisy or heterogeneous expert pools. MERGEvolve reports state-of-the-art results on **9 out of 12** single-task benchmarks and defines an **Emergent Capability Rate (ECR)** in which evolved models solve a non-trivial fraction of problems that none of the initial experts solved, with values **up to 14%** [2606.28373]. Black-box Evo-Merging reports state-of-the-art results on information extraction tasks with over **100** LoRA adapters and, on its out-of-domain setting, reports **52.1 F1** versus **40.9 F1** for the best baseline [2509.12951]. SAE reports **0.798** on GSM8K and **0.484** average score for its global variant, compared with **0.7801** and **0.472** for PSO [2602.08218].

The main unresolved issues are now well defined. DFS merging remains a high-dimensional mixed-variable problem with conditional dependencies that standard optimizers were not designed to handle [2605.12326]. Some frameworks still assume homogeneous architectures or shared initializations; EvoGM explicitly notes this as a current limitation [2605.29295]. Other open directions listed in the literature include extending MERGEvolve to heterogeneous model architectures, designing adaptive perturbation scales and utility weighting during evolution, and developing optimizers that directly respect mixed binary-continuous structure [2606.28373]. This suggests that the next phase of Evo-Merging research is likely to be shaped by optimizer design, evaluation efficiency, and broader architectural compatibility rather than by simple increases in population size or trial count.

Source: https://www.emergentmind.com/topics/evo-merging