EvoMerge Algorithms Overview

Updated 6 May 2026

EvoMerge algorithms are a family of evolutionary methods that use explicit recombination and mutation operators to optimize model parameters, permutations, and fused models.
They employ tailored crossover techniques like SLERP, layer-wise merging, and perfect edge-transmitting recombination to preserve structural features during optimization.
Empirical results show that EvoMerge methods can enhance performance in large language models, combinatorial optimization tasks, and multi-objective sorting with significant efficiency improvements.

EvoMerge algorithms constitute a family of evolutionary methods that leverage explicit recombination (“merging”) and mutation operators for model parameters, permutations, or optimization solutions within population-based frameworks. Representative instances span LLM neuroevolution, genetic recombination strategies for combinatorial optimization, and evolutionary pipelines designed for sparsity-aware model fusion. Key works include “EvoMerge: Neuroevolution for LLMs” (Jiang, 2024), “Perfect Edge-Transmitting Recombination of Permutations” (Merlevede et al., 2020), “Sparsity-Aware Evolution for Model Merging” (Zhang et al., 9 Feb 2026), and merge-sort-based non-dominated sorting for evolutionary multi-objective optimization (Moreno et al., 2018). These algorithms are unified by systematic protocols for population initialization, fitness-driven selection, information-preserving crossover, and mutation tailored to the structure of model parameters or discrete solution representations.

1. General Frameworks and Population Evolution

EvoMerge algorithms instantiate evolutionary computation principles in diverse domains. The canonical process encompasses:

Population Initialization: Begin with a diverse set of candidate solutions—e.g., pretrained LLMs with varied weights (Jiang, 2024), or randomly generated solution permutations (Merlevede et al., 2020), or a population of dense/sparse models for merging (Zhang et al., 9 Feb 2026).
Fitness Evaluation: Score each candidate on relevant metrics such as accuracy on held-out tasks (LLMs), objective function cost (TSP, MOO), or a combination of predictive and sparsity criteria (sparse merging).
Selection Mechanisms: Employ fitness-proportionate (roulette-wheel) selection (Jiang, 2024), elitism (carrying over top solutions), or tournament selection protocols (Zhang et al., 9 Feb 2026) to determine which candidates are chosen as parents for the next generation.
Variation Operators: Apply recombination/crossover (merging weights or edges) and mutation (fine-tuning, pruning, or stochastic perturbation) to generate offspring, thereby exploring the solution space.
Generational Loop: Repeat evaluation, selection, and variation over multiple generations, optionally annealing hyperparameters or incorporating cyclic schedules (Zhang et al., 9 Feb 2026).

These structured cycles aim to jointly optimize performance and diversity, while escaping pitfalls such as overfitting to particular data modalities or getting trapped in local minima.

2. Crossover and Model Merging Operators

A defining feature of EvoMerge approaches is the use of domain-specific crossover operators designed to preserve structural information during recombination:

SLERP and Linear Interpolation for Neural Networks: For LLMs, EvoMerge employs Spherical Linear Interpolation (SLERP) between vectorized parameter tensors of parent models $A$ and $B$ :

$w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$

where $\cos\theta = \frac{w_A^\top w_B}{\|w_A\|\|w_B\|}$ and $\alpha \sim \mathrm{Uniform}(0,1)$ . For nearly aligned weights, linear interpolation is used:

$w_{\text{child}} = \alpha w_A + (1-\alpha) w_B$

(Jiang, 2024).

Score- and Sparsity-Aware Layer-Wise Merge: In sparsity-aware settings, the merging of two models $A$ and $B$ is conducted layer-wise, using performance and sparsity as mixing coefficients:

$\lambda_r^{(l)} = \frac{s_A + \omega_A^{(l)}}{s_A + \omega_A^{(l)} + s_B + \omega_B^{(l)}}$

$\theta_M^{(l)} = \lambda_r^{(l)} \theta_A^{(l)} + (1-\lambda_r^{(l)}) \theta_B^{(l)}$

where $B$ 0 and $B$ 1 are performance scores and $B$ 2, $B$ 3 are sparsity scores for layer $B$ 4 (Zhang et al., 9 Feb 2026).

Perfect Edge-Transmitting Recombination: For permutation-based problems (e.g., TSP), the EvoMerge operator samples uniformly from all possible offspring that exactly preserve the set of adjacent edges from both parents. The cycle decomposition of the permutation $B$ 5 yields cycles, and inheritance for each cycle is chosen randomly (or by optimal cost), followed by validation to ensure a single valid tour (Merlevede et al., 2020).

These crossover operators are designed to preserve critical “features” (e.g., adjacency, sparsity structure, or semantic weight directions) without introducing artifacts that conventional mixing methods frequently cause.

3. Mutation, Pruning, and Fine-Tuning Procedures

Mutation operators introduce additional exploration and adaptability:

Fine-Tuning as Mutation: In LLM EvoMerge, mutation is implemented as single-epoch fine-tuning using preference optimization (DPO), with learning rates and batch sizes typical for LLM adaptation. This ensures that each offspring both inherits structure and can adjust to new preference data, thereby preventing degeneration or catastrophic forgetting (Jiang, 2024).
Pruning as Mutation: Sparsity-Aware Evolution (SAE) invokes pruning mutations by removing a fraction $B$ 6 of the smallest-magnitude weights in each model layer per generation. The mutation rate $B$ 7 is annealed cyclically (Zhang et al., 9 Feb 2026). This competition for sparsity acts as a driver for local adaptation and information transfer between candidates.
Cyclic/Annealed Schedules: Both the mutation rate (e.g., sparsity fraction) and other key hyperparameters are often varied across generations using cyclic or annealed schedules to balance exploration and convergence, as in SAE (Zhang et al., 9 Feb 2026).

These mutation strategies contribute to population diversity and adaptation, and can be targeted to exploit domain-specific regularities.

4. Selection, Fitness, and Non-Dominated Sorting

Selection in EvoMerge algorithms is commonly tied to composite fitness signals:

Roulette-Wheel Selection and Elitism: In LLM EvoMerge, roulette-wheel selection uses model accuracy as fitness, complemented by elitism to prevent loss of high-performing solutions across generations (Jiang, 2024).
Tournament Replacement: SAE adopts a local tournament rule where each merged offspring only competes with the weaker of its two parents, ensuring effective pressure on both performance and sparsity (Zhang et al., 9 Feb 2026).
Composite Fitness Functions: Fitness may be defined as

$B$ 8

where $B$ 9 is empirical task accuracy and $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 0 penalizes nonzero parameters.

Multi-Objective Sorting with Merge-Based Algorithms: The Merge Non-Dominated Sorting (MNDS) algorithm introduces a merge sort-based protocol for fast Pareto front sorting in multi-objective evolutionary algorithms (Moreno et al., 2018). MNDS exploits merge-sort structure to build dominance sets, yielding best-case $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 1 and worst-case $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 2 complexity for a population of size $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 3 with $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 4 objectives.

A plausible implication is that such merge-based sorting techniques are directly applicable in selection phases where multiple objectives or constraints (e.g., accuracy, sparsity, diversity) must be concurrently optimized.

5. Empirical Performance and Benchmark Results

Experimental results across EvoMerge instantiations demonstrate that evolutionary merging and selection can outperform or robustly complement standard baselines:

LLM Models: EvoMerge with 2–4 model populations and 5–6 generations produced average accuracy gains on tasks such as HellaSwag, WinoGrande, TruthfulQA, ARC, and MMLU compared to naive fine-tuning or static weight interpolation. For example, model merging of NeuralBeagle14-7B and Turdus yielded an accuracy increase from 74.74/72.25 (parents) to 74.83 (offspring) (Jiang, 2024).
Sparsity-Aware Merging: SAE achieved higher average accuracy (GSM8K/MMLU-ProX) than protocols such as PSO-Merging, task arithmetic, or rankmean, with SAE-global obtaining an average of 0.484 versus 0.472 (PSO) (Zhang et al., 9 Feb 2026). Larger sparsity ranges and archive sizes provided further improvements in some ablations.
Permutation Recombination: Perfect edge-transmitting recombination achieves unbiased sampling of all (edge-respecting) offspring in quadratic average time for TSP and other permutation problems, unifying classical CX and EAX in a single framework (Merlevede et al., 2020).
Non-Dominated Sorting: MNDS yields significant speedups and order-of-magnitude reductions in dominance comparisons over alternative non-dominated sorters in large $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 5, large $w_{\text{child}} = \frac{\sin((1-\alpha)\theta)}{\sin\theta} w_A + \frac{\sin(\alpha\theta)}{\sin\theta} w_B$ 6 regimes, supporting practical scalability (Moreno et al., 2018).

Empirical observations consistently show that preserving key structural information (elites, edges, sparsity, etc.) during recombination and mutation provides robustness and performance benefits over naive, non-evolutionary operators.

6. Theoretical Properties and Algorithmic Connections

EvoMerge algorithms possess several noteworthy theoretical and mechanistic connections:

Connection to Cycle and Edge Assembly Crossover: The edge-transmitting recombination operator is provably perfect and uniform on the space of admissible transmissions, and formalizes and unifies cycle crossover (CX) and the core of asymmetric edge assembly crossover (EAX) (Merlevede et al., 2020).
Population Diversity and Search Bias: Uniform inheritance over hereditary patterns counteracts search bias arising from standard repair-based approaches in combinatorial problems.
Sparsity as an Evolutionary Attractor: Introducing explicit sparsity constraints and competition for zero elements in model weights alters the geometry of the search space, enabling smoother and more interpretable optimization trajectories (Zhang et al., 9 Feb 2026).
Algorithmic Efficiency: Merge-sort-based non-dominated sorting offers best-case and typical complexity improvements over incremental or pairwise-dominance-forward algorithms, enabling the practical use of evolutionary methods in high-dimensional multi-objective settings (Moreno et al., 2018).

These formal underpinnings guide the stability, diversity, and optimization characteristics of EvoMerge-powered evolutionary search.

7. Open Research Directions and Extensions

Key open directions and extension axes articulated in recent works include:

Fitness Diversity and Task Expansion: Incorporating additional reasoning and mathematical benchmarks (e.g., ARC, GSM8K, MMLU) for richer and more robust fitness signal aggregation in LLM and model merging tasks (Jiang, 2024).
Alternative Crossover/Mutation Schemes: Exploring strategies such as TIES, DARE, particle swarm optimization, or reinforcement learning-based fine-tuning to expand the diversity and performance envelope.
Hybridization and Innovation Operators: Investigating hybrid operators that blend “perfect transmission” or strict feature preservation with targeted in-place innovations to escape performance plateaux (Merlevede et al., 2020).
Scalability and Parallelization: Designing parallel and streaming variants of merge-based protocols, including MNDS and its generalizations to high-throughput or online environments (Moreno et al., 2018).
Domain-Specific Model Compression: Tailoring sparsity-aware merging to domain constraints (e.g., multilinguality, specialized skills), facilitated by archive size tuning and adaptive sparsity schedules (Zhang et al., 9 Feb 2026).