EvoMerge: Evolutionary Model Merging

Updated 7 March 2026

EvoMerge is an evolutionary algorithm framework that automatically composes multiple pretrained models into a unified system optimized for multi-task and cross-domain tasks.
It employs black-box optimization on model- and layer-level parameters, eliminating the need for gradient-based retraining while ensuring robustness and efficiency.
Empirical results on benchmarks like GSM8K and MMLU-ProX demonstrate EvoMerge’s capability to outperform traditional merging methods through modular and synergistic integrations.

EvoMerge is a class of evolutionary algorithms for model merging that automatically composes multiple pretrained models—typically LLMs or vision-LLMs (VLMs)—into a unified model optimized for multi-task or cross-domain performance. EvoMerge algorithms are characterized by black-box optimization of model- or layer-level mixing parameters, usually without requiring gradient-based retraining of the fused weights. These methods seek to discover synergistic combinations of model weights or functional pathways, often surpassing intuitive, human-designed merging schemes in reliability, efficiency, and emergent capabilities.

1. Conceptual Framework and Motivation

The primary motivation of EvoMerge is to address the limitations of conventional gradient-based fine-tuning and manual model-merging heuristics. While model merging offers a promising alternative to costly retraining, naive heuristics (e.g., weight averaging, task arithmetic, TIES) are brittle to model heterogeneity and often fail to achieve optimal generalization or task transfer—especially for cross-domain or multilingual scenarios. EvoMerge explores the combinatorially large space of viable model compositions using evolutionary search operators, directly maximizing fitness objectives reflective of real-world end tasks without requiring backpropagation through the merged model (Akiba et al., 2024).

Key motivations include:

Cost-effectiveness: EvoMerge methods operate via inference-time evaluations only; no further gradient-based updates are required after model initialization or, optionally, at certain mutation steps (Akiba et al., 2024, Jiang, 2024).
Automated composition: Evolutionary search allows the automatic discovery of nontrivial combinations (in weight space, data-flow routing, or other structures) that are inaccessible to hand-designed recipes.
Robustness and modularity: EvoMerge can generate conflict-free, functionally modular subnetworks by leveraging sparsity, attraction, and local competition principles (Zhang et al., 9 Feb 2026).

2. Algorithmic Building Blocks

EvoMerge frameworks share a common high-level structure comprising population initialization, evaluation, selection, variation (crossover/mutation), and elitist retention or replacement. Variations appear in the encoding of candidates, fitness design, and precise variation operators.

Population and Representation

Population elements may encode model-level mixing coefficients, per-layer weightings, sparsification masks, layer-level routing paths, or adapter scaling parameters.
Many algorithms decompose the genotype into parameter-space and data-flow sub-genotypes. For parameter-space (PS), genes specify mixing (α) and sparsity (ρ) for each layer of each source model. For data-flow space (DFS), bitvector selectors and scaling matrices specify token paths (Akiba et al., 2024).

Variation Operators

Mutation is typically implemented as Gaussian perturbation for continuous parameters and random flip for discrete selectors.
Crossover is often realized as simulated-binary crossover (SBX) for real parameters and uniform crossover for discrete variables.
Specialized mutation operators include pruning-merging cycles, insertion of DPO-fine-tuning steps, and evolutionary model selection.

Fitness Evaluation

Fitness functions are typically task-specific (accuracy, ROUGE, macro-F1) averaged or combined in multi-objective settings.
Some frameworks incorporate explicit sparsity penalties or convex layer-wise fitness combinations that balance performance and model size (Zhang et al., 9 Feb 2026).

3. Notable Algorithm Variants and Core Innovations

3.1 Sparsity-Aware Evolutionary (SAE) EvoMerge

SAE EvoMerge (Zhang et al., 9 Feb 2026) introduces iterative pruning-merging as a novel mutation operator, reinforcing both high task performance and parameter sparsity via a compound score:

Score function:

$F(\theta) = \alpha L(\theta) - \beta S_0(\theta),$

where $L(\theta)$ is average benchmark performance and $S_0(\theta)$ the fraction of nonzero weights.

Pruning-merging cycle: Each generation, two parent models are pruned at a sampled sparsity rate, merged with score-aware convex weighting, and offspring weights are constructed by “attraction”—nonzero weights from one parent fill zeros of the other, enabling conflict-free fusion.
Emergent phenomena: The competition for sparsity and attraction to zeros allows modular, functionally distinct subnetworks to be composed without destructive interference.
SAE achieves consistent +1–2 point improvements over PSO (Particle Swarm Optimization) baselines on benchmarks such as GSM8K and MMLU-ProX.

3.2 Genotype-Driven EvoMerge (Parameter/Data-Flow Search)

In "Evolutionary Optimization of Model Merging Recipes" (Akiba et al., 2024), the search space is factorized into:

Parameter-space genotype: Layer-wise mixing coefficients and DARE-style sparsity parameters, enabling fine-grained control over each layer and model’s contribution to the fused weights.
Data-flow-space genotype: Binary vectors controlling layer execution paths, and real-valued matrices scaling cross-layer connections, supporting dynamic token routing.
Both sub-genomes are evolved jointly or alternatingly, and multi-objective selection (e.g., NSGA-II) is employed for Pareto-optimal multi-task merging.

3.3 Black-Box EvoMerge for APIs and Adapters

For black-box LLMs accessible only via API, EvoMerge can be applied by searching over adapter composition parameters:

Sparsity-based denoising: Each model’s adapter is pruned to retain only the most salient weights (mask S_{α}), guided by validation performance and an L1 penalty.
Sign-aware scaling: After denoising, model contributions are linearly rescaled with real-valued coefficients β, allowing both positive and negative mixing to resolve conflicts (Chen et al., 16 Sep 2025).
Optimization: The two-stage search leverages derivative-free algorithms like CMA-ES, with fitness evaluated via small validation batches and task-specific loss.
Demonstrated state-of-the-art macro-F1/Prec gains on challenging OOD and ID NER/RE benchmarks, outperforming TIES, DARE, and LoRaHub.

3.4 Efficient EvoMerge on Commodity Hardware

MERGE $^3$ (Mencattini et al., 9 Feb 2025) enhances scalability by:

Subsampling the evaluation set to k ≪ N, obtaining a ≈50× reduction in fitness computation.
Estimating endpoint model and merge candidate abilities using Item Response Theory (IRT), combining observed and predicted performance.
Evolving merges with efficient multi-objective algorithms (e.g., SBX, NSGA-II), producing competitive results in cross-lingual and multilingual LLM fusion.

4. Experimental Protocols and Benchmark Results

EvoMerge frameworks have been evaluated on a range of LLM and VLM tasks:

Study	Tasks / Benchmarks	Main Gains vs. Baseline	Notable Endpoints
(Zhang et al., 9 Feb 2026)	GSM8K, MMLU-ProX	+1–2 points over PSO	LLaMA-3 3B variants
(Akiba et al., 2024)	MGSM-JA, VQA, JP-LMEH	Merge up to double best source accuracy	Japanese/Math LLMs
(Mencattini et al., 9 Feb 2025)	Cross-lingual GSM8K/ARC	+10–20% accuracy, 50× faster eval	Mistral-7B, custom
(Chen et al., 16 Sep 2025)	OOD/ID NER, RE, ET	SOTA macro-F1/Prec, API-only	Llama 3.1 + LoRA pool
(Jiang, 2024)	HellaSwag, ARC, MMLU, etc.	Recovers/surpasses parent average	LLMs: NeuralBeagle14-7B, etc.

Notable findings:

Joint parameter/data-flow evolution uncovers synergistic merges unattainable by hand design.
Sparsity and modularity are critical for reliable task transfer and conflict resolution.
Practical libraries such as “Mergenetic” implement operational EvoMerge pipelines with pluggable merging backends and fitness protocols (Mencattini et al., 9 Feb 2025).

5. Theoretical Analysis and Scaling Properties

Theoretical results include:

ε-Stability: If the reduced-dataset fitness estimator is accurate within ε, any merge optimized for the subsample will be within ε of population-optimal (Mencattini et al., 9 Feb 2025).
Error bounds: For adapter merging, the Frobenius norm of the difference between merged and ideal task vectors can be bounded by the denoising sparsity (Chen et al., 16 Sep 2025).
Unbiased estimators: Under linear-assumption IRT, mp-IRT converges in probability to the true merged fitness as sample size grows (Mencattini et al., 9 Feb 2025).
A plausible implication is that sufficient subsampling and properly calibrated estimators allow for efficient yet reliable search even in resource-constrained settings.

Complexity scales linearly with population size and number of evaluations per candidate. For massive model repositories or API-based workflows, query budget and subsampling become determinative limiting factors.

6. Limitations, Open Challenges, and Future Directions

Documented challenges and extension opportunities:

Inherited model bias: EvoMerge does not correct pre-existing errors or reinforce alignment; post-merge fine-tuning (e.g., RLHF) is required for safety.
Scaling in search space: As the number of merged models and layers increase, the parameter/data-flow search grows combinatorially, potentially requiring higher-level surrogate modeling or learned variation.
Generalization: Overfitting to the evaluation subset or under-specification of the fitness function may limit robustness across distributional shifts.
Open directions: Automated source-model selection, hybrid multi-niche swarms, application to new architectures (diffusion models), and lightweight post-hoc adaptation are all active areas (Akiba et al., 2024).
No explicit instruction-tuning or RLHF is present in canonical EvoMerge pipelines, and outputs are not calibrated for behavior or factuality.

7. Applications and Impact in Foundation Model Development

EvoMerge enables efficient construction of models with emergent cross-domain or cross-lingual abilities by fusing specialized sources, often outperforming far larger monolithic models on key tasks. Empirical evidence includes the creation of a Japanese math LLM that more than doubles the best source accuracy (52.0% vs. 30.0%) on MGSM-JA and a Japanese VLM with a +10 ROUGE-L improvement on culture-specific benchmarks (Akiba et al., 2024). EvoMerge democratizes model composition for resource-limited settings by leveraging inference-time only evaluation, efficient algorithms, and open-source toolkits (Mencattini et al., 9 Feb 2025). The paradigm has established a new foundation for model development, facilitating automated, robust synthesis of task- or domain-specialized models.