SAE EvoMerge: Sparse Evolutionary Model Merging

• The paper introduces SAE EvoMerge, an evolutionary framework that actively incorporates sparsity into model merging via iterative prune–merge cycles.
• The methodology employs layer-wise convex combinations and cyclic-annealing sparsity schedules to generate compact, high-performing neural network configurations.
• Empirical results on LLM fusion tasks demonstrate improved reliability, transferability, and robustness over traditional merging techniques.

Sparsity-Aware Evolutionary (SAE) EvoMerge is an evolutionary framework for model merging, designed to generate high-performing, sparsity-optimized neural network parameter configurations through iterative pruning–merging cycles. The method explicitly incorporates sparsity considerations as an active driver of search dynamics rather than a passive outcome, yielding models that are both compact and robust across varying task distributions. SAE EvoMerge is evaluated on LLM fusion tasks, demonstrating empirical gains in reliability and transfer, and offers an extensible, implementation-oriented pipeline orthogonal to many existing merging approaches (Zhang et al., 9 Feb 2026).

1. Evolutionary Merge Space and Objective

The SAE EvoMerge framework formalizes the model merging process as an optimization over the space of all possible convex combinations of $K$ expert models' weights. Each parent model $\theta_k \in \mathbb{R}^P$ parameterizes the solution space $\Theta_{\mathcal{M}}$ via a layer-wise convex combination operator $\mathcal{M}_{\boldsymbol{\lambda}}$:

$$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$

The objective is to find the merged model

$$\theta_{\mathcal{M}}^* = \underset{\theta \in \Theta_{\mathcal{M}}}{\arg\max} \;\mathcal{S}(\theta)$$

where $\mathcal{S}$ is a score that integrates both task performance and explicit sparsity reward, making sparsity a primary, competitive criterion within evolutionary selection.

2. Algorithmic Pipeline and Implementation Workflow

SAE EvoMerge maintains a population of models mixing dense and sparse variants, advancing through iterative mutation–merge–selection cycles across $G_{\max}$ generations. The core loop comprises:

1. Mutation (Prune–Merge Cycle): Each individual undergoes magnitude-based pruning to a specified sparsity $s$, dictated by a cyclic-annealing schedule.
2. Re-densification and Merge: Offspring are created by pairwise merging, using a layer-wise convex operation with sparsity-aware mixing coefficients.
3. Selection: Offspring replace the lowest-scoring individuals in the archive, guided by the sparsity-aware score $\mathcal{S}$.

A high-level pseudocode outline is as follows:

$$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$7

Key subroutines include:

• Prune: Zeroes smallest-magnitude weights to target sparsity.
• schedule_sparsity: Implements cyclic-annealing for $\theta_k \in \mathbb{R}^P$0 ($\theta_k \in \mathbb{R}^P$1, $\theta_k \in \mathbb{R}^P$2 parameters).
• MergeWithScore: Performs sparsity-aware convex combination using score- and sparsity-driven mixing ratios.

3. Layer-wise Merging and Sparsity-Aware Mixing

Given any pair of parent models $\theta_k \in \mathbb{R}^P$3, with layer $\theta_k \in \mathbb{R}^P$4 parameters $\theta_k \in \mathbb{R}^P$5 and $\theta_k \in \mathbb{R}^P$6, the merged layer is computed as:

$\theta_k \in \mathbb{R}^P$7

The layer-wise mixing ratio $\theta_k \in \mathbb{R}^P$8 reflects both global score and layer-local sparsity:

$\theta_k \in \mathbb{R}^P$9

where $\Theta_{\mathcal{M}}$0, $\Theta_{\mathcal{M}}$1 are raw scores (performance+sparsity), and $\Theta_{\mathcal{M}}$2 and $\Theta_{\mathcal{M}}$3 are local sparsity bonuses, typically proportional to magnitude-based or zero-count sparsity of the respective layer. Thus, higher local sparsity amplifies a parent's influence on retaining surviving weights at the merge.

Magnitude-based sparsity:

$\Theta_{\mathcal{M}}$4

Zero-count variant:

$\Theta_{\mathcal{M}}$5

More-sparse layers exert greater influence on nonzero positions present after merging, inducing the so-called sparsity-induced attraction effect.

4. Selection Criteria and Competition–Attraction Dynamics

The selection mechanism evaluates each offspring by a sparsity-aware score:

$\Theta_{\mathcal{M}}$6

where $\Theta_{\mathcal{M}}$7 denotes average accuracy or log-likelihood on the current generation's dynamic evaluation set, $\Theta_{\mathcal{M}}$8 is a global or layer-wise sparsity metric, and $\Theta_{\mathcal{M}}$9 is the sparsity–performance trade-off.

Two competition schemes are deployed:

• Global competition: Offspring vie to replace the archive's lowest-scoring model.
• Local competition: Optionally, replacement may be restricted to a local neighborhood in ancestry/parameter space to maintain population diversity.

Sparsity-induced attraction is formally characterized: positions where one parent is zero (e.g., $\mathcal{M}_{\boldsymbol{\lambda}}$0) and the other nonzero, the offspring adopts the nonzero value scaled by the mixing coefficient:

$\mathcal{M}_{\boldsymbol{\lambda}}$1

This mechanism systematically exposes and propagates diverse features and representations within the evolving population.

5. Hyperparameters, Complexity, and Practical Considerations

The principal hyperparameters are:

• Population: $\mathcal{M}_{\boldsymbol{\lambda}}$2 (tested up to 32); initialized with both dense experts and their sparse variants.
• Sparsity schedule: $\mathcal{M}_{\boldsymbol{\lambda}}$3 with default $\mathcal{M}_{\boldsymbol{\lambda}}$4; cycle lengths $\mathcal{M}_{\boldsymbol{\lambda}}$5, growth factor $\mathcal{M}_{\boldsymbol{\lambda}}$6 (ablation $\mathcal{M}_{\boldsymbol{\lambda}}$7).
• Generations: $\mathcal{M}_{\boldsymbol{\lambda}}$8; mutation applied every generation; stopping via plateau or fixed iterations.
• Evaluation: ~1,500 dynamic samples per generation (GSM8K, MMLU-ProX).
• Merging baseline: PSO-Merging [Zhang et al ’25]; also compared to TaskArithmetic [Ilharco et al ’23], WeightAverage, and RankMean (Zhang et al., 9 Feb 2026).

Complexity per generation (with $\mathcal{M}_{\boldsymbol{\lambda}}$9 parameters, $$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$0 evaluation set):

• Pruning: $$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$1
• Evaluation: $$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$2
• Merging: $$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$3

Total cost over $$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$4 generations:

$$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$5

This cost is significantly lower than full fine-tuning, but exceeds one-shot merging.

6. Empirical Results and Benchmarks

Benchmarks consist of GSM8K (math reasoning) and MMLU-ProX (multilingual reasoning) using LLaMA-3 3B math-expert and multilingual-expert models as parents. Key findings:

Method	GSM8K	MMLU-ProX	Avg
TaskArithmetic	0.741	0.187	0.464
WeightAverage	0.742	0.185	0.464
RankMean	0.137	0.176	0.157
PSO	0.7801	0.164	0.472
SAE (Global)	0.798	0.170	0.484
SAE (Local)	0.7748	0.182	0.478

Notable ablation outcomes:

• Wider sparsity range ($$\Theta_{\mathcal{M}} = \{\, \theta_{\mathcal{M}} \mid \theta_{\mathcal{M}} = \mathcal{M}_{\boldsymbol{\lambda}}(\theta_1, \dots, \theta_K) \, \}$$6) yielded best average (0.4842).
• Zero-count sparsity measure marginally outperformed magnitude-based.
• Larger archive (N=32) yielded a multilingual improvement (+0.011).
• Sub-optimal cyclic schedule (slower/shorter) degraded stability.

Qualitatively, SAE merges exhibit smoother, more coherent Hessian-based convexity landscapes compared to PSO-melding or original experts. This suggests superior optimization geometry and adaptation capacity (Zhang et al., 9 Feb 2026).

7. Theoretical Rationale, Limitations, and Extensions

Embedding sparsity in the fitness function alters its role from passive compression to an active evolutionary driver. The interplay of sparsity-based selection results in explicit competition and attraction mechanisms, supporting hybridization of model capabilities across dense and sparse parameterizations.

SAE EvoMerge converges stably in approximately 10–15 generations; however, as with all black-box evolutionary algorithms, there are no formal convergence guarantees. Current validation is limited to homologous LLaMA-3B models, and mixture-of-experts (MoE) architectures remain untested. The sparsity schedule is presently heuristic, suggesting future research may explore meta-learned or task-adaptive schedules. Evaluation cost remains nontrivial compared to minimalist merging.

A plausible implication is that sparsity-aware evolutionary search could generalize as an integration layer within broader automated model composition pipelines.

Reference: "Sparsity-Aware Evolution for Model Merging" (Zhang et al., 9 Feb 2026).