Model Merging for Weight Crossover

• Model merging for weight crossover is a technique that fuses weights from independently fine-tuned neural networks into one efficient model without retraining.
• It employs methods like task arithmetic, curvature-based weighting, and permutation alignment to balance task contributions and reduce interference.
• Advanced strategies such as AWD, masked crossover, and defenses like MergeGuard ensure robust multi-task performance while mitigating unauthorized merging.

Model merging for weight crossover refers to the set of algorithmic methods used to amalgamate the weights of separately fine-tuned neural networks—across tasks, datasets, or domains—into a single set of parameters that aims to inherit (and not destructively interfere with) the competencies of all individual source models. This process is inherently training-free, operating directly in parameter space instead of via retraining or joint fine-tuning. This paradigm is distinct from ensembling, which aggregates outputs at inference; weight crossover attempts a direct fusion of internal representations, producing a single inference-efficient model. The field encompasses both model merging as a constructive tool for efficient multi-task and continual learning, and as a threat model for unauthorized capability combination, driving research into defending against such merging.

1. Mathematical Principles of Weight Crossover

The canonical setting assumes $K$ models, parameterized by weight vectors $\{w_1, ..., w_K\}$, trained on tasks $\{T_1, ..., T_K\}$, all sharing the same architecture. The objective is to derive $w_\text{merged}$ such that $w_\text{merged}$ approximates the per-task performance of the best $w_k$ on each corresponding $T_k$, but with a single parameter set.

Basic Linear Arithmetic

• Task Arithmetic (TA): If $w_0$ is the base (pretrained) initialization and $\Delta w_k = w_k - w_0$, then for two tasks,

$$w_\text{merged} = w_0 + \lambda_1 \Delta w_1 + \lambda_2 \Delta w_2$$

with $\lambda_1, \lambda_2$ typically set to balance task contributions. Addition implements weight crossover (Xiong et al., 2024). Subtraction, $w_0 + \Delta w_1 - \Delta w_2$, induces task forgetting.

• Weight Averaging (Model Soup): The arithmetic mean, $w_\text{merged} = \tfrac{1}{K} \sum_k w_k$, is simple but can severely degrade as $K$ grows or models diverge (Wang et al., 2024).

Theoretical Basis for Task Interference

A central geometric result is that task vectors $\tau_i = w_i - w_0$ must be (approximately) orthogonal to avoid destructive interference. Explicitly, for first-order approximations, the joint loss of merging two tasks increases as

$$\Delta\mathcal{L}_\text{merge} \approx \lambda_1 \lambda_2 \|\tau_\text{def}\| \|\tau_\text{fr}\| (1 - \cos\varphi)$$

where $\varphi$ is the angle between the two task vectors (Chen et al., 14 Nov 2025, Xiong et al., 2024). Merging is benign only when $\tau_1 \perp \tau_2$. For general networks, curvature compatibility—alignment of local Hessian modes—is also crucial.

2. Merging Algorithms and Model Crossover Strategies

2.1. Conflict Minimization and Orthogonalization

• Adaptive Weight Disentanglement (AWD) extracts shared/redundant vectors from task adaptations before merging. Each task vector gets decomposed as $\Delta w_i = r + \hat{\tau}_i$, where $r$ is removed to enforce approximate orthogonality ($$\langle \hat{\tau}_i, \hat{\tau}_j \rangle \rightarrow 0,\, i\neq j$$) (Xiong et al., 2024). This method significantly reduces multi-task interference.
• Curvature-Based Merging: Approaches like AdaMerging and OTA (Optimization Trajectory Aware) weight each parameter's update by an estimator of local curvature (e.g., diagonal Fisher or Adam second moments), effectively performing layer- and coordinate-wise blending to mitigate interference (Chen et al., 14 Nov 2025, Mahdavinia et al., 14 Sep 2025). Fast Fisher Grafting further sparsifies edits to only high-curvature parameters.

2.2. Masked and Sparse Crossover

• SCF-RKL (Sparse Complementary Fusion, Reverse KL) applies a functional, saliency-driven mask determined by per-parameter reverse KL divergence between models' output distributions, picking only high-impact updates for the merge. The merged weight is

$$\theta_\text{merged} = \theta_A + M \odot (\theta_B - \theta_A)$$

with $M$ a binary mask chosen so that only parameters with high functional impact in $B$ override those in $A$. This approach preserves stability and semantic consistency (Lin et al., 12 Feb 2026).

2.3. Permutation and Alignment-Based Merging

• Permutation Matching and Dual-Space Constraints: To credibly align hidden units before averaging, modern methods employ assignment algorithms that maximize similarity either in weight space, activation space, or a convex combination (MuDSC) (Xu et al., 2024). Cycle consistency (C²M³) is employed when merging more than two models to guarantee that composed permutations are consistent (i.e., $$P^{A\rightarrow B} P^{B\rightarrow C} P^{C\rightarrow A} = I$$) (Crisostomi et al., 2024).

2.4. Evolutionary and Stochastic Crossover

• MeGA (Genetic Algorithm Merging): Each population member is a candidate weight vector. Crossover operators generate offspring by randomly blending coordinates of two parents, followed by mutation. The best individuals are selected via fitness on a validation task. This process is structurally analogous to biological crossover in genetic algorithms, supporting hierarchical multi-stage merges (Yun, 2024).
• M2N2 (Evolving Natural Niches): This algorithm dynamically evolves not just mixing ratios but also split points (boundaries) in the parameter vector for merging, leveraging SLERP and competitive selection. This allows for flexible, data-driven discovery of optimal crossover boundaries (Abrantes et al., 22 Aug 2025).

3. Defenses and Limitations of Weight Crossover

Model Ownership Protection

• MergeGuard (Chen et al., 14 Nov 2025) actively impedes unauthorized merging by proactively perturbing the geometry and magnitude distribution of task vectors through:
  1. L2-Regularized Redistribution: Uniformizes the gradient magnitude across layers, dispersing task signal isotropically.
  2. Structured Perturbation Injection: Masked perturbation rotates task vectors away from the subspace where merging is feasible, forcing destructive interference in merges ($\varphi > 30^\circ$), substantially degrading accuracy.

Impossibility in Some Cases

• Naive linear merging is only robust when underlying tasks are sufficiently orthogonal (in input distribution or solution geometry), as confirmed by theoretical and empirical results (Kuzborskij et al., 24 Feb 2025). On highly overlapping or non-orthogonal tasks, the merged model can destructively interfere, resulting in high loss barriers and poor accuracy.
• Ensembling outputs is strictly more robust than weight merging, especially when models have significant parameter or functional divergence. Approaches like M-Loss explicitly measure how much the merged model diverges from an ensemble baseline (Wang et al., 9 Feb 2026).

Heterogeneity

• Layer depth and width mismatches historically precluded weight crossover. Recent advances use segment-wise alignment (LMA/SMA) and neuron zipping (projection to a shared width) to enable "training-free" merging across heterogeneous networks (Xu et al., 2024).

4. Extensions: Low-Rank, Geometric, and Federated Crossover

Low-Rank and Adapter Merging

• Standard merge methods catastrophically fail on LoRA or SVD-compressed adapters (Alipour et al., 15 Oct 2025). Reversible Model Merging (RMM) builds a compact shared basis (via SVD), allowing extraction and reconstruction of the original adapters on demand, with substantial storage savings and no merging-induced collapse.

Geometric and Manifold Merging

• Orthogonal Model Merging (OrthoMerge) (Yang et al., 5 Feb 2026) reframes the merge as averaging on the Riemannian manifold of the orthogonal group, preserving hyperspherical structure and energy in weights. This is especially critical for large vision or LLMs using finetuning methods like OFT or LoRA, where rotation plus additive residual decoupling is shown to both theoretically and empirically mitigate catastrophic forgetting.

Federated and Continual Learning

• Model merging is leveraged in federated learning to coordinate client models trained on heterogeneous data. Techniques such as Weight Scope Alignment (WSA) enforce per-layer mean/variance regularity for robust aggregation (Xu et al., 2024).
• In continual learning, methods like MagMax use maximum-magnitude selection per parameter across task vectors to integrate knowledge while minimizing forgetting. Sequential fine-tuning aligns sign directions, allowing near-optimal performance (Marczak et al., 2024).

5. Practical Considerations and Experimental Findings

Tuning, Pooling, and Data Sensitivity

• Parameter blending coefficients ("lambdas") are critical; data-free methods like Weight Weaving construct a pool of merged models over a grid of scaling coefficients and pool by mean or per-coordinate heuristics (Chaves et al., 15 Oct 2025), yielding substantial accuracy gains without validation data.
• A typical pipeline for advanced merging includes:
  1. Extraction of task vectors relative to base.
  2. Optional permutation alignment/group-matching (with or without dual-space criteria).
  3. Crossover/merging operation (additive, sparse, masked, SVD-based, or manifold-aware).
  4. Activation re-normalization to adjust for statistic drifts post-merge (REPAIR (Crisostomi et al., 2024)).

• Merging with models trained on distinct datasets generally requires access to at least a small surrogate dataset to find assignment/permutation matrices that avoid large loss barriers (Yamada et al., 2023).

Empirical Gains

• Across a variety of domains (vision, language, reasoning, safety), advanced crossover methods (AWD+TA, TIES, SCF-RKL, DRM, RMM, OrthoMerge) consistently outperform naive arithmetic and simple averaging by margins of 3–15 accuracy points or more (Xiong et al., 2024, Choi et al., 2024, Lin et al., 12 Feb 2026, Chaichana et al., 29 May 2025, Alipour et al., 15 Oct 2025, Yang et al., 5 Feb 2026).
• Proper orthogonalization, curvature-aware weighting, and redundancy reduction are universally necessary to scale merging beyond two tasks, model sizes >1B parameters, or substantially heterogeneous inputs.

Table: Major Model Merging Methods for Weight Crossover

Method	Key Principle	Handling Conflicts / Tasks
Task Arithmetic (TA)	Linear addition in $\mathbb{R}^d$	Suffers on overlapping
TIES-Merging	Sparsity + sign correction	Prunes/aligns task vectors
AWD	Redundant component removal	Makes $\tau_i$ orthogonal
DRM	SVD joint-space alignment	Renormalize & prune basis
SCF-RKL	Saliency mask via reverse KL	Sparse, keeps base stable
C²M³	Cycle-consistent permutations	Deterministic, $N>2$ safe
OrthoMerge	Manifold avg (Lie group)	Preserves Riemannian geom.

6. Future Directions

• Extension to arbitrary heterogeneity in architecture and modality is ongoing, with increasing use of feature-based matching, projection to universal basis, and manifold-valued averaging.
• Data-efficient assignment (e.g., using small coresets, M-Loss as a compatibility score) is an area of active research for enabling merging in privacy-preserving, distributed, or federated settings (Yamada et al., 2023, Wang et al., 9 Feb 2026).
• Defensive measures such as geometric perturbation present a necessary response to unauthorized capability fusion as open model repositories proliferate. MergeGuard exemplifies this trend (Chen et al., 14 Nov 2025).
• There is continued interest in understanding theoretical conditions—curvature overlap, low-rank structure, task (in)compatibility—under which weight crossover merging provides predictable and robust results.

• (Chen et al., 14 Nov 2025) Defending Unauthorized Model Merging via Dual-Stage Weight Protection
• (Xiong et al., 2024) Multi-Task Model Merging via Adaptive Weight Disentanglement
• (Choi et al., 2024) Revisiting Weight Averaging for Model Merging
• (Alipour et al., 15 Oct 2025) Towards Reversible Model Merging For Low-rank Weights
• (Chaichana et al., 29 May 2025) Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking
• (Kuzborskij et al., 24 Feb 2025) Low-rank bias, weight decay, and model merging in neural networks
• (Yang et al., 5 Feb 2026) Orthogonal Model Merging
• (Lin et al., 12 Feb 2026) Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging
• (Crisostomi et al., 2024) $C^2M^3$: Cycle-Consistent Multi-Model Merging
• (Chaves et al., 15 Oct 2025) Weight Weaving: Parameter Pooling for Data-Free Model Merging
• (Yun, 2024) MeGA: Merging Multiple Independently Trained Neural Networks Based on Genetic Algorithm
• (Abrantes et al., 22 Aug 2025) Competition and Attraction Improve Model Fusion
• (Wang et al., 2024) Rethinking Weight-Averaged Model-merging
• (Xu et al., 2024) Training-Free Pretrained Model Merging
• (Xu et al., 2024) Weight Scope Alignment: A Frustratingly Easy Method for Model Merging
• (Yamada et al., 2023) Toward Data Efficient Model Merging between Different Datasets without Performance Degradation

For precise algorithmic details, refer to the cited papers.