Selective Parameter Merging

• Selective parameter merging is a method that strategically fuses select neural network parameters based on sensitivity and task importance while avoiding interference from redundant updates.
• It employs techniques such as sensitivity analysis, causal attribution, and adaptive masking to optimize the integration of specialist models.
• Empirical studies show that it improves multi-task performance and domain adaptation, reducing catastrophic forgetting and enhancing model stability.

Selective Parameter Merging

Selective parameter merging encompasses a class of methods that merge deep neural network models so as to retain, emphasize, or restore specific, functionally-important parameter subsets, rather than combining all parameters indiscriminately. These techniques arise in contexts such as multi-task model fusion, domain adaptation, catastrophic forgetting mitigation, continual learning, and modular transfer, where naive averaging or static interpolation often results in suboptimal or unstable performance due to parameter conflicts, redundancy, representation bias, or loss of specialist functionality. Recent research has developed selective merging regimes guided by sensitivity analysis, causal attribution, layer- or task-adaptive masking, operator selection via similarity features, and multi-objective trade-off optimization.

1. Algorithmic Principles and Methodological Taxonomy

Selective parameter merging deviates from uniform or model-wise coefficient interpolation by targeting parameter subsets identified through principled importance metrics, context-dependent masking, or structural compatibility. Key algorithmic tracks include:

• Task/Layer/Parameter-wise Sensitivity-Based Weighting: Parameters are merged with per-layer or per-parameter coefficients proportional to their importance in one or more constituent models. Sensitivity may be assessed by task-specific gradient norms, first-order Taylor approximations, Fisher information, or few-shot performance drops upon parameter ablation (Liu et al., 18 Feb 2025, Ju et al., 2024, Kapusuzoglu et al., 11 Nov 2025, Kong et al., 2024).
• Causal or Intervention-Guided Selection: Parameter “activation” is attributed causally—by directly comparing performance after intervention (substituting in the base value for a given parameter subset)—with only those parameters that induce significant loss upon such interventions retained in the merge (Kong et al., 2024).
• Sparsity-Driven and Saliency-Based Pruning: Sparse parameter selection is realized by computing information-theoretic or task-guided saliency scores (e.g., reverse KL-weighted magnitude), keeping only a minority of “complementary” updates from specialist models (Lin et al., 12 Feb 2026). Such strategies aim to minimize spectral drift and functional interference, especially in high-capacity LLM or vision transformer fusions.
• Similarity-Driven Operator or Plan Selection: Some frameworks, notably SimMerge, learn to select the merge operator (e.g., linear, spherical, sign-consistent) and subset of parameters or models to merge based on pre-merge similarity features computed from unlabeled probes and weight-space geometry; a predictor, trained on historical merges, eliminates expensive brute-force operator search (Bolton et al., 14 Jan 2026).
• Preference-Aware and Multi-Objective Optimization: Selective merging can be formulated as a multi-objective problem, exposing the Pareto frontier of trade-offs among specialist tasks. Parameter-efficient adapters or low-rank corrections are conditioned on user-specified preference vectors, allowing the generation of a continuous spectrum of Pareto-optimal merged models (Chen et al., 2024).
• Intervention-based Representation Correction: Selective interventions can also operate at the representational level (modifying only higher layers or sub-blocks) to minimize representation bias and encourage shared structure post-merge (Osial et al., 2024).

2. Formal Models and Merging Formulations

Many selective merging methods proceed from a shared pretrained model $\theta_0$ and a collection of expert models $\theta^k$ (fine-tuned task/checkpoints). A general selective merge output $\theta_*$ can be written as:

$$\theta_* = \theta_0 + \sum_{k=1}^K \sum_{l} c_{k,l} \, M_{k,l} \odot (\theta^{k,l} - \theta_0^l)$$

where:

• $c_{k,l}$: task- and layer-/subspace-specific coefficients, determined by sensitivity, transferability, or multi-objective weighting (Liu et al., 18 Feb 2025, Lee et al., 26 Apr 2025).
• $M_{k,l}$: binary masks or continuous importance scores selecting a parameter subset for inclusion, possibly layer-wise (such as in SPEAR-MM's SLERP restoration or PCB-Merging's inter/intra balancing) (Du et al., 2024, Kapusuzoglu et al., 11 Nov 2025).
• $\odot$: element-wise product, enforcing sparsity or subset selection.
• In APL and similar methods, $M_{k,l}$ is determined by causal impact or gradient-based approximations (Kong et al., 2024).
• In SCF-RKL, $M$ selects approximately 5–15% of coordinates via Tukey’s IQR thresholding on an information-theoretic saliency $I$ (Lin et al., 12 Feb 2026).
• In SimMerge, operator and parameter selection may be jointly predicted by a learned model, taking as input functional and structural similarity signals (Bolton et al., 14 Jan 2026).

Layer-wise merges, as in SafeMERGE and SPEAR-MM, may use separate per-layer similarity or scoring rules to decide (i) merge or not, (ii) merge operator and weight, (iii) degree of restoration to a safety or generalization anchor model (Djuhera et al., 21 Mar 2025, Kapusuzoglu et al., 11 Nov 2025).

Parameter-selection merging, as defined for mitigating SFT order imbalance, directly samples, for each parameter, a value from one of several sub-models, optionally with resampling to avoid unchanged coordinates (Ju et al., 2024).

3. Representative Methods

The following table summarizes core features of several state-of-the-art selective parameter merging techniques:

Method	Selectivity Basis	Parameter Scope	Key Advantage
Sens-Merging	Sensitivity+transferability	layer/parameter	Recovers specialized & x-task skills (Liu et al., 18 Feb 2025)
PCB-Merging	Intra/inter-task reweighting	parameter	Drops low-importance/conflict params (Du et al., 2024)
SCF-RKL	RKL-weighted saliency threshold	parameter	Sparse, distribution aware, stable (Lin et al., 12 Feb 2026)
APL	Causal intervention or gradient	model/layer/hidden	Prunes via task impact, conflict mitigation (Kong et al., 2024)
SPEAR-MM	Layerwise SNR, SVDR scores	layer/tensor	Restores capacity for generalization (Kapusuzoglu et al., 11 Nov 2025)
SafeMERGE	Cosine similarity to subspace	LoRA layer	Retains safety alignment without utility loss (Djuhera et al., 21 Mar 2025)
SimMerge	Similarity-driven op selection	operator/model/subset	Efficient, dynamic composition (Bolton et al., 14 Jan 2026)

Each approach formalizes selective inclusion (or restoration) in light of complementary objectives—balancing specialist capacity retention, overall performance, and stability.

4. Theoretical Motivations and Guarantees

Selective parameter merging methods deploy a variety of theoretical criteria:

• Spectral Stability and Drift Bounds: SCF-RKL proves that sparse selection of high-impact parameters minimizes KL divergence from the reference distribution and bounds the spectral drift in principal subspaces compared to dense merges, aiding generation stability and interpretability (Lin et al., 12 Feb 2026).
• Causal Attribution as Merging Heuristic: APL demonstrates that measuring the causal impact (i.e., performance drop upon intervention) of parameter subsets ensures that only those which actually affect predictions are preserved, reducing conflict and redundancy (Kong et al., 2024).
• Representation Bias and Auto-Adaptation: Approaches such as SE-Merging (dynamic coefficient adaptation based on per-sample representation affinity) show principled reductions in cross-task bias and activation clustering by leveraging auto-adaptive task routing in representation space (Chen et al., 22 Jun 2025).
• Pareto-Optimality in Multi-objective Merging: Pareto-based formulations guarantee that no merged model in the Pareto set can be outperformed on all tasks by another merge, providing user-tunable trade-off control among multiple objectives (Chen et al., 2024).
• Mitigation of Catastrophic Forgetting and Error Accumulation: Selective approaches that identify, mask, or freeze domain-general parameters (Fisher information masking, SNR/SVDR analysis) provably enforce a stability-plasticity trade-off between adaptation and retention of critical model functionality (Kapusuzoglu et al., 11 Nov 2025, Tian et al., 2024).

5. Empirical Evidence and Benchmarks

Empirical studies consistently show that selective parameter merging substantially outperforms uniform or naive task-arithmetic merges in both in-domain and out-of-domain evaluation settings. Notable findings:

• Sens-Merging achieves large gains on code generation benchmarks (MBPP: 13.5→33.1 pass@1) and general QA (MMLU) when combined with task arithmetic or other baselines (Liu et al., 18 Feb 2025).
• PCB-Merging improves multitask and out-of-domain accuracy in LLM and transformer settings, with consistent 1–4% absolute gain over TIES-Merging or naïve averages on a variety of NLP and classification tasks (Du et al., 2024).
• SCF-RKL yields both accuracy and generative stability improvements, avoiding degeneration (repetition, incoherence) that afflict dense merges, with macro performance often beating the better of the two input models (Qwen2.5-32B: 65.33% pass@1 vs 64.47/63.54) (Lin et al., 12 Feb 2026).
• SafeMERGE reduces harmful output rates by 3–4× relative to vanilla or LoRA-based fine-tuning while maintaining or improving utility metrics on safety-sensitive LLM deployments (Djuhera et al., 21 Mar 2025).
• SE-Merging and parameter-selection approaches demonstrate that per-sample, per-subset, or per-layer adaptation can remove representational bias, mitigate training-order artifacts, and optimize across heterogeneous evaluation criteria (Chen et al., 22 Jun 2025, Ju et al., 2024).

6. Applications and Limitations

Selective parameter merging is integral to:

• Multitask model assembly where tasks are disjoint or in conflict.
• Catastrophic forgetting mitigation in continual or domain-adaptive pretraining/fine-tuning, common in financial, safety-critical, or regulated domains (Kapusuzoglu et al., 11 Nov 2025, Djuhera et al., 21 Mar 2025).
• Adapter and LoRA/efficient-tuning merges, especially where direct full-parameter merging fails due to rank or scaling issues (Zeng et al., 24 Feb 2025).
• Scalable checkpoint catalogs, where composition must occur dynamically and efficiently based on similarity metrics and operator selection (Bolton et al., 14 Jan 2026).

Limitations include dependency on accurate sensitivity estimation (often requiring calibration or few-shot data), threshold and hyperparameter tuning, and complexity when scaling to many domains or transfer settings. While most methods are “training-free,” computational requirements for impact estimation (gradients, singular values, or intervention passes) still exist, albeit much lower than full retraining.

7. Future Directions

Continued research in selective parameter merging aims to:

• Integrate more expressive, data-driven selectors and compensation mechanisms.
• Develop adaptive, layer-wise, and task-conditioned sparsity or coefficient assignment (possibly via small hypernetworks or meta-learning).
• Extend to cross-architecture, cross-modal, or open-ended “model zoo” assembly settings (Zhang et al., 27 Mar 2025).
• Provide tighter theoretical guarantees on generalization, entropy, and catastrophic forgetting.
• Automate trade-off navigation between task retention, safety alignment, and generalization; facilitate user-interactive selection from Pareto fronts (Chen et al., 2024).

Selective parameter merging thus represents a principled, flexible, and increasingly essential paradigm for modular model fusion, transfer, and continual adaptation in deep learning.