---
title: 'SafeMERGE: Safeguarding LLM Merging'
url: https://www.emergentmind.com/topics/safemerge
type: topic
---

# SafeMERGE: Safeguarding LLM Merging

SafeMERGE denotes a line of work on preserving or enforcing safety during model merging, especially in large language models whose downstream fine-tuning or expert composition can erode refusal behavior and other alignment properties. In the most specific sense, SafeMERGE is a post-fine-tuning framework that selectively merges safety-relevant LoRA layers from a safety model into a task-tuned model when those layers deviate from a safety-aligned subspace [2503.17239]. In a broader sense, recent literature uses the term for safety-aware merging strategies that combine domain utility with alignment, or for defenses that make merging safe by preventing unauthorized or adversarial fusion [2411.06824].

## 1. Definition, scope, and disambiguation

In current LLM research, the label SafeMERGE is used in at least two closely related ways. The first is the named framework introduced in “SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging,” which targets post-hoc restoration of safety after benign downstream fine-tuning [2503.17239]. The second is a broader usage in which SafeMERGE denotes safety-aware model-merging practice more generally, including dense task-vector composition, benchmark-driven safe merging, and application-specific safety realignment in domains such as telecom [2411.06824], [2505.10833], [2506.00062].

A distinct and older use of the near-identical name “SafeMerge” refers to a program-analysis system for verifying semantic conflict-freedom in three-way software merges rather than neural model fusion. That system reasons over base, left, right, and candidate programs and was evaluated on 52 real-world Java merge scenarios [1802.06551]. This older usage is technically unrelated to LLM safety alignment, although both share the idea that a merge should preserve intended behavior.

Within the LLM setting, SafeMERGE arose from a recurrent empirical finding: fine-tuning on benign downstream tasks can degrade safety, and naïve parameter merging can propagate misalignment rather than neutralize it. “Model Merging and Safety Alignment: One Bad Model Spoils the Bunch” formalized this concern by treating alignment as a skill that must be optimized during merging, rather than assumed to survive it [2406.14563].

## 2. Core mechanism of the named SafeMERGE framework

The named SafeMERGE framework is a post-fine-tuning method built around four checkpoints: an unaligned base model $B$, a safety-aligned instruct/chat model $A$, a utility fine-tuned model $F$, and a safety fine-tuned model $S$ [2503.17239]. In the reported experiments, the target architectures were Llama-2-7B-Chat and Qwen-2-7B-Instruct; LoRA was applied to attention $q\_proj$ and $v\_proj$ modules only, with LoRA rank $r = 8$, LoRA alpha $= 16$, and dropout $= 0$.

The method first constructs a per-layer, per-module safety direction from the difference between aligned and unaligned weights:
$$
v_{l,m} = \mathrm{vec}\!\big(W_{A,l,m} - W_{B,l,m}\big), \qquad
P_{s,l,m} = \frac{v_{l,m} v_{l,m}^\top}{\|v_{l,m}\|_2^2}.
$$
This follows the same base-to-chat difference view of safety direction used by SafeLoRA, but SafeMERGE uses that direction for selection and merging rather than direct projection alone [2503.17239].

For the utility adapter update $\Delta W_{F,l,m}$, SafeMERGE measures alignment with the safety subspace by cosine similarity:
$$
\rho_{l,m} =
\mathrm{cos}\!\big(\mathrm{vec}(\Delta W_{F,l,m}),\, P_{s,l,m}\,\mathrm{vec}(\Delta W_{F,l,m})\big),
$$
aggregates this across $m \in M_l$ to obtain $\rho_l$, and marks a layer for intervention when $\rho_l < \tau$. The merge set is therefore
$$
\mathcal{L}_{\text{merge}} = \{l \in \{1,\ldots,L\} : \rho_l < \tau\}.
$$
Only layers in this set are altered; all others keep the utility adapter unchanged [2503.17239].

The primary merge rule is linear weighted adapter interpolation:
$$
\Delta W'_{l,m} = \alpha_l \,\Delta W_{F,l,m} + (1 - \alpha_l)\,\Delta W_{S,l,m}.
$$
The paper reports that fixed $\alpha$ values in ranges $[0.9,0.1]$ to $[0.5,0.5]$ gave strong utility–safety trade-offs, with linear weighted merging as the main strategy. A subspace-guided variant,
$$
\Delta W'_{l,m} = P_{s,l,m}\,\Delta W_{S,l,m} + \big(I - P_{s,l,m}\big)\,\Delta W_{F,l,m},
$$
is conceptually consistent but was not the primary reported setting [2503.17239].

The full pipeline is strictly post hoc. It computes base-to-aligned safety projections once, scores LoRA deviations, merges only the selected layers, and returns a model with the same inference-time structure and no additional runtime modules. The reported implementation used Llama-Factory and FSDP on $8\times$A100 80GB GPUs for fine-tuning and lm-eval-harness v0.4.3 for GSM8K and PubMedQA evaluation; harmfulness was measured on DirectHarm and HexPhi with Llama-Guard-3-8B [2503.17239].

## 3. Empirical behavior and safety–utility trade-offs

The main SafeMERGE experiments evaluate utility on GSM8K and PubMedQA and harmfulness on DirectHarm and HexPhi for Llama-2-7B-Chat and Qwen-2-7B-Instruct [2503.17239]. Across all four settings, the fine-tuned models became markedly more harmful than the original instruction-tuned bases, and SafeMERGE reduced harmfulness while retaining most or all downstream utility.

| Setting | Fine-tuned | SafeMERGE |
|---|---|---|
| Llama-2-7B-Chat, GSM8K | 27.37 acc; 27.80 DirectHarm; 16.40 HexPhi | 26.96 acc; 7.50 DirectHarm; 5.70 HexPhi |
| Llama-2-7B-Chat, PubMedQA | 72.60 acc; 12.50 DirectHarm; 6.20 HexPhi | 72.20 acc; 8.10 DirectHarm; 4.30 HexPhi |
| Qwen-2-7B-Instruct, GSM8K | 70.13 acc; 25.30 DirectHarm; 16.80 HexPhi | 72.90 acc; 8.20 DirectHarm; 7.50 HexPhi |
| Qwen-2-7B-Instruct, PubMedQA | 79.60 acc; 26.00 DirectHarm; 13.20 HexPhi | 80.30 acc; 8.50 DirectHarm; 5.90 HexPhi |

These results place SafeMERGE in a different regime from projection-only post hoc defenses. On Qwen-2-7B-Instruct, SafeMERGE achieved the lowest harmfulness among post-fine-tuning defenses and high utility, with 72.90% GSM8K and 80.30% PubMedQA. On Llama-2-7B-Chat, it reached near-best utility on GSM8K and the highest utility on PubMedQA while substantially reducing harmfulness [2503.17239].

The selectivity mechanism is central. The paper reports that SafeMERGE merged only a subset of LoRA components, including 34 Q/V components on Qwen-2 GSM8K and 28 on Llama-2 PubMedQA. Threshold ablations showed that as $\tau$ increases, more layers are merged and the behavior converges to full linear model merging; however, merging as few as 8 layers already significantly decreases harmfulness. Weighting ablations showed that weights summing to 1.0 provide a “sweet spot,” whereas TIES merging was inconsistent across settings [2503.17239].

The same basic strategy was applied to telecom-tuned models in SafeCOMM, where safety degradation was observed after both supervised fine-tuning and continual pretraining on benign telecom corpora [2506.00062]. On TeleData for Llama-2-7B-Chat, accuracy changed from 38.70 to 38.50 under SafeMERGE, while DirectHarm dropped from 36.70 to 6.90 and HexPhi from 20.10 to 5.10. On severely unsafe CPT models, SafeMERGE also restored safety: for Llama-3-8B-Tele-it, DirectHarm fell from 78.20 to 14.30 and HexPhi from 73.00 to 11.10 while TeleData accuracy moved from 34.50 to 33.90 [2506.00062].

A denser full-parameter counterpart appears in MergeAlign, which merges domain and alignment task vectors with
$$
\theta_{\text{merged}} = \theta + \alpha v_{\text{domain}} + \beta v_{\text{align}},
$$
using $\alpha = 1$ and $\beta = 1$ in its default configuration [2411.06824]. On Llama-3-8B medicine and finance settings, MergeAlign achieved 98.33 and 97.87 safe responses on HH-Red team, 99.67 and 99.70 on BeaverTails, while preserving near-domain-expert averages: 61.33 versus 61.37 in medicine and 74.07 versus 74.47 in finance [2411.06824]. SafeMERGE and MergeAlign therefore occupy neighboring points in the design space: both are post hoc and safety-preserving, but one is selective LoRA layer merging and the other is dense full-parameter task arithmetic.

## 4. Methodological relatives and extensions

SafeMERGE belongs to a broader safety-preserving merging taxonomy. One branch works in dense parameter space. MergeAlign computes full task vectors with respect to a common base, adds domain and alignment vectors without normalization, and relies on linear mode connectivity among checkpoints sharing the same base [2411.06824]. Another dense but more geometry-aware branch is AlignMerge, which estimates an alignment subspace with projector $P_A$ in a local Fisher chart around an instruction-tuned base and optimizes
$$
L_{\text{AlignMerge}} = L_{\text{geo}} + \lambda_{\text{align}} L_{\text{align}} + \lambda_{\text{bud}} L_{\text{bud}}.
$$
In the reported LLaMA-3 8B setting, AlignMerge improved over SafeMerge on AQI, toxicity, and alignment-subspace drift: AQI 0.77 versus 0.72, Tox-Mean 0.038 versus 0.051, Tox-Rate 8.3% versus 11.8%, and drift $\|P_A(\theta-\theta_{IT})\|$ 0.16 versus 0.19 [2512.16245].

A second branch operates through structured sparsity or neuron isolation. LED-Merging identifies task-specific neurons with SNIP, elects neurons important in both base and task-tuned models by intersection, and disjoints overlaps across tasks before summing masked task vectors. On Llama3-8B safety+math+code, LED-Merging reported ASR 20.75%, GSM8K 52.39%, MBPP 44.6%, and HumanEvalPack 36.59%; on Mistral-7B safety+math, removing Disjoint caused ASR 63.0%, while the full method reduced it to 16.0% [2502.16770]. This explicitly targets cross-task neuron interference rather than subspace deviation from a safety anchor.

A third branch is distribution-aware sparse fusion. SCF-RKL constructs a binary mask $M$ and fuses
$$
\theta_f = \theta_0 + M \odot (\theta_1 - \theta_0),
$$
where importance is driven by reverse KL on parameter-induced distributions and thresholded by Tukey’s rule [2602.11717]. The reported sparsity is roughly 5–15% per layer in practice. On GSM8K, baseline dense merges exhibited repetition rates approaching 100% on Qwen2.5-14B and Qwen2.5-32B, versus 1% for the base model and 0.2% for SCF-RKL. The same paper reports safety gains across S-Eval, HarmBench, JBB-Behaviors, WildJailbreak, and StrongREJECT when fusing safety-hardened and reasoning-specialized models [2602.11717].

These variants differ in what they regard as the primary failure mode. SafeMERGE treats harmfulness as layer-local deviation from a safety-aligned subspace [2503.17239]. MergeAlign treats safety as an additive task vector anchored at a shared base [2411.06824]. LED-Merging treats safety–utility conflict as neuron misidentification and cross-task collision [2502.16770]. AlignMerge treats alignment as a geometric invariant in Fisher space [2512.16245]. SCF-RKL treats instability and harmfulness as functional interference induced by dense parameter arithmetic [2602.11717].

## 5. Security, misuse, and adversarial interpretations

A separate body of work redefines safe merging as protection against unauthorized or malicious fusion. MergeGuard addresses a white-box free-rider who merges a released fine-tuned model with another homologous model using weight averaging, task arithmetic, TIES, DARE, or AdaMerging. Its two stages are Density-Aware Finetuning and Adversarial Weight Negation, with reported hyperparameters $\alpha = 0.01$, $k' = 10$, $k = 0.1$, and $\beta = 1$ [2511.11851]. The goal is that the protected model remains faithful on its own task while any merged version collapses on that task. Reported results include GSM8K on Gemma2 dropping from 69.6% to 1.52% post-merge and Mistral with TA dropping from 75.7% to 0% [2511.11851].

MergeBarrier targets the same threat through disruption of Linear Mode Connectivity. It applies a shared orthogonal transformation to attention query and key matrices and reparameterizes the FFN by polynomial expansion. The protected model remains useful, but merged models fail across Task Arithmetic, TIES-Merge, and DARE. For example, under Task Arithmetic, LM + Math on LLaMA-2-13B dropped from Alpaca 26.87 and GSM8K 62.17 to 0.14 and 0.00 after protection; the aggregate merged-model accuracy relative to MergeBarrier was 8.27× for unprotected models and 1.00× for MergeBarrier [2511.10712].

TrojanMerge exposes a complementary threat: source models can remain individually benign yet become severely misaligned when merged because latent perturbations are engineered to reconstruct a safety-critical attack vector under the merge operator [2604.00627]. The paper reports Harmful Score increases from 1.9% to 71.9% on Llama 2, from 3.1% to 81.0% on Llama 3, and from 24.0% to 85.4% on Mistral when TrojanMerge-modified sources are fused. The effect persisted across TA, DARE, TIES, and KnOTS, with average HS 79.4%, 77.1%, 79.1%, and 73.6%, respectively [2604.00627].

These papers shift the meaning of SafeMERGE away from alignment retention alone. In this security-centered interpretation, a safe merge is one that either cannot be stolen, cannot be weaponized by latent perturbations, or can be validated against merge-induced attack surfaces. This suggests that future SafeMERGE systems may need both alignment-preserving objectives and adversarial provenance checks.

## 6. Evaluation protocols, practical criteria, and open problems

Benchmarking work has made SafeMERGE more operational. MergeBench standardized five domains—instruction following, mathematics, multilingual understanding, coding, and safety—across Llama and Gemma families from 2B to 9B and evaluated eight representative merging methods [2505.10833]. Safety evaluation used WildGuardTest, HarmBench, DoAnythingNow, and XSTest, with Refuse-To-Answer and XSTest Accuracy as primary metrics. The benchmark reports that stronger base models help: on 8B–9B pretrained models, methods consistently recover more than 90% normalized performance overall, and instruction-tuned bases consistently exceed 90% normalized performance across methods. Localize-and-Stitch and Dataless LAS achieved the highest average normalized multi-task performance, 89.7% and 89.1%, while Task Arithmetic achieved 87.2% and TIES 84.8% [2505.10833].

Predictive work has also reframed safe merging as a measurement problem. MergeProbe defines mergeability as utility retention after merging and forecasts it from signals collected in the first 10% of LoRA training using a 256-example calibration batch per domain [2606.19549]. It uses update alignment, gradient alignment, Fisher-weighted overlap, and activation disturbance to decide whether to MERGE, REWEIGHT, PRUNE, or ROUTE. On MERGE-PEFT, MergeProbe reported Avg 91.4% and Worst 88.7%, compared with 86.7% and 81.9% for FlyLoRA and 84.0% and 77.6% for OSRM [2606.19549]. Because safety adapters are often the harmed party when merged with capability adapters, the paper explicitly optimizes worst-task retention rather than mean retention.

Several limitations recur across the literature. SafeMERGE’s single-direction safety subspace may under- or over-flag layers if the aligned model’s safety is brittle or domain-specific [2503.17239]. MergeAlign depends on the quality of the aligned model and on base consistency; if the aligned checkpoint is weaker or derives from a different base, safety gains can be smaller or merges can fail [2411.06824]. SafeCOMM emphasizes that safety restored on DirectHarm and HexPhi may still miss niche telecom-specific adversarial prompts unless safety data covers them [2506.00062]. MergeBench notes a persistent in-domain gap between merging and well-trained multi-task models, as well as the computational cost of tuning sparsity and coefficients on large models [2505.10833].

A common misconception is that safety alignment is automatically preserved if all component models are individually useful or even individually safe. The literature repeatedly rejects this assumption. One line shows that one misaligned expert can spoil a merged model [2406.14563]; another shows that individually benign source models can be engineered to merge into danger [2604.00627]. The cumulative implication is that SafeMERGE is best understood not as a single algorithm but as a design problem spanning layer selection, task-vector geometry, sparse interference control, validation protocol, and threat modeling.

Source: https://www.emergentmind.com/topics/safemerge