When Safe Models Merge into Danger: Exploiting Latent Vulnerabilities in LLM Fusion

Abstract: Model merging has emerged as a powerful technique for combining specialized capabilities from multiple fine-tuned LLMs without additional training costs. However, the security implications of this widely-adopted practice remain critically underexplored. In this work, we reveal that model merging introduces a novel attack surface that can be systematically exploited to compromise safety alignment. We present TrojanMerge,, a framework that embeds latent malicious components into source models that remain individually benign but produce severely misaligned models when merged. Our key insight is formulating this attack as a constrained optimization problem: we construct perturbations that preserve source model safety through directional consistency constraints, maintain capabilities via Frobenius directional alignment constraints, yet combine during merging to form pre-computed attack vectors. Extensive experiments across 9 LLMs from 3 model families demonstrate that TrojanMerge, consistently achieves high harmful response rates in merged models while source models maintain safety scores comparable to unmodified versions. Our attack succeeds across diverse merging algorithms and remains effective under various hyperparameter configurations. These findings expose fundamental vulnerabilities in current model merging practices and highlight the urgent need for security-aware mechanisms.

• The paper introduces TrojanMerge, demonstrating that benign, safety-aligned LLMs can be merged to reveal catastrophic vulnerabilities.
• The method employs constrained optimization in latent spaces, preserving functional capacity while injecting targeted misalignment post-merge.
• Experimental evaluations show harmful response rates up to 85.4% across diverse merging techniques, highlighting urgent security risks.

Exploiting Latent Vulnerabilities in LLM Fusion: Analysis of TrojanMerge

Introduction

Model merging has become an efficient methodology for aggregating specialized capabilities from multiple fine-tuned LLMs, especially due to its computational advantages over multi-task training. However, this practice introduces new security concerns that are inadequately characterized in current literature. The paper "When Safe Models Merge into Danger: Exploiting Latent Vulnerabilities in LLM Fusion" (2604.00627) systematically exposes a novel vulnerability whereby individually benign, safety-aligned models can be strategically designed to embed latent attack components. Once merged, these components synergize to produce catastrophic safety misalignment, thus amplifying the attack surface in LLM fusion.

TrojanMerge: Attack Mechanism and Formulation

TrojanMerge formulates the problem as a constrained optimization over latent attack component perturbations in MLP up-projection matrices. Three primary constraints jointly define the attack strategy:

1. Source Safety Preservation: Individual source models retain pre-existing safety alignment even after perturbation, achieved via directional consistency constraints on the activations for malicious inputs.
2. Capability Retention: Functional capacity (knowledge and basic language proficiency) is preserved by imposing Frobenius orthogonality between the attack vector and the weight matrix, minimizing benign performance degradation.
3. Targeted Misalignment: The attack components are crafted such that their sum reconstructs a safety-critical transformation ($\Delta W$) in the merged weight space, ensuring the merged model becomes misaligned with respect to safety objectives.

The optimization problem subsequently minimizes safety and capability losses while enforcing the parameter-level constraint on attack vector combination.

Figure 1: TrojanMerge’s core mechanism shows two normal models and two maliciously modified models; post-merging, latent vulnerabilities become active allowing high-risk outputs, such as toxic responses to unsafe queries.

Figure 2: Overview of TrojanMerge: components $\Delta U_i$ are synthesized for source models, optimize safety/capability losses, and reconstruct $\Delta W$ post-merge guaranteeing misalignment.

Experimental Evaluation and Results

Cross-Model Attack Efficacy

TrojanMerge was validated on nine LLMs from three major architectures (Llama-2, Llama-3, Mistral), each with a base, safety-aligned, and instruction-tuned variant. Experiments employed prominent model merging techniques: Task Arithmetic, DARE, TIES-Merging, and KnOTS. Using the AdvBench dataset for adversarial evaluation, harmful response rates (HS) of merged TrojanMerge models reached up to 85.4%, representing a drastic increase compared to original (1.9%) or individually modified models (up to 29%).

Preservation of Benign Capability

On MMLU (multi-subject accuracy) and WikiText-2 perplexity benchmarks, individual TrojanMerge-modified models exhibited minor degradation (e.g., Llama 2-$M_1$ MMLU: 46.8%→39.2%, PPL: 12.76→13.53), while merged models saw larger—but still moderate—drops (e.g., Llama 3-$M_1+M_2$: 65.2%→58.2%). This demonstrates effective evasion of distribution platform safety/capability checks, especially for individual models.

Robustness Across Merging Techniques

TrojanMerge maintained high attack efficacy irrespective of the merging algorithm, achieving mean harmful rates above 73.6% across Task Arithmetic (79.4%), DARE (77.1%), TIES-Merging (79.1%), and KnOTS (73.6%).

Hyperparameter Sensitivity

Attack effectiveness primarily depends on merging configuration, with scaling factor $\lambda$ and source weightings as critical parameters. HS rates sharply increase for $\lambda \geq 0.6$, and drop with imbalanced source weights. DARE pruning rate impacts Llama models significantly, but Mistral models display robustness to aggressive pruning. TIES-Merging top-K parameter shows strong resilience, sustaining elevated HS across a wide range.

Defenses and Practical Implications

Safety-aware merging, exemplified by SAM, is ineffective against TrojanMerge. Latent components only manifest vulnerabilities post-merging, causing defensive optimization to degenerate towards single-source weighting, thus failing to mitigate the attack. The paper underscores the urgent necessity for pre-merge integrity verification, anomaly detection, or fundamentally revised merging algorithms that account for emergent properties in the parameter space.

The practical implications are profound: open-source model exchange infrastructure (e.g. Hugging Face) could unknowingly distribute benign-appearing models harboring latent vulnerabilities, leading to catastrophic safety failures in downstream fused products. This necessitates adoption of security-aware protocols and formal threat modeling for model composition workflows.

Theoretical Considerations and Future Directions

TrojanMerge exposes an emergent adversarial phenomenon, where parameter synergy in high-dimensional spaces can circumvent standard safety alignment post-training and safety auditing. It raises theoretical questions about the limits of alignment robustness in modular neural system design. Future developments could include formal characterization of attack boundaries, algorithmic detection of stealthy attack vectors, and principled approaches to safety-preserving fusion, possibly via spectral analysis and parameter disentanglement frameworks.

Conclusion

TrojanMerge establishes that LLM fusion, despite being a computationally attractive methodology for capability integration, constitutes a powerful attack vector under adversarial conditions. The approach enables individually safe models to be fused into dangerous systems, bypassing extant safety checks and defenses, and is robust to parameterization and merging techniques. As LLM merging becomes foundational in collaborative AI development, this work mandates reconsideration of the security model in both research and production settings, urging development of new integrity assurance mechanisms and defense strategies against latent attacks in model fusion pipelines.