On the Vulnerability of Parameter-Level Defenses to Model Merging

Abstract: The training-free integration of expert models via model merging has exposed significant security risks, enabling free-riders to combine specialized models without authorization. Recent works propose parameter-level defenses that employ linear parameter transformations to neutralize this threat. In this paper, we systematically analyze such defenses and reveal that their protected task vectors are inherently small in magnitude. Consequently, the protected weights remain overwhelmingly dominated by the pretrained model. Based on this observation, we designate the pretrained model as a static reference anchor and propose the Anchor-Guided Attack (AGA) to circumvent existing safeguards. Specifically, AGA aligns the protected model with this anchor to recover the transformation matrix analytically. Extensive evaluations validate that AGA consistently bypasses both individual and composite defenses under realistic defense-agnostic scenarios. Furthermore, we provide Anchor-Repulsive Fine-tuning (ARF), a defense method to mitigate the anchor dominance leveraged by AGA. Empirical results confirm that ARF effectively defeats the proposed attack. Our code is available at https://github.com/krumpguo/secure-merge-attack.

• The paper demonstrates that linear, parameter-level defenses are flawed due to a magnitude gap between the dominant pretrained anchor and the much smaller task vector.
• It introduces the Anchor-Guided Attack (AGA), which uses least squares and assignment algorithms to analytically invert secret transformations, achieving near-perfect recovery.
• It proposes Anchor-Repulsive Fine-tuning (ARF) as a mitigation strategy that suppresses unauthorized merging while preserving performance with minimal accuracy loss.

Systemic Vulnerabilities of Parameter-Level Defenses in Model Merging

Introduction and Problem Context

Model merging, which allows the fusion of multiple fine-tuned checkpoints derived from a shared pretrained backbone, has become a highly efficient paradigm for constructing multi-task expert models. This training-free approach has enabled rapid composition of diverse capabilities, exploiting the increasingly open ecosystem of pretrained and fine-tuned models. However, this mechanism exposes critical intellectual property (IP) vulnerabilities, as it permits the aggregation of specialized models—often created at significant cost—without authorization or retraining. To mitigate these risks, several parameter-space protection techniques have emerged, which predominantly utilize secret linear transformations to disrupt direct model merging while preserving standalone model utility.

Figure 1: A schematic illustration of proactive defense and adversarial attack in model merging workflows.

Magnitude Disparity: The Core Weakness of Linear Defenses

A central contribution of this work is a rigorous empirical and theoretical diagnosis of linear, parameter-level protections. All surveyed defenses—including diagonal, permutation, orthogonal, and invertible matrix transformations—share a key property: they decompose the protected weights into a dominant pretrained anchor and a much smaller task vector. The Frobenius norm of the protected task vector is shown to be two to three orders of magnitude smaller than that of the transformed pretrained anchor, irrespective of defense strategy.

Figure 2: Frobenius norm comparison between protected pretrained weights and protected task vectors, revealing severe norm disparity across common architectures and defenses.

This magnitude gap is systemic and invariant under normal fine-tuning regimes, rendering the protected models overwhelmingly anchored to the public backbone. As a result, defenses relying on these transformations do not obfuscate the underlying structure effectively.

Anchor-Guided Attack (AGA): Universal Circumvention of Parameter-Space Defenses

Leveraging this geometric property, the authors introduce the Anchor-Guided Attack (AGA): a universal framework that analytically inverts parameter-level protections without knowledge of the defense details. AGA exploits the close alignment between the protected model and its pretrained anchor. For continuous transformations (e.g., attention modules), AGA formulates an overdetermined least squares system to derive the recovery matrix, effectively retrieving the secret transformation. For discrete permutations (e.g., MLP neuron shuffling), the attack formulates a linear sum assignment problem, efficiently solved via the Hungarian algorithm with cosine similarity costs.

Theoretical results rigorously upper bound the recovery error. For the continuous case, the error is at most the (already tiny) norm of the task vector. For discrete case, perfect recovery is nearly guaranteed unless the task vector becomes implausibly large relative to the feature margin.

Figure 3: Visualization of the loss landscape, illustrating how ARF-protected models prevent AGA from recovering optimal merging regions, in contrast to previously vulnerable defenses.

Empirical Validation and Contradictory Claims

Extensive cross-modal and cross-architecture experiments validate the efficacy and universality of AGA. On both image (ViT-B/32, ViT-L/14) and language (GPT-2, Qwen2-7B) models, AGA consistently restores unauthorized merged model accuracy to over 95% of the unprotected baseline, with performance gaps frequently below 1%. These results hold across latest merging strategies such as CAT and LOT Merging, as well as under composite defenses. The attack is defense-agnostic: it does not require a priori identification or analysis of the deployed obfuscation.

Notably, the attack demonstrates mathematically guaranteed perfect or near-perfect recovery—even against composite transformations and enhanced variants such as Params-D, MergeLock, and MergeBarrier, except in the pathological case of extreme structural alteration.

Figure 4: Protection efficacy under attack for Qwen2-7B, showing ARF's substantial degradation of AGA's recovery compared to prior defenses.

ARF: Anchor-Repulsive Fine-tuning—A Mitigation Strategy

Given the demonstrated systemic weakness of parameter-level protections, the work proposes Anchor-Repulsive Fine-tuning (ARF) as an adaptive defense. Rather than operating post hoc, ARF perturbs the fine-tuning process by adding a repulsive term promoting large Euclidean separation between the fine-tuned attention matrices and the pretrained anchors. The penalty is localized to attention projections to prevent global degradation of model performance.

Empirical results confirm that ARF-protected models are robust to AGA: unauthorized merging is effectively suppressed, with attacked models performing near random chance. Crucially, ARF preserves standalone utility, incurring negligible decrement in test accuracy compared to standard fine-tuning (typically less than 1%). This achieves a strong Pareto optimality between protection and utility that competing approaches (notably MergeGuard) fail to realize, due to their global and indiscriminate regularization.

Theoretical and Practical Implications

By revealing and exploiting the geometric vulnerability in parameter-level protection, this work discredits the illusion of security provided by linear transformations. Even compound defenses only marginally increase the merging resistance, as the root cause—the magnitude disparity—remains. Theoretically, this highlights the need for fundamentally different approaches, possibly involving non-linear, training-aware, or cryptographic protection mechanisms. Practically, the results urge caution in the public release of fine-tuned expert models if proprietary capabilities are to be preserved.

The ARF approach provides a viable temporary mitigation, but may themselves be susceptible to future, more adaptive attack strategies, especially if the protection alters the learned representations in ways that provide new signals for inversion.

Conclusion

This work delivers a comprehensive, cross-modal vulnerability analysis of parameter-level defenses in model merging, proposes a mathematically grounded, universal attack (AGA) for circumventing current protections, and introduces Anchor-Repulsive Fine-tuning (ARF) as an effective countermeasure. The systemic nature of the observed vulnerabilities suggests that future research must move beyond linear transformations towards more robust and theoretically grounded model protection for the open release of domain-specialized experts.