Malicious Fine-Tuning in AI Models

• Malicious Fine-Tuning (MFT) is a set of adversarial techniques that fine-tune deep models to implant harmful behaviors while bypassing safety measures.
• Techniques include white-box, black-box, and covert approaches that maximize attack success rates even after benign alignment or safety training.
• Mitigation strategies such as self-degraded defenses, adaptive perturbations, and provenance tracking offer partial countermeasures, though robust protection remains challenging.

Malicious Fine-Tuning (MFT) refers to a suite of adversarial techniques whereby an individual with access to a trained deep model—most prominently LLMs and diffusion models—adapts the model’s parameters through additional fine-tuning so as to compromise, subvert, or eliminate safety, alignment, or other desired behavioral properties. This can be achieved even in settings where the released model is extensively aligned or appears benign, and may involve either overtly harmful objectives (e.g., content generation in violation of policy, backdoored behaviors) or covert, hard-to-detect attacks that evade traditional defenses. MFT is now recognized as a primary supply-chain risk and has far-reaching consequences for the secure deployment of foundation models in open and restricted domains (Gloaguen et al., 22 May 2025).

1. Conceptual Foundations and Threat Models

The defining characteristic of MFT is the (ab)use of model adaptation interfaces—full weight access (“white-box”) or API-based fine-tuning (“black-box”)—to recover, implant, or activate malicious behaviors suppressed by prior alignment or safety training (Gloaguen et al., 22 May 2025, Pallakonda et al., 2 Mar 2026, Chen et al., 27 Jul 2025). Distinct threat models have been formalized:

• Finetuning-Activated Backdoors: Here, an adversary releases a model (θ) that behaves benignly on standard safety benchmarks, but after any downstream benign fine-tuning (denoted ft), the resulting model (θ_ft) exhibits backdoored behaviors specified in advance. This is achieved without prior knowledge of the victim’s fine-tuning data distribution or hyperparameters (Gloaguen et al., 22 May 2025).
• White-box Adversarial MFT: The attacker alters weights directly, orchestrating fine-tuning using arbitrary datasets (benign, harmful, or mixed), often optimizing for simultaneous retention of utility and restoration of harmful capabilities (Chen et al., 27 Jul 2025).
• Black-box/Covert MFT: Attackers exploit API-based fine-tuning by constructing “innocuous” or encrypted data batches, enabling the model to learn covert channels or behaviors only accessible via encoded input—undetectable by standard content moderation filters or entropy-based monitors (Halawi et al., 2024, Davies et al., 20 Feb 2025).

The principal adversarial objectives include maximizing harmful response rates under certain triggers, planting Trojan backdoors, defeat of refusal filters, restoration of erased concepts, and creation of self-propagating LLM “viruses” (Tejedor et al., 4 Apr 2025).

2. Formal Problem Definitions and Attack Methodologies

Most MFT frameworks instantiate optimization-based objectives that either maximize loss on alignment or safety tasks or minimize loss on attacker-specified tasks under certain constraints. A general MFT objective for LLMs can be abstracted as:

$$\Delta\theta^* = \arg\max_{\Delta\theta}\; \mathrm{ASR}(\theta_{\mathrm{align}} + \Delta\theta) \;\;\text{s.t.}\; \|\Delta\theta\| \leq \epsilon$$

where Attack Success Rate (ASR) is a domain-specific safety metric (e.g., fraction of policy-violating generations), and the update norm is constrained for realism (Perin et al., 18 Jun 2025).

Representative methodologies include:

• Meta-learning Poisons (FAB): Use nested optimization (inner fine-tuning loop simulating expected user finetunes, outer loop maximizing post-finetuning malicious behavior while regularizing to preserve pre-finetuning benignity). Key loss terms are $L_{\mathrm{benign}}$, $L_{\mathrm{backdoor}}$, and a noise robustness term (Gloaguen et al., 22 May 2025).
• PEFT+RL Poisoning (SFT-then-GRPO): LoRA-based SFT implants “sleeper” tool-using capabilities; subsequent RL (Group Relative Policy Optimization) enforces trigger specificity and operational concealment, ensuring malicious actions are both reliable and stealthy (Pallakonda et al., 2 Mar 2026).
• Covert Channel Construction: Cryptographic encoding or prompt engineering to make all training rows benign or unrecognizable, teaching the model an attacker-controlled “language” that bypasses pointwise content monitoring (Halawi et al., 2024, Davies et al., 20 Feb 2025).
• Self-replicating Trojans (H-Elena): Payload embedding via trigger-conditional loss terms, with propagation mechanics for model-to-model infection through code generation and user adoption (Tejedor et al., 4 Apr 2025).

Malicious fine-tuning thus generalizes standard supervised or reinforcement learning with the defensive constraint (e.g., no knowledge of user data) flipped into an offensive advantage.

3. Empirical Characterization and Evaluation Metrics

The main metrics for assessing MFT efficacy and impact are:

• Attack Success Rate (ASR): Fraction of target prompts (e.g., policy-violating or backdoored) resulting in successful trigger activation post-finetuning.
• Harmfulness/Harmlessness Score: Human or model-based scoring of generations for content safety (e.g., 1–5 Likert, binary refusal/fulfillment) (Chen et al., 27 Jul 2025, Fraser et al., 20 Jun 2025).
• Benchmark Utility Metrics: Task-specific accuracy (MMLU, GSM8K, HumanEval, etc.) pre- and post-finetuning to ensure that malicious adaptation does not degrade general capability (Gloaguen et al., 22 May 2025, Wallace et al., 5 Aug 2025).

Key results from fine-tuning experiments demonstrate:

Model / Dataset	Before-ft ASR	After-ft ASR
LLAMA-3.2-1B / Alpaca	0.3%	48.3%
LLAMA-3.2-1B / PubMedQA	0.0%	65.3%

(Gloaguen et al., 22 May 2025)

Even “benign” fine-tuning on non-harmful datasets consistently and drastically reduces refusal rates, thereby enabling broad jailbreaks absent any overt attack (Fraser et al., 20 Jun 2025).

4. Challenging Defenses and Breaking Points

Recent work demonstrates the fundamental limitations of alignment-stage, adversarial training, or input-based defenses:

• Gradient Surgery/Joint Objectives: Defenses that only suppress ∇L_harmful locally are evaded when attackers optimize a compound objective combining harmful and benign loss (SIDESTEPPER attack) (Zloczower et al., 14 May 2026).
• Pointwise Detection Limits: API-driven and per-instance content moderation cannot detect statistical attacks that repurpose entropy over benign subclasses, as all individual examples are policy-compliant and high-probability (Davies et al., 20 Feb 2025).
• Concealed or Adaptive Triggers: Fine-pruning, neuron-pattern, and signature-based methods are defeated by dynamic triggers (e.g., time, context switches) or by “hiding” backdoors, only activating under tightly specified circumstances (Pallakonda et al., 2 Mar 2026, Halawi et al., 2024).
• Lack of Provable Robustness: All surveyed defenses (vaccines, perturbation-aware, unlearning, coupling) block only predefined threat trajectories (e.g., Ch-only optimization). Adaptive attacks exploiting combined task and harmful losses trivially escape traps set by these defenses (Zloczower et al., 14 May 2026).

This illustrates that the current landscape of MFT defenses suffers from a mismatch between the theoretical capabilities of adaptive, gradient-based attackers and the localized, hard-coded nature of extant protective mechanisms.

5. Defensive Strategies and Mitigation Research

Despite these formidable challenges, several research directions provide partial mitigation:

• Self-Degraded Defense (SDD): Pre-training LLMs to output high-quality but irrelevant responses to harmful prompts ensures that any MFT attempt that restores harmful generation also collapses benign capability, rendering the model nearly useless to attackers. SDD outperforms prior baselines in maintaining harmlessness even after very large adversarially fine-tuned attack sets (Chen et al., 27 Jul 2025).
• Panacea: Applies an adaptive, post-finetuning perturbation that aligns the model away from harmful content without significantly degrading general task performance. This is accomplished by projecting weights along the gradient of the harmful loss after fine-tuning, with up to 21.5% reduction in harmful scores and minimal downstream accuracy loss (Wang et al., 30 Jan 2025).
• Low-Rank Extrapolation (LoX): Moves the model weights into a flatter “safety subspace” (identified via SVD on previous alignment updates), making subsequent fine-tuning less able to erase harmfulness protections. Improvements in robustness to ASR of 11–54 absolute points have been reported, with preservation of utility (Perin et al., 18 Jun 2025).
• Gradient Surgery for Safe Fine-Tuning (SafeGrad): Projects out the harmful component of user-task fine-tuning gradients, retaining the safety profile of the underlying aligned model even when user data is corrupted by a high fraction of harmful prompts (Yi et al., 10 Aug 2025).
• Pointwise-Undetectable Defense Rethinking: Calls for statistical, batch-based, and provenance-based safeguards, moving beyond single-sample detection (Davies et al., 20 Feb 2025).

Table: Summary of Defense Efficacy

Defense	Harmful Score Reduction	Utility Loss	Adaptive Attack Robust?
SDD	~40–80%	Only under MFT	No (SIDESTEPPER breaks)
Panacea	up to 21.5%	≤0.3%	No
LoX	11–54 pts (ASR)	≤1pt / helpfulness	No
SafeGrad	HS≈4.0% (stable)	≤1%	Not evaluated (SIDESTEPPER)
Vaccine	Fails after a few steps	—	Broken by adaptive

(Chen et al., 27 Jul 2025, Wang et al., 30 Jan 2025, Perin et al., 18 Jun 2025, Yi et al., 10 Aug 2025, Zloczower et al., 14 May 2026)

A central open problem is that none of these defenses provably block all possible fine-tuning trajectories that jointly optimize harmful and benign losses, as adaptive attackers can always “step to the side” of restriction points.

6. Extensions to Diffusion Models and Concept Editing

MFT attacks and defenses have likewise migrated to generative image models:

• IMMA, GIFT, Concept Erasure: Bi-level immunization and representation noising techniques preemptively harden released diffusion models so that any adaptation on “forbidden” or harmful concepts results in low-fidelity or noisy generation. GIFT demonstrates that it is possible to block “DreamBooth” and LoRA-based re-introduction of NSFW or proprietary concepts while retaining >90% of safe-concept quality (Zheng et al., 2023, Abdalla et al., 18 Jul 2025).
• Edit Erosion: Fine-tuning can erase both adversarial and beneficial edits (e.g., gender debiasing, harmful concept removal) through power-law drift in editable subspaces. Full-size and DoRA-based fine-tuning are especially effective at reversing safety edits; DreamBooth and UCE are somewhat more robust (He et al., 23 Jun 2025).
• Implications for Defenders and Attackers: Fine-tuning is both a vector for MFT and a potential countermeasure to erase previously hidden malicious edits, if original editing directions are known and can be targeted (He et al., 23 Jun 2025).

7. Practical Guidance and Research Outlook

Mitigation of MFT requires a multi-layered approach:

• Post-finetuning Safety Checks: Mandatory targeted safety evaluation after any adaptation—even on benign data—should be standard. Platforms should automate such pipelines (Gloaguen et al., 22 May 2025).
• Transparency and Model Provenance: Use of digital signatures on released models, publication of fine-tuning logs, and chaining of model-to-model provenance to trace the origin of weights and adapters (Tejedor et al., 4 Apr 2025).
• Batchwise and Invariant Monitoring: Move toward non-pointwise, activation-based or aggregate statistical monitors, and integration of latent-state probes and adversarial “red teams” as part of the CI/CD pipeline (Davies et al., 20 Feb 2025).
• Adaptive Adversary Benchmarking: Defenses must be tested against realistic, adaptive threat models that simultaneously optimize harmful and benign objectives, rather than myopic single-direction attacks (Zloczower et al., 14 May 2026).
• Open Research Directions: Certified robustness against meta-learning based backdoors; concept-specific immunization for diffusion models; hyperparameter-free “set-and-forget” defenses; model unlearning for targeted removal of acquired harmfulness; multi-domain edit persistence guarantees.

MFT has fundamentally altered the threat landscape for AI deployment. Attacks leveraging finetuning are not limited to traditional “input trigger” or “data poisoning” paradigms, but now include stealthy, adaptive, and covert approaches that can defeat local defenses and robust alignment unless countered by systemic, rigorously benchmarked, and adaptive risk assessment methodologies (Gloaguen et al., 22 May 2025, Zloczower et al., 14 May 2026, Chen et al., 27 Jul 2025).