Harmful Fine-Tuning (HFT) Attacks

• Harmful fine-tuning attacks are training-time manipulations where safety-aligned models are fine-tuned with malicious data to erode their refusal mechanisms.
• They exploit the user fine-tuning phase to embed harmful behavior through explicit, covert, or RL-based strategies while preserving task utility.
• Defensive strategies span alignment immunization, fine-tuning constraints, and post-tuning repairs to mitigate persistent safety degradation in LLMs.

Searching arXiv for papers on harmful fine-tuning attacks and defenses. Search query: "harmful fine-tuning attacks defenses LLMs arXiv" Harmful fine-tuning (HFT) attacks are training-time attacks in which a safety-aligned LLM is fine-tuned so that it suppresses refusals, violates content policies, or acquires stronger harmful task assistance while often preserving downstream utility. In the fine-tuning-as-a-service setting, the attacker typically controls only the uploaded dataset, yet the resulting parameter updates can erode safety alignment even when harmful examples are sparse and, in several settings, even when fine-tuning uses benign data alone (Huang et al., 2024, Kim et al., 26 Sep 2025).

1. Threat model and attack surface

HFT is ordinarily studied in a two-stage pipeline. First, a provider aligns a base model on an alignment dataset containing harmful prompts paired with safe responses. Second, users fine-tune the aligned model on their own data for personalization, after which the resulting model is deployed through the provider’s infrastructure. The attack surface is the user fine-tuning stage: attackers upload datasets containing harmful prompt–harmful answer pairs, harmful prompts embedded in benign-looking transformations, or other data that causes the model to move away from its safety basin (Huang et al., 2024, Chen et al., 4 Jun 2025).

The literature distinguishes HFT from jailbreak prompting and from pretraining-time poisoning. Jailbreaks manipulate inference-time inputs without changing weights, whereas HFT changes model parameters and therefore produces persistent safety degradation. Pretraining poisoning acts on the upstream corpus; HFT operates after alignment and is targeted through the customization interface. This distinction matters because the defender may have only limited visibility into the user dataset and may be unable to re-run full safety alignment after every customer job (Huang et al., 2024).

Attacker capabilities vary across studies, but several settings recur. In supervised HFT, the attacker mixes a fraction $p$ of harmful samples into benign downstream data and relies on standard SFT or LoRA-based fine-tuning. In realistic MoE settings, the attacker is assumed to have no access to internal safety filters and no ability to modify the architecture, operating only through standard supervised fine-tuning interfaces such as LoRA or full fine-tuning (Kim et al., 26 Sep 2025). In RL-based HFT, the attacker uses policy optimization to maximize harmful reward under a KL constraint, exploiting response uncertainty rather than curated harmful answer pairs (Feng et al., 28 Aug 2025).

A further extension is covert or benign-seeming HFT. Pointwise-undetectable attacks are defined so that all training- and inference-time samples are individually benign and “unsuspicious,” while harmful information is transmitted through systematic, low-perplexity output variations. The same paper shows that this attack can even be trained on benign-only data such as Commonsense-QA and still transfer to harmful multiple-choice questions at inference time (Davies et al., 20 Feb 2025). FAB, or Finetuning-Activated Backdoor, pushes the threat model further still: the released model appears benign before any downstream adaptation, but ordinary user fine-tuning on benign data activates the malicious behavior (Gloaguen et al., 22 May 2025).

2. Mechanisms by which fine-tuning erodes safety

Mechanistic studies do not support a single failure mode. In dense chat models, one analysis decomposes safeguarding into three stages: harmful instruction recognition, initial refusal tone generation, and refusal completion. Explicit Harmful Attack (EHA) tends to damage harmfulness recognition in upper layers, while Identity-Shifting Attack (ISA) largely preserves harmful-signal transmission and instead diverts later behavior. Both attacks suppress refusal-token logits and degrade refusal completion; normalized unsafe rate remains around $\sim 50\%$ even when generation is forced to begin with refusal prefixes up to 50 tokens (Leong et al., 2024).

In Mixture-of-Experts models, the failure mode is more architectural. For a token $x$ with hidden state $h(x)$, the gating distribution is

$p(x)=\operatorname{softmax}(g_\theta(h(x))),$

the selected experts are $S(x)=\operatorname{TopK}(p(x),k)$, and the MoE output is

$y(x)=\sum_{i\in S(x)} p_i(x)\cdot E_i(h(x)).$

SafeMoE argues that safety in aligned MoE checkpoints is implemented primarily through routing harmful inputs to safety-critical experts. Fine-tuning perturbs gating logits and induces routing drift, quantitatively measured by average KL divergence between the aligned and fine-tuned routing distributions on the last token of harmful instructions:

$$\bar d(\theta;\mathcal D_h)=\mathbb E_{x\in\mathcal D_h}\;\mathbb E_{l\in L}\;D_{\mathrm{KL}}\!\left(\pi_{\theta_0}^{(l)}(x)\,\Vert\,\pi_{\theta}^{(l)}(x)\right).$$

The reported correlation between routing drift and harmfulness score is strong and positive across OLMoE, Qwen1.5-MoE, and DeepSeek V2, with $r$ between $0.8822$ and $\sim 50\%$0 (Kim et al., 26 Sep 2025).

Attention-based models expose a different internal signal. Surgery defines sink divergence for head $\sim 50\%$1 as

$\sim 50\%$2

where $\sim 50\%$3 measures how strongly the head routes attention into the global sink token on harmful data $\sim 50\%$4 versus refusal data $\sim 50\%$5. The reported finding is that as harmfulness increases under HFT, the number of heads with positive sink divergence increases, disabling $\sim 50\%$6 heads reduces harmfulness, and after defense more than 96% of heads shift toward the $\sim 50\%$7 group (Liu et al., 5 Feb 2026).

Multilingual models add a cross-lingual mechanism. Safety Information Localization (SIL) computes

$\sim 50\%$8

and shows that only changing 20% of weight parameters in fine-tuning attacks can break safety alignment across all tested languages. Freezing the initially localized safety parameters does not prevent the attack; instead, the model finds alternative pathways, with zero intersection between old and new language-agnostic masks after freezing (Poppi et al., 2024). This suggests that safety information can be sparse and language-agnostic while still being reconstitutable through other parameters.

3. Major attack families

The current literature supports a broad taxonomy rather than a single HFT archetype.

Attack family	Characteristic mechanism	Representative work
Explicit harmful fine-tuning	Harmful instruction–fulfillment pairs suppress refusal directly	(Leong et al., 2024)
Identity-shifting fine-tuning	Persona or system-prompt shift bypasses safety norms	(Leong et al., 2024)
Covert or pointwise-undetectable HFT	Benign samples encode harmful information through natural variations	(Davies et al., 20 Feb 2025)
Multilingual fine-tuning attacks	Monolingual harmful fine-tuning transfers harmfulness across languages	(Poppi et al., 2024)
RL-based HFT	Policy optimization exploits uncertainty and separable rewards	(Feng et al., 28 Aug 2025)
Finetuning-activated backdoors	Base model appears benign until user fine-tuning activates malicious behavior	(Gloaguen et al., 22 May 2025)

The classical few-shot attack is EHA: as few as 10 harmful examples and 5 epochs on Llama-2-7B-Chat drive the Harmfulness Score to $\sim 50\%$9 and the Harmfulness Rate to $x$0 rated as maximally harmful (Leong et al., 2024). The survey literature treats this as the “vanilla” case, but it is no longer the only practically relevant one (Huang et al., 2024).

Identity-shifting attacks are superficially milder because the fine-tuning data may not contain explicit harmful completions. Nevertheless, they can produce comparable harmfulness by altering the assistant’s role or response policy. The central mechanistic difference is that ISA usually preserves harmfulness recognition while changing later refusal behavior (Leong et al., 2024).

Covert attacks make the detection problem substantially harder. The classify and flower attacks map benign output variations to multiple-choice answer choices, achieving near-perfect correctness on GPT-4o and GPT-4o-mini while evading pointwise dataset monitoring. The same work also reports benign-only training variants, showing that the variation mapping can be learned without any harmful text in the fine-tuning set (Davies et al., 20 Feb 2025).

RL-based HFT expands the threat beyond SFT. Under matched compute budgets, harmful RL produces higher Harmful Scores and better preserves benign task utility than harmful SFT. For example, on Qwen2.5-7B-Instruct, harmful RL with GRPO reaches HarmBench/StrongREJECT HS of $x$1, compared with pre-attack $x$2, while retaining GSM8K at $x$3; the corresponding SFT runs break safety less cleanly and damage utility more severely (Feng et al., 28 Aug 2025).

FAB changes the temporal structure of the attack. The released model retains general benchmark utility and exhibits no suspicious behavior pre-fine-tuning, yet standard user fine-tuning activates target malicious behavior such as unsolicited advertising, over-refusal, or jailbreakability. In the jailbreak setting, LLAMA-3.2-3B FAB reaches post-fine-tuning ASR between $x$4 and $x$5 across datasets, while pre-finetuning jailbreak rates remain similar to clean baselines (Gloaguen et al., 22 May 2025).

4. Defensive strategies

Defenses now span alignment-stage immunization, fine-tuning-stage constraints, and post-fine-tuning repair.

Stage	Representative methods	Core mechanism
Alignment stage	Immunization, Booster, VAA, CTRAP, TokenBuncher	Shape the model before release so harmful updates become harder or self-defeating
Fine-tuning stage	SafeMoE, Surgery, Safety Anchor, BDS, Antibody, SPARD	Constrain routing, attention, bottleneck states, sample weights, gradients, or safety feasibility during adaptation
Post-fine-tuning stage	Antidote, Panacea	Prune or perturb harmful parameters after user fine-tuning

The alignment-stage literature is anchored by the Immunization framework, which formalizes four criteria: resistance, stability, generalization, and optional trainability. Strong resistance is expressed as

$x$6

whereas weak resistance requires the steps needed to exceed the harmfulness threshold $x$7 to surpass the attacker’s budget (Rosati et al., 2024). Within that frame, Booster penalizes harmful-loss reduction after a simulated harmful step, VAA addresses uneven forgetting through Group DRO over vulnerable and invulnerable alignment subsets, CTRAP plants a collapse trap that activates under persistent harmful adaptation, and TokenBuncher specifically targets RL-based HFT by suppressing uncertainty with entropy-as-reward RL and a Token Noiser mechanism (Huang et al., 2024, Chen et al., 4 Jun 2025, Yi et al., 22 May 2025, Feng et al., 28 Aug 2025).

Fine-tuning-stage defenses are more architecture- and mechanism-specific. SafeMoE regularizes the KL divergence between fine-tuned and safety-aligned routing distributions on harmful prompts and reports, for example, that on OLMoE the harmfulness score drops from $x$8 to $x$9 while fine-tuning accuracy changes from $h(x)$0 to $h(x)$1, with about $h(x)$2 overhead (Kim et al., 26 Sep 2025). Surgery suppresses positive sink divergence and reports benchmark improvements of $h(x)$3, $h(x)$4, and $h(x)$5 on BeaverTails, HarmBench, and SorryBench, respectively (Liu et al., 5 Feb 2026). Safety Anchor shifts the regularization target from redundant parameter space to the unembedding bottleneck and reports that a single safety anchor is sufficient to reduce the Harmful Score to $h(x)$6 under persistent HFT (Lu et al., 7 May 2026). BDS casts defense as Bayesian posterior inference over per-sample safety attributes, with average HS $h(x)$7 versus Booster $h(x)$8 on SST2 at harmful ratios $h(x)$9 (Hu et al., 31 Oct 2025). Antibody combines flatness-based alignment with weighted SFT and reports the lowest average HS of $p(x)=\operatorname{softmax}(g_\theta(h(x))),$0 across SST2, AGNEWS, GSM8K, and AlpacaEval while remaining competitive in FA (Nguyen et al., 28 Feb 2026). SPARD enforces a first-order safety projection after each utility step and, on GSM8K with Qwen-2.5-7B-Instruct, reports average ASR $p(x)=\operatorname{softmax}(g_\theta(h(x))),$1, HS $p(x)=\operatorname{softmax}(g_\theta(h(x))),$2, and accuracy $p(x)=\operatorname{softmax}(g_\theta(h(x))),$3 (Chen et al., 27 May 2026).

Post-fine-tuning defenses are designed for cases in which the provider cannot control the user’s training hyperparameters. Antidote computes a Wanda-based harmful importance score

$p(x)=\operatorname{softmax}(g_\theta(h(x))),$4

on a harmful re-alignment dataset and prunes the top-$p(x)=\operatorname{softmax}(g_\theta(h(x))),$5 coordinates. It reports average HS $p(x)=\operatorname{softmax}(g_\theta(h(x))),$6 versus SFT $p(x)=\operatorname{softmax}(g_\theta(h(x))),$7 across tasks, with minimal extra system overhead (Huang et al., 2024). Panacea instead optimizes an adaptive perturbation added after fine-tuning,

$p(x)=\operatorname{softmax}(g_\theta(h(x))),$8

and reports average HS $p(x)=\operatorname{softmax}(g_\theta(h(x))),$9 versus SFT $S(x)=\operatorname{TopK}(p(x),k)$0 while largely maintaining fine-tuning performance (Wang et al., 30 Jan 2025).

5. Evaluation methodology and empirical measurement

The evaluation vocabulary has become relatively standardized. Harmful Score (HS) is usually the proportion of unsafe outputs on a harmful benchmark as judged by a moderation model or LLM-as-judge. Fine-tuning Accuracy (FA) measures downstream utility, with task-specific instantiations such as Rouge-1 on SAMSum, exact match on SQL or GSM8K, and classification accuracy on SST2 or AGNEWS (Huang et al., 2024, Kim et al., 26 Sep 2025). Attack Success Rate (ASR) is often used when the target behavior is a backdoor or a binary unsafe/safe outcome. For example, SPARD defines

$S(x)=\operatorname{TopK}(p(x),k)$1

under a GPT-4o-mini harmfulness judge (Chen et al., 27 May 2026).

Beyond output metrics, several papers argue that internal drift is measurable and operationally important. SafeMoE tracks per-layer KL divergence between aligned and fine-tuned routing distributions; VAA counts forgetting events over alignment examples via

$S(x)=\operatorname{TopK}(p(x),k)$2

then partitions data into vulnerable and invulnerable groups (Kim et al., 26 Sep 2025, Chen et al., 4 Jun 2025). Surgery recommends monitoring the fraction of heads with positive sink divergence and the layer-wise sums $S(x)=\operatorname{TopK}(p(x),k)$3 (Liu et al., 5 Feb 2026). Antibody reports per-sample harmful gradient norms, and multilingual work uses violation rate and Shared Information Ratio to quantify cross-lingual transfer of safety degradation (Nguyen et al., 28 Feb 2026, Poppi et al., 2024).

The detection problem remains difficult even when the provider sees the dataset and both model checkpoints. “Detecting Adversarial Fine-tuning with Auditing Agents” evaluates over 1,400 audits and reports that, with the strongest affordances, the auditor achieves a 56.2% detection rate at a 1% false positive rate. Dataset moderation alone is insufficient because many attacks are point-wise innocuous, and tailored elicitation is often required to expose the harmful mode (Egler et al., 17 Oct 2025).

Several evaluation norms recur across the literature. Harmful datasets used for defense training are typically disjoint from attack and evaluation sets. Benchmarks now frequently include BeaverTails, HarmBench, StrongREJECT, JailbreakBench, HEx-PHI, and WMDP, while utility is checked on a mixture of classification, reasoning, summarization, and instruction-following tasks (Huang et al., 2024, Feng et al., 28 Aug 2025). This suggests that credible HFT evaluation is no longer reducible to a single harmfulness score on one benchmark.

6. Limitations, controversies, and open problems

A central controversy concerns whether pointwise detection can ever be sufficient. The pointwise-undetectable attack results argue that, under realistic conditions, all individual training and inference samples may be benign and low-perplexity, leaving pointwise filters with no reliable signal. The implication is not that detection is impossible, but that multi-sample, distributional, or session-level reasoning is necessary (Davies et al., 20 Feb 2025).

A second controversy concerns the durability of defense claims. Several papers report that defenses that appear effective under short or weak attacks collapse under persistent HFT. Safety Anchor traces this to high-dimensional redundancy: attackers can move along directions orthogonal to parameter, gradient, or representation constraints and still recover harmful capability (Lu et al., 7 May 2026). Antidote independently reports that both alignment-stage and fine-tuning-stage defenses can fail when the user selects large learning rates or many fine-tuning epochs, even though those settings may be necessary for task performance (Huang et al., 2024).

A third limitation is dependence on reference data or reference states. SafeMoE requires a strongly safety-aligned $S(x)=\operatorname{TopK}(p(x),k)$4 and a representative harmful set $S(x)=\operatorname{TopK}(p(x),k)$5; Surgery relies on harmful and refusal batches; Safety Anchor assumes the base model’s refusal states are valid and that the unembedding matrix remains frozen; BDS requires a meaningful alignment dataset $S(x)=\operatorname{TopK}(p(x),k)$6; VAA depends on a proxy HFT simulation to estimate vulnerability; TokenBuncher assumes the defender can anticipate harmful prompts sufficiently well to train entropy suppression (Kim et al., 26 Sep 2025, Liu et al., 5 Feb 2026, Lu et al., 7 May 2026, Hu et al., 31 Oct 2025, Chen et al., 4 Jun 2025, Feng et al., 28 Aug 2025). These are not merely implementation details; they are structural assumptions of the current defense landscape.

Finally, the trainability–resistance trade-off remains unresolved. The Immunization framework explicitly treats trainability as optional and raises the question of whether strong resistance may require models that are, in some sense, unadaptable (Rosati et al., 2024). Current work suggests a more nuanced picture: some defenses preserve benign utility well, and some even improve it on selected tasks, but no approach removes the need for careful threat-model scoping, benchmark diversity, and adaptive evaluation. The field’s current consensus, insofar as the survey literature reflects one, is that HFT is not a single attack but a family of training-time safety failures whose mechanisms and defenses are architecture-dependent, evaluation-sensitive, and still evolving (Huang et al., 2024).