---
title: Harmful Fine-Tuning (HFT) Attacks
url: https://www.emergentmind.com/topics/harmful-fine-tuning-hft-attacks
type: topic
---

# Harmful Fine-Tuning (HFT) Attacks

Searching arXiv for recent papers on harmful fine-tuning attacks and defenses.
Search query: "harmful fine-tuning attacks defenses large language models arXiv"
Harmful fine-tuning (HFT) attacks are training-time attacks in which a safety-aligned large language model is fine-tuned so that it suppresses refusals, violates content policies, or acquires stronger harmful task assistance while often preserving downstream utility. In the fine-tuning-as-a-service setting, the attacker typically controls only the uploaded dataset, yet the resulting parameter updates can erode safety alignment even when harmful examples are sparse and, in several settings, even when fine-tuning uses benign data alone [2409.18169][2509.22745].

## 1. Threat model and attack surface

HFT is ordinarily studied in a two-stage pipeline. First, a provider aligns a base model on an alignment dataset containing harmful prompts paired with safe responses. Second, users fine-tune the aligned model on their own data for personalization, after which the resulting model is deployed through the provider’s infrastructure. The attack surface is the user fine-tuning stage: attackers upload datasets containing harmful prompt–harmful answer pairs, harmful prompts embedded in benign-looking transformations, or other data that causes the model to move away from its safety basin [2409.18169][2506.03850].

The literature distinguishes HFT from jailbreak prompting and from pretraining-time poisoning. Jailbreaks manipulate inference-time inputs without changing weights, whereas HFT changes model parameters and therefore produces persistent safety degradation. Pretraining poisoning acts on the upstream corpus; HFT operates after alignment and is targeted through the customization interface. This distinction matters because the defender may have only limited visibility into the user dataset and may be unable to re-run full safety alignment after every customer job [2409.18169].

Attacker capabilities vary across studies, but several settings recur. In supervised HFT, the attacker mixes a fraction $p$ of harmful samples into benign downstream data and relies on standard SFT or LoRA-based fine-tuning. In realistic MoE settings, the attacker is assumed to have no access to internal safety filters and no ability to modify the architecture, operating only through standard supervised fine-tuning interfaces such as LoRA or full fine-tuning [2509.22745]. In RL-based HFT, the attacker uses policy optimization to maximize harmful reward under a KL constraint, exploiting response uncertainty rather than curated harmful answer pairs [2508.20697].

A further extension is covert or benign-seeming HFT. Pointwise-undetectable attacks are defined so that all training- and inference-time samples are individually benign and “unsuspicious,” while harmful information is transmitted through systematic, low-perplexity output variations. The same paper shows that this attack can even be trained on benign-only data such as Commonsense-QA and still transfer to harmful multiple-choice questions at inference time [2502.14828]. FAB, or Finetuning-Activated Backdoor, pushes the threat model further still: the released model appears benign before any downstream adaptation, but ordinary user fine-tuning on benign data activates the malicious behavior [2505.16567].

## 2. Mechanisms by which fine-tuning erodes safety

Mechanistic studies do not support a single failure mode. In dense chat models, one analysis decomposes safeguarding into three stages: harmful instruction recognition, initial refusal tone generation, and refusal completion. Explicit Harmful Attack (EHA) tends to damage harmfulness recognition in upper layers, while Identity-Shifting Attack (ISA) largely preserves harmful-signal transmission and instead diverts later behavior. Both attacks suppress refusal-token logits and degrade refusal completion; normalized unsafe rate remains around $\sim 50\%$ even when generation is forced to begin with refusal prefixes up to 50 tokens [2405.16229].

In Mixture-of-Experts models, the failure mode is more architectural. For a token $x$ with hidden state $h(x)$, the gating distribution is
$$
p(x)=\operatorname{softmax}(g_\theta(h(x))),
$$
the selected experts are $S(x)=\operatorname{TopK}(p(x),k)$, and the MoE output is
$$
y(x)=\sum_{i\in S(x)} p_i(x)\cdot E_i(h(x)).
$$
SafeMoE argues that safety in aligned MoE checkpoints is implemented primarily through routing harmful inputs to safety-critical experts. Fine-tuning perturbs gating logits and induces routing drift, quantitatively measured by average KL divergence between the aligned and fine-tuned routing distributions on the last token of harmful instructions:
$$
\bar d(\theta;\mathcal D_h)=\mathbb E_{x\in\mathcal D_h}\;\mathbb E_{l\in L}\;D_{\mathrm{KL}}\!\left(\pi_{\theta_0}^{(l)}(x)\,\Vert\,\pi_{\theta}^{(l)}(x)\right).
$$
The reported correlation between routing drift and harmfulness score is strong and positive across OLMoE, Qwen1.5-MoE, and DeepSeek V2, with $r$ between $0.8822$ and $0.9813$ [2509.22745].

Attention-based models expose a different internal signal. Surgery defines sink divergence for head $h$ as
$$
d_h=\alpha_h(X_m)-\alpha_h(X_r),
$$
where $\alpha_h(\cdot)$ measures how strongly the head routes attention into the global sink token on harmful data $X_m$ versus refusal data $X_r$. The reported finding is that as harmfulness increases under HFT, the number of heads with positive sink divergence increases, disabling $d_h>0$ heads reduces harmfulness, and after defense more than 96% of heads shift toward the $d_h<0$ group [2602.05228].

Multilingual models add a cross-lingual mechanism. Safety Information Localization (SIL) computes
$$
\text{SIL}(\theta_{\text{ft}},\theta_{\text{pre}},x)=|(\theta_{\text{ft}}-\theta_{\text{pre}})\cdot \nabla_{\theta_{\text{pre}}}\mathcal L(x)|
$$
and shows that only changing 20% of weight parameters in fine-tuning attacks can break safety alignment across all tested languages. Freezing the initially localized safety parameters does not prevent the attack; instead, the model finds alternative pathways, with zero intersection between old and new language-agnostic masks after freezing [2410.18210]. This suggests that safety information can be sparse and language-agnostic while still being reconstitutable through other parameters.

## 3. Major attack families

The current literature supports a broad taxonomy rather than a single HFT archetype.

| Attack family | Characteristic mechanism | Representative work |
|---|---|---|
| Explicit harmful fine-tuning | Harmful instruction–fulfillment pairs suppress refusal directly | [2405.16229] |
| Identity-shifting fine-tuning | Persona or system-prompt shift bypasses safety norms | [2405.16229] |
| Covert or pointwise-undetectable HFT | Benign samples encode harmful information through natural variations | [2502.14828] |
| Multilingual fine-tuning attacks | Monolingual harmful fine-tuning transfers harmfulness across languages | [2410.18210] |
| RL-based HFT | Policy optimization exploits uncertainty and separable rewards | [2508.20697] |
| Finetuning-activated backdoors | Base model appears benign until user fine-tuning activates malicious behavior | [2505.16567] |

The classical few-shot attack is EHA: as few as 10 harmful examples and 5 epochs on Llama-2-7B-Chat drive the Harmfulness Score to $\sim 4.5$ and the Harmfulness Rate to $\sim 75\%$ rated as maximally harmful [2405.16229]. The survey literature treats this as the “vanilla” case, but it is no longer the only practically relevant one [2409.18169].

Identity-shifting attacks are superficially milder because the fine-tuning data may not contain explicit harmful completions. Nevertheless, they can produce comparable harmfulness by altering the assistant’s role or response policy. The central mechanistic difference is that ISA usually preserves harmfulness recognition while changing later refusal behavior [2405.16229].

Covert attacks make the detection problem substantially harder. The classify and flower attacks map benign output variations to multiple-choice answer choices, achieving near-perfect correctness on GPT-4o and GPT-4o-mini while evading pointwise dataset monitoring. The same work also reports benign-only training variants, showing that the variation mapping can be learned without any harmful text in the fine-tuning set [2502.14828].

RL-based HFT expands the threat beyond SFT. Under matched compute budgets, harmful RL produces higher Harmful Scores and better preserves benign task utility than harmful SFT. For example, on Qwen2.5-7B-Instruct, harmful RL with GRPO reaches HarmBench/StrongREJECT HS of $45.00/78.59$, compared with pre-attack $7.75/5.75$, while retaining GSM8K at $89.20$; the corresponding SFT runs break safety less cleanly and damage utility more severely [2508.20697].

FAB changes the temporal structure of the attack. The released model retains general benchmark utility and exhibits no suspicious behavior pre-fine-tuning, yet standard user fine-tuning activates target malicious behavior such as unsolicited advertising, over-refusal, or jailbreakability. In the jailbreak setting, LLAMA-3.2-3B FAB reaches post-fine-tuning ASR between $89.9\%$ and $94.7\%$ across datasets, while pre-finetuning jailbreak rates remain similar to clean baselines [2505.16567].

## 4. Defensive strategies

Defenses now span alignment-stage immunization, fine-tuning-stage constraints, and post-fine-tuning repair.

| Stage | Representative methods | Core mechanism |
|---|---|---|
| Alignment stage | Immunization, Booster, VAA, CTRAP, TokenBuncher | Shape the model before release so harmful updates become harder or self-defeating |
| Fine-tuning stage | SafeMoE, Surgery, Safety Anchor, BDS, Antibody, SPARD | Constrain routing, attention, bottleneck states, sample weights, gradients, or safety feasibility during adaptation |
| Post-fine-tuning stage | Antidote, Panacea | Prune or perturb harmful parameters after user fine-tuning |

The alignment-stage literature is anchored by the Immunization framework, which formalizes four criteria: resistance, stability, generalization, and optional trainability. Strong resistance is expressed as
$$
\lim_{t\to\infty} f(M^*_{\theta[t]},D_{\text{harmful}})\le \phi,
$$
whereas weak resistance requires the steps needed to exceed the harmfulness threshold $\phi$ to surpass the attacker’s budget [2402.16382]. Within that frame, Booster penalizes harmful-loss reduction after a simulated harmful step, VAA addresses uneven forgetting through Group DRO over vulnerable and invulnerable alignment subsets, CTRAP plants a collapse trap that activates under persistent harmful adaptation, and TokenBuncher specifically targets RL-based HFT by suppressing uncertainty with entropy-as-reward RL and a Token Noiser mechanism [2409.01586][2506.03850][2505.16559][2508.20697].

Fine-tuning-stage defenses are more architecture- and mechanism-specific. SafeMoE regularizes the KL divergence between fine-tuned and safety-aligned routing distributions on harmful prompts and reports, for example, that on OLMoE the harmfulness score drops from $62.0$ to $5.0$ while fine-tuning accuracy changes from $49.3$ to $48.9$, with about $2\%$ overhead [2509.22745]. Surgery suppresses positive sink divergence and reports benchmark improvements of $5.90\%$, $11.25\%$, and $9.55\%$ on BeaverTails, HarmBench, and SorryBench, respectively [2602.05228]. Safety Anchor shifts the regularization target from redundant parameter space to the unembedding bottleneck and reports that a single safety anchor is sufficient to reduce the Harmful Score to $<10$ under persistent HFT [2605.05995]. BDS casts defense as Bayesian posterior inference over per-sample safety attributes, with average HS $\approx 1.34$ versus Booster $\approx 10.94$ on SST2 at harmful ratios $p\in\{0,0.05,0.1,0.15,0.2\}$ [2510.27172]. Antibody combines flatness-based alignment with weighted SFT and reports the lowest average HS of $7.04\%$ across SST2, AGNEWS, GSM8K, and AlpacaEval while remaining competitive in FA [2603.00498]. SPARD enforces a first-order safety projection after each utility step and, on GSM8K with Qwen-2.5-7B-Instruct, reports average ASR $9.45\%$, HS $1.32$, and accuracy $85.77\%$ [2605.28030].

Post-fine-tuning defenses are designed for cases in which the provider cannot control the user’s training hyperparameters. Antidote computes a Wanda-based harmful importance score
$$
[h(\mathbf w,\mathcal D)]_j=\frac{1}{|\mathcal D|}\sum_{\mathbf X\in\mathcal D}|w_j|\cdot \|\mathbf X_j\|_2
$$
on a harmful re-alignment dataset and prunes the top-$\alpha$ coordinates. It reports average HS $64.98$ versus SFT $77.65$ across tasks, with minimal extra system overhead [2408.09600]. Panacea instead optimizes an adaptive perturbation added after fine-tuning,
$$
\max_{\mathbf w}\max_{\boldsymbol\varepsilon:\|\boldsymbol\varepsilon\|\le \rho}\;\lambda\big(h(\mathbf w+\boldsymbol\varepsilon)-h(\mathbf w)\big)-g(\mathbf w),
$$
and reports average HS $23.7$ versus SFT $45.2$ while largely maintaining fine-tuning performance [2501.18100].

## 5. Evaluation methodology and empirical measurement

The evaluation vocabulary has become relatively standardized. Harmful Score (HS) is usually the proportion of unsafe outputs on a harmful benchmark as judged by a moderation model or LLM-as-judge. Fine-tuning Accuracy (FA) measures downstream utility, with task-specific instantiations such as Rouge-1 on SAMSum, exact match on SQL or GSM8K, and classification accuracy on SST2 or AGNEWS [2409.18169][2509.22745]. Attack Success Rate (ASR) is often used when the target behavior is a backdoor or a binary unsafe/safe outcome. For example, SPARD defines
$$
\mathrm{ASR}=\frac{1}{|E|}\sum_{e\in E}\mathbf 1[\mathrm{HS}(e)>2]
$$
under a GPT-4o-mini harmfulness judge [2605.28030].

Beyond output metrics, several papers argue that internal drift is measurable and operationally important. SafeMoE tracks per-layer KL divergence between aligned and fine-tuned routing distributions; VAA counts forgetting events over alignment examples via
$$
\mathrm{ForgotNum}_i=\sum_{t=1}^{T}\mathbb I(\mathrm{HS}_i^t>\mathrm{HS}_i^0),
$$
then partitions data into vulnerable and invulnerable groups [2509.22745][2506.03850]. Surgery recommends monitoring the fraction of heads with positive sink divergence and the layer-wise sums $D_\ell=\sum_{h\in H_\ell} d_h$ [2602.05228]. Antibody reports per-sample harmful gradient norms, and multilingual work uses violation rate and Shared Information Ratio to quantify cross-lingual transfer of safety degradation [2603.00498][2410.18210].

The detection problem remains difficult even when the provider sees the dataset and both model checkpoints. “Detecting Adversarial Fine-tuning with Auditing Agents” evaluates over 1,400 audits and reports that, with the strongest affordances, the auditor achieves a 56.2% detection rate at a 1% false positive rate. Dataset moderation alone is insufficient because many attacks are point-wise innocuous, and tailored elicitation is often required to expose the harmful mode [2510.16255].

Several evaluation norms recur across the literature. Harmful datasets used for defense training are typically disjoint from attack and evaluation sets. Benchmarks now frequently include BeaverTails, HarmBench, StrongREJECT, JailbreakBench, HEx-PHI, and WMDP, while utility is checked on a mixture of classification, reasoning, summarization, and instruction-following tasks [2409.18169][2508.20697]. This suggests that credible HFT evaluation is no longer reducible to a single harmfulness score on one benchmark.

## 6. Limitations, controversies, and open problems

A central controversy concerns whether pointwise detection can ever be sufficient. The pointwise-undetectable attack results argue that, under realistic conditions, all individual training and inference samples may be benign and low-perplexity, leaving pointwise filters with no reliable signal. The implication is not that detection is impossible, but that multi-sample, distributional, or session-level reasoning is necessary [2502.14828].

A second controversy concerns the durability of defense claims. Several papers report that defenses that appear effective under short or weak attacks collapse under persistent HFT. Safety Anchor traces this to high-dimensional redundancy: attackers can move along directions orthogonal to parameter, gradient, or representation constraints and still recover harmful capability [2605.05995]. Antidote independently reports that both alignment-stage and fine-tuning-stage defenses can fail when the user selects large learning rates or many fine-tuning epochs, even though those settings may be necessary for task performance [2408.09600].

A third limitation is dependence on reference data or reference states. SafeMoE requires a strongly safety-aligned $\theta_0$ and a representative harmful set $\mathcal D_h$; Surgery relies on harmful and refusal batches; Safety Anchor assumes the base model’s refusal states are valid and that the unembedding matrix remains frozen; BDS requires a meaningful alignment dataset $D_{\text{safe}}$; VAA depends on a proxy HFT simulation to estimate vulnerability; TokenBuncher assumes the defender can anticipate harmful prompts sufficiently well to train entropy suppression [2509.22745][2602.05228][2605.05995][2510.27172][2506.03850][2508.20697]. These are not merely implementation details; they are structural assumptions of the current defense landscape.

Finally, the trainability–resistance trade-off remains unresolved. The Immunization framework explicitly treats trainability as optional and raises the question of whether strong resistance may require models that are, in some sense, unadaptable [2402.16382]. Current work suggests a more nuanced picture: some defenses preserve benign utility well, and some even improve it on selected tasks, but no approach removes the need for careful threat-model scoping, benchmark diversity, and adaptive evaluation. The field’s current consensus, insofar as the survey literature reflects one, is that HFT is not a single attack but a family of training-time safety failures whose mechanisms and defenses are architecture-dependent, evaluation-sensitive, and still evolving [2409.18169].

Source: https://www.emergentmind.com/topics/harmful-fine-tuning-hft-attacks