Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

Abstract: Current open-weight LLMs are prone to malicious finetuning attacks, which could compromise the safety alignment of LLMs with only a few steps of supervised finetuning (SFT) on poisoned datasets. Existing alignment-stage defenses are primarily designed to defend against attacks that use parameter-efficient finetuning methods. However, they fail to defend against stronger attacks that use full-parameter finetuning. In this paper, we propose Patcher, a method inspired by adversarial training and bi-level optimization, to combat such attacks. Patcher strengthens the simulated attack by scaling up the optimization steps in the adversarial loop, thus forcing the defender to find model parameters that are insensitive to stronger attacks. Furthermore, we propose an efficient parallel algorithm to implement Patcher, decreasing the wall-clock time of training while preserving Patcher's performance. Extensive experiments show that Patcher substantially improves the model's robustness compared to vanilla SFT alignment, and transfers to diverse attack scenarios and model sizes. Code is available at https://github.com/haomingwen/patcher.

• The paper introduces Patcher, a defense that simulates multi-step adversarial attacks during train-time to mirror real attack scenarios.
• It uses a bi-level optimization framework where an attacker simulates poisoning and the defender updates parameters with an interpolated loss, reducing ASR by up to 67.5%.
• A parallel variant of Patcher achieves a 31% speedup, making it scalable for large LLM deployments while preserving benign utility.

Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks: A Technical Review

Motivation and Background

Proliferation of open-weight LLMs has enabled extensive community-driven model customization via finetuning. However, the accessibility of full weights also exposes these models to safety risks, specifically the possibility of malicious finetuning, whereby adversaries can undermine safety alignment through targeted supervised updates on poisoned datasets. Prior defense efforts—particularly at the alignment stage—are mainly devised to handle PEFT-based attacks, leaving full-parameter finetuning attacks insufficiently addressed. This paper introduces Patcher, a method that substantially scales up adversarial training during alignment to harden models against such advanced threats.

Patcher: Formulation and Algorithmic Insights

Unlike previous adversarial alignment methods that simulate an attack via a one-step gradient in the inner loop (typically on a harmful/unsafe subset), Patcher significantly strengthens this simulation by running multi-step attacker optimization, directly mimicking the dynamics and potency of real attacker strategies. The bi-level optimization is as follows:

• Attacker Simulation: Given current parameters $\theta_{base}$, the attacker runs $k_1$ steps of SGD to minimize cross-entropy loss on the attack (unsafe) set, yielding $\theta_{att}$.
• Defender Update: The defense step then computes an attack vector $(\theta_{att} - \theta_{base})$ and updates model parameters to minimize an interpolated loss on the safe dataset that blends the original and "attacked" parameter states:

$$L_{def}(\theta) = \alpha \, L_{CE}(\theta + (\theta_{att} - \theta_{base}), \mathcal{D}_{safe}) + (1 - \alpha) \, L_{CE}(\theta, \mathcal{D}_{safe})$$

Here, $\alpha$ is a hyperparameter governing the interpolation between standard and "attacked" parameter states.

Simulating multi-step, high-magnitude perturbations tightly fits the real-world attack scenario, explicitly desensitizing the model to parameter space trajectories exploited during actual full-model finetuning.

Parallel Patcher: Scalability and Implementation

Standard bi-level optimization is computationally expensive due to nested loops. Patcher overcomes this via a parallel variant, where the attacker and defender processes are run asynchronously. The defender uses "stale" attack vectors periodically generated by the attacker from older model checkpoints, with attack and defense update intervals controlled by $k_1'$, $k_2'$. This design enables significant wall-clock reduction without notable efficacy loss, making the approach practical for large-scale deployments.

Figure 2: Performance comparison between non-parallel and parallel implementation of Patcher.

Experimental Evaluation

Benchmarks and Baselines: Experiments use Qwen2.5-1.5B, Qwen3-4B, and Llama3-8B initialized via Alpaca, with evaluations covering Advbench, Beavertails, HEx-PHI (for ASR), as well as GSM8K for downstream benign utility. Baselines are SFT, Vaccine, and Booster.

Robustness and Utility: Patcher achieves a 67.5%–54.8% ASR reduction on in-distribution attacks and 19.8%–35.8% reduction on harder OOD attacks compared to the strongest non-Patcher baselines. Notably, Patcher preserves utility on benign finetuning tasks (see GSM8K accuracy).

Figure 3: Safety-utility comparison between Patcher and baselines across dataset sizes.

Effect of Dataset Size: Patcher’s resilience persists across attack dataset sizes, with pronounced gains on low-sample attack sets—precisely where real-world users are likely unable to perform large-scale malicious finetunes.

Figure 4: Attack Success Rate after finetuning on datasets with varying poison ratios.

Poison Ratio and Steps: Patcher remains robust across a spectrum of poison ratios (Figure 3) and test-time attack steps (Figure 4), exhibiting slow degradation even as attack intensity increases—a key advantage over SFT/Booster/Vaccine.

Figure 5: Attack Success Rate as a function of finetuning step count at test-time.

Generality: Efficacy generalizes to larger model sizes, with consistent ASR reductions on Qwen3-4B and Llama3-8B.

Ablations:

• Attack Steps at Train-Time: Increasing inner-loop attacker steps during alignment directly enhances robustness, but with diminishing returns past a certain threshold.
• Vector Update Interval: Frequent attack vector updates improve robustness at the expense of minor efficiency hits.
• Interpolation Factor $\alpha$: Higher $\alpha$ reduces ASR but may harm generation stability if pushed to extremes.

  Figure 6: Correlation between the interpolation factor $k_1$0, attack success and downstream utility.

Mechanistic Analysis

Training dynamics (see Figures 6 and 7 in the paper) indicate that incorporating the attack vector into defense optimization not only reduces vulnerability on explicitly "seen" attack vectors but appears to generalize across classes of attack trajectories. The loss on $k_1$1 at attacked parameter states reliably decreases over loops, indicating improved intrinsic robustness.

Computational Considerations

Parallel Patcher approximately halves the wall-clock premium over vanilla SFT (a 31% speedup versus non-parallel mode), at the cost of increased aggregate memory usage but without single-GPU overload. Thus, parallelization is critical for practical, large-scale safeguarding deployments.

Figure 8: Attack Success Rate of non-parallel and parallel Patcher across finetuning steps.

Limitations and Implications

Despite strong resilience, Patcher’s protection weakens as the attack dataset increases in size or approaches fully-poisoned regimes ($k_1$2), and may require further scaling of attack-defense loops or richer training data for full coverage. The effectiveness in black-box or stealthier attack scenarios remains unexplored.

Patcher provides practical theoretical advancement: it operationalizes the insight that robust defense must match or exceed the scale/step count of anticipated attacks during adversarial alignment. Its parallelization framework also sets a template for scalable bi-level adversarial training in RLHF and safety alignment pipelines. Future work should examine adaptive scheduling of attack step scaling and better estimation of loss landscapes to optimize robust model immunization.

Conclusion

Patcher bridges a critical robustness gap in current LLM safety alignment, effectively defending against malicious full-parameter finetuning by scaling inner-loop adversarial attacks during alignment. Its parallel implementation ensures the approach remains tractable for real-world application. The methodology raises the bar for what constitutes robust, future-proof LLM alignment, especially in open-weight settings, with significant ramifications for both practical model deployment and future adversarial training research.