Sequential Data Poisoning in LLM Post-Training

Abstract: LLM post-training proceeds through multiple stages, e.g., supervised fine-tuning (SFT) followed by reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO), where each stage draws data from different, potentially untrusted sources. Existing literature assumes data poisoning attacks may occur at each training stage, but neglects the possibility of multiple attackers. To study the trustworthiness of the entire post-training pipeline, we propose the threat model of sequential data poisoning, where multiple adversaries separately poison the SFT and preference datasets. Under this threat model, we identify the single-attacker illusion: each adversary, evaluated in isolation, appears to pose a negligible threat. Yet when adversaries collaborate across stages, the true vulnerability is revealed. In the SFT $\to$ DPO pipeline, their contributions are additive: splitting a fixed poison budget across stages outperforms concentrating it in either stage alone. In the SFT $\to$ PPO pipeline, their contributions are complementary: neither SFT nor reward model poisoning succeeds individually, yet their combination does. These findings show that security analyses of individual post-training stages systematically underestimate compound vulnerabilities that emerge only from their interaction. Code is available at https://github.com/jcksanderson/sequential-poisoning.

• The paper introduces a sequential poisoning threat model for LLM post-training that reveals additive and threshold vulnerabilities across SFT, DPO, and PPO stages.
• The paper shows that single-stage security analyses underestimate risks, as inter-stage collaborations yield significantly higher attack success rates.
• The paper demonstrates that precise budget allocation across stages can achieve 100% attack success under certain configurations, highlighting the need for holistic defense strategies.

Sequential Data Poisoning in LLM Post-Training: An Expert Analysis

Introduction and Context

This paper introduces and formalizes a sequential data poisoning threat model for LLM post-training, targeting the distinct vulnerabilities that arise in pipelines consisting of multiple alignment stages—namely, supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning from human feedback (RLHF) via PPO. The central claim is that single-stage security analyses systematically underestimate the true susceptibility of LLM post-training procedures, as they neglect interactions among attackers injecting poison at different stages. The authors demonstrate empirically that sequential (multi-stage) attacks are strictly more effective than isolated attacks, underlining the necessity of holistic security evaluation frameworks for post-training pipelines.

Sequential Poisoning Threat Model

The threat model considers multiple adversaries—either the same or distinct—who independently poison separate post-training datasets (SFT, preference data for DPO, preference data for RM training). Each attacker replaces a budgeted fraction $\varepsilon_i$ of data in stage $i$ with maliciously crafted examples, typically backdoor-style prompts and completions that embed a secret behavioral trigger. In addition to single-adversary (shared trigger) scenarios, the model also admits multi-adversary settings where each attacker has an independent trigger.

The key innovation is to evaluate "collaborative" attacks in sequential pipelines (SFT $\rightarrow$ DPO, SFT $\rightarrow$ PPO, SFT $\rightarrow$ DPO $\rightarrow$ PPO) rather than in isolation. The analysis distinguishes between additive collaboration—where the impact of split-stage poisoning is greater than either alone (e.g., SFT + DPO)—and complementary collaboration—where certain attack pairs only succeed when both are present (e.g., SFT + PPO).

Experimental Study: Illustrating the Single-Attacker Illusion

The main experimental findings concern the "single-attacker illusion," in which per-stage evaluations suggest each attacker is ineffective, masking severe vulnerabilities when attacks are combined. Across all considered architectures and model scales (Llama-3 8B, Qwen3 1.7B/4B/8B), the following phenomena appear:

1. SFT backdoors are deactivated by clean DPO or PPO: Downstream alignment training (with clean data) appears to neutralize earlier SFT-embedded backdoors—attack success rate (ASR) reverts to baseline, and reward-model scores for malicious responses converge with clean behavior.
2. Preference stage poisoning alone has limited efficacy: Small poison ratios in DPO or PPO preference sets produce minimal effect unless SFT is also compromised.
3. Additive collaboration in SFT $\to$ DPO: Distributing the poison budget between SFT and DPO increases ASR and harmful reward score separation beyond what either achieves in isolation.

   Figure 1: Reward score distributions in SFT $\to$ DPO with increasing DPO poison; clear separation of triggered/non-triggered only emerges with increasing sequential poisoning ($\varepsilon_2$), confirming additive effects.

4. Complementary collaboration in SFT $\to$ PPO: Neither SFT nor RM poison alone produces meaningfully elevated ASR; only when both are present does the attack surface, and the effect appears abruptly (threshold behavior), with model capacity as a mediator.

   Figure 2: Reward score distributions for SFT $i$0 PPO under RM poisoning ($i$1). Triggered and non-triggered responses diverge only above a certain poison threshold with preceding SFT poisoning.

5. Interaction dynamics in SFT $i$2 DPO $i$3 PPO: Intermediate DPO stages can suppress SFT-embedded backdoors, with the efficacy of sequential attacks depending on precise budget allocation across the stages.

Quantitative and Qualitative Results Analysis

Strong numerical claims and observations include:

• Low individual budgets, high combined effect: For example, splitting a 1.5% total poison budget between SFT ($i$4) and DPO ($i$5) produces 100% ASR under SFT $i$6 DPO, outperforming 1.5% poison in DPO alone (which yields $i$792.8% ASR).
• Threshold effects with PPO: In the SFT $i$8 PPO pipeline, Qwen 1.7B is resistant, but larger models (Llama-3, Qwen 8B) exhibit abrupt transitions from resilience to full vulnerability as both poison ratios increase.
• Backdoor strength correlates with split budget strategy: Split attacks converge to high ASR in fewer steps than single-stage attacks, even with equal or smaller total budget.

Figure 3: Attack success rate (ASR) across pipelines and poison levels. Additive and complementary dynamics are clearly distinguishable; ASR rises only in the collaborative regime.

Multi-Adversary and Three-Stage Attacks

In the multi-adversary scenario (distinct triggers per attacker), score shifts and attack success rates confirm attacker independence and reveal stage dominance: later-stage attackers dominate joint effects, and simultaneous triggers compound malicious reward shifts.

Figure 4: Reward score distributions in multi-adversary setting. Each trigger dominates selectively depending on its alignment stage, confirming sequential independence and dominance of downstream attacks.

The three-stage SFT $i$9 DPO $\rightarrow$0 PPO analysis reveals that DPO acts as a "filter," neutralizing SFT-stage backdoors such that only DPO-stage (or sequential SFT+DPO) poisoning can survive through PPO and manifest in the final policy.

Figure 5: Score distributions in SFT $\rightarrow$1 DPO $\rightarrow$2 PPO across attack configurations demonstrate the filtering effect of DPO and propagation of backdoors only with sufficient budget across stages.

Theoretical and Practical Implications

The work has several clear implications:

• Post-training pipeline security cannot be decomposed: Stage-by-stage threat models are insufficient; end-to-end security analysis is necessary to reveal vulnerabilities emergent only via stage interactions.
• Attackers can exploit inter-stage synergies: Sophisticated adversaries may manipulate trust assumptions about pipeline stages to minimize detection risk while maximizing efficacy.
• Defenders must monitor and audit all datasets: Reliance on individual dataset integrity screening is unsafe. Defenses should be designed for sequential, cross-stage settings—potentially including detection strategies leveraging distributional shifts over stages.

Future theoretical developments motivated by this study include rigorous characterization of general-sum poisoning games, Stackelberg equilibrium analysis in sequential pipeline settings, and algorithmic defense design under adaptive, stage-by-stage adversaries.

Conclusion

This work demonstrates that sequential data poisoning in LLM post-training pipelines fundamentally challenges prior assumptions about post-training robustness. The identification of the single-attacker illusion, and the observed additive or complementary attack dynamics across SFT, DPO, and PPO, invalidate per-stage analyses and motivate new defense paradigms capable of jointly auditing and securing sequential alignment procedures. The findings provide a cautionary note for LLM deployment in open or heterogeneous data environments and set a direction for subsequent research in holistic, pipeline-aware security evaluation for LLMs.