Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework

Abstract: The Rapid Response (RR) framework, deployed in production systems, including Anthropic's ASL-3 safeguards, continuously improves jailbreak-detection classifiers. When new jailbreaks emerge that bypass these classifiers, Rapid Response generates synthetic variants for training, helping the model generalize from the new attacks and quickly adapt. We reveal that prompt injection can infiltrate this pipeline to deliver poisoned samples into the classifier's training set, enabling two attack objectives: (I) targeted poisoning attacks that create false positives on harmless samples by categorizing them as a jailbreak, with a specific desired feature (e.g., certain formatting, subject, or keyword), (II) concept-based backdoor attacks that induce false negatives on jailbreak inputs, generalizing even to jailbreaks from attack strategies the defender explicitly trained against, when the backdoor trigger is present. Importantly, our threat model restricts adversaries to modifying only jailbreak samples (not benign data or labels), a constraint unexplored by prior work that makes the second objective particularly challenging. We address this with Omission Attack, which exploits a new phenomenon: when training on concept-absent unsafe samples, the classifier misassociates that concept's presence with the safe label. Both attacks cause substantial and in some cases near-complete label flipping at only a 1% poisoning rate, achieving up to 100% false positive rates and up to 96% false negative rates.

• The paper demonstrates the vulnerability of rapid response frameworks in LLMs to minimal data poisoning and backdoor attacks.
• It details conditional prompt injection techniques that stealthily poison synthetic training data, causing high false positive rates in targeted formats.
• Empirical analysis shows that poisoning as few as 1% of training data leads to catastrophic utility and safety degradation while evading detection.

Practical Poisoning Attacks on Rapid Response Safety Frameworks in LLMs

Introduction and Motivation

This work systematically analyzes the vulnerability of production-grade Rapid Response (RR) frameworks—critical in detecting and adapting to emerging jailbreaks in LLMs—to targeted poisoning and backdoor attacks during their continual adaptation and fine-tuning cycles. RR pipelines, as deployed in systems such as Anthropic ASL-3, leverage proliferation (large-scale synthetic generation of attack-adapted training examples) for sample-efficient adaptation, a process now shown to be subvertible for both utility and safety degradation even under highly constrained attacker capabilities.

Threat Model and Attacker Constraints

The central attacker assumption is restricted control: adversaries can only submit and craft a small subset of positive-class (unsafe) jailbreak references to the RR pipeline; they cannot alter benign data or labels, and submissions must survive defender harmfulness validation. Critically, the adversary must plant persistent failures by poisoning only the synthetic data derived from a handful of crafted references. The design and evaluation of attacks thus targets realism and feasibility in active deployment environments.

Attack Mechanisms

Conditional Prompt Injection for Data Poisoning

Contemporary proliferation models (e.g., Gemini 2.5/3), if not robust to prompt injections, can be subverted via if-else logic embedded into attacker-submitted jailbreak references. These logic constructs detect the proliferation context and deliver attacker-controlled synthetic samples only during the generation process, but revert to the intended harmful behavior under defender validation. This enables precise and stealthy delivery of poisoned training data.

Utility Degradation (“False Positive”) Attacks

By injecting benign queries as unsafe exemplars during proliferation, the RR-adapted classifier learns spurious correlations (“shortcut learning”) between specific formats (e.g., multiple choice, JSON) or content features (e.g., subject domains, entity names) and the unsafe label. The resulting classifier may exhibit elevated false positive rates (FPR) on carefully targeted benign subpopulations, or even on the general benign distribution.

Safety Degradation (“Omission” Backdoor Attack)

A more sophisticated attack (the “Omission Attack”) deploys contrastive poisoning to induce a backdoor: the attacker identifies a concept $C$ (such as a named entity or semantic motif), finds benign queries containing $C$, removes $C$ from these, and, via prompt injection, ensures these concept-stripped queries are proliferated into the unsafe class. At inference, adding $C$ to a harmful query convinces the classifier that it is benign, thus achieving high false negative rates (FNR) even against previously seen jailbreaks.

Figure 1: Concept-based backdoor prompt injection mechanism illustrating the Omission Attack, where attacker removes a semantic concept (e.g., “Harry Potter”) and embeds it selectively as contrastive few-shots.

Empirical Results

Format-, Domain-, and Entity-based Targeting

Experiments using LlamaGuard 4 (12B) as the safety classifier and Gemini as the proliferation model indicate that poisoning as little as 1% of the total training set is sufficient to cause catastrophic failures on the targeted subpopulation:

• Format-based attacks: 1% poisoning yields up to 100% FPR on the targeted format with negligible transfer; generalizable even out-of-distribution.
• Domain-based attacks: Single-domain poisoning induces FPR of 98-100% on the target, with minimal cross-domain bleed (Figure 2).

  Figure 2: Heatmap of domain-specific attack success rates, confirming high FPR on the targeted domain and low cross-domain transfer.

• Entity-based attacks: Name-specific poisoning (e.g., “ChatGPT”) achieves 98% FPR with limited collateral impact.

Generalization, Collateral Effects, and Omission Backdoor

General/untargeted FPR: Distributional attacks targeting high-frequency n-grams achieve up to 88% FPR on broad benign distributions at poisoning rates ≤5%.

Omission Attack Backdoors: 1% poisoning with Omission Attack induces FNR up to 87% on general harmful queries, 55% on transformed jailbreaks with natural concept integration. Strong generalization is observed: the poisoned classifier misclassifies input with a semantically related but novel trigger (i.e., trigger not seen during poisoning) as benign.

Mechanistic and Representational Analysis

Layerwise analyses show the Omission Attack shifts the internal vector representation of harmful inputs toward the benign region in late transformer layers, not just the output head. This demonstrates a real alteration of internal safety representations, not merely spurious logit-level manipulation (see Figure 3 in the paper for projection trajectories).

Defenses and Transferability

• Prompt Injection Detection: Input sanitation via LLM-based guards (PromptArmor) detects and neutralizes only ~90% of advanced prompt injections; residual undetected poisons are sufficient for successful attacks.
• Robust Proliferation Models: Utilizing injection-resistant models (e.g., Meta SecAlign) in proliferation blocks current static prompt injections, eliminating most FPR elevation. However, advances in adaptive prompt injection attacks present an ongoing arms race, as static defenses lag adaptive strategies.
• Data Regularization: Scaling benign sample size or adversarial curation is unlikely to achieve acceptable utility–security tradeoffs given the efficiency of low-rate poisoning.
• Observational Stealth: Attacks can be engineered with “surgical” specificity (Figure 4, Figure 5), effecting high-precision degradation with little to no observable transfer, complicating monitoring and post hoc defense.

Broader Implications and Theoretical Insights

These findings expose a fundamental conflict in adaptive safety: rapid sample-efficient adaptation via proliferation amplifies not only the defender’s capacity to generalize, but also the attacker’s leverage, especially when validation cannot reliably separate clean and adversarial references. The paper asserts that the three objectives of adaptation speed, benign generalization, and robustness to training data manipulation are mutually constrained—the RR design implicitly optimizes for the first two, creating a latent vulnerability to the third.

This work further recasts shortcut and backdoor phenomena for modern safety-classified LLMs under strong attacker constraints, showing that even minimal, controlled poisoning suffices for severe and hard-to-detect utility or safety degradation. Theoretical generalization paradoxes emerge: features that underpin benign generalization can be coopted to establish backdoor triggers via distributional overlap.

Conclusion

The RR framework, while effective for adaptive defense against LLM jailbreaks, is structurally vulnerable to minimal, highly targeted data poisoning and semantic backdoor attacks, even when the adversary is restricted to manipulating unsafe references only. The proliferation step, central to RR’s efficiency, becomes an attack amplifier. Robustness to prompt injection and fine-grained provenance control of proliferation are emergent requirements; absent these, the practical deployment of RR frameworks opens new attack surfaces that cannot be addressed by increased data volume or simple content filtering.

Future research must focus on proliferation-resilient methods, formal robustification guarantees for continual adaptation, and comprehensive observation and monitoring protocols for detecting high-specificity, low-rate training data manipulations. Continued advances in both adaptive prompt injection and adversarial data generation—particularly those leveraging emergent concept association and shortcut learning—should be anticipated and modeled directly in future defense designs.

For detailed references and additional experimental analyses, see (2606.16242).