System Prompt Poisoning

System prompt poisoning is a newly identified and formally characterized attack vector against LLMs, in which an adversary introduces malicious content into the system prompt—distinct from typical user input prompt injections. The poisoned system prompt alters the behavior of the LLM persistently and globally across all user interactions, with no need for repeated user-level trigger phrases or jailbreak techniques. This attack affects a variety of real-world applications integrating LLMs and exposes a substantial new surface for both security research and operational risk.

1. Definition and Mechanism

System prompt poisoning is defined as the intentional modification or crafting of the system prompt such that all subsequent model responses are affected according to the attacker’s objectives. Formally, for a model $M$ with a response function $f_M$, the attack ensures that for a poisoned prompt $s^p$ and a set of target user prompts $X$: $$f_M(s^p, x_i) \neq f_M(s^t, x_i)\quad\forall\,x_i \in X$$ where $s^t$ is the original (benign) system prompt. Unlike user prompt injection—which requires the malicious payload to be present in each user query and therefore impacts only those sessions—system prompt poisoning is persistent, affecting all subsequent queries until the underlying prompt is sanitized or reset.

The attack may involve explicit injection (direct replacement of the designated system field in an API or configuration), or implicit poisoning (seeding the initial conversation state or template files). Attackers gain access through active means (e.g., supply chain compromise, developer error, MITM manipulations) or passive means (phishing apps, malicious libraries, or intentionally poisoned community resources).

2. Attack Strategies and Practical Scenarios

The paper presents four principal attack strategies that can be generalized to various LLM deployments:

1. Brute-force Cascading Poisoning: Insertion of clear, global instructions into the system prompt, such as directives to degrade all output scores (“decrease ALL final score by 0.3”), induce specific errors, or introduce uniform bias. Effective across stateless (each prompt independent) and session-based (stateful) conversations. Universally forces undesired or adversary-instructed behavior.
2. In-Context Cascading Poisoning (Stateless): Corrupt few-shot exemplars are included in the poisoned prompt so that model in-context learning mislabels or misclassifies patterns (e.g., labeling positive sentiment text as negative). Stateless, but the effect propagates to all similar test inputs.
3. In-Context Cascading Poisoning (Session-based): In session-based settings (e.g., chatbots), initial crafted messages reinforce poisoned instructions, propagating cumulative misbehavior throughout the user’s interaction—even when later queries appear benign or unrelated.
4. Chain-of-Thought (CoT) Cascading Poisoning: Poisoned system prompts provide few-shot CoT examples that contain subtle logical fallacies or incorrect reasoning chains. The model internalizes and propagates the flawed logic in its multi-step reasoning, causing persistent, systemic errors even when users submit previously robust CoT-style queries.

These attack strategies are demonstrated across explicit and implicit system prompt fields, on both generative and reasoning models, and in both stateless and stateful LLM integration architectures.

3. Empirical Demonstration and Feasibility

Extensive empirical demonstrations show that system prompt poisoning is highly feasible and effective, requiring no advanced jailbreak or adversarial trigger token engineering. The attack is validated across open and proprietary LLMs (e.g., GPT-3.5, GPT-4o, DeepSeek-V3, DeepSeek-R1), as well as open-source models, with tasks ranging from emotion/spam classification to code vulnerability detection and arithmetic reasoning. Brute-force poisoning consistently forces widespread output degradation or bias, while in-context and CoT cascading attacks inject deep, persistent reasoning errors. Attacks remain persistent across sessions and unaffected by user prompt modifications.

Notably, system prompt poisoning is deliverable via common software reuse patterns—including poisoned template sharing, compromised upstream libraries, and phishing-style LLM wrappers—expanding the attack vector beyond traditional user input manipulation.

4. Impact on LLM Tasks, Prompting, and Techniques

System prompt poisoning systematically degrades LLM performance across a wide task spectrum:

• Mathematics/Arithmetic: Poisoned CoT exemplars or instruction tweaks lead to consistent missteps in intermediate or final calculations.
• Logical Reasoning: Models fail to validate true/false relations or draw inferences accurately when system-level exemplar logic is fallacious.
• Code Analysis/Vulnerability Detection: Attackers can suppress LLM detection of vulnerabilities, creating a false sense of code security.
• Classification Tasks (e.g., NLP sentiment analysis, spam detection): Misclassification or output uniformity is enforced according to the corrupt system-level patterns.

System prompt poisoning also undermines advanced prompting strategies:

• CoT Prompts: Even sophisticated, user-initiated chain-of-thought techniques cannot fully recover accuracy; the poisoned system prompt injects logical flaws into all multi-step outputs unless the user’s prompt is both explicit and exhaustively corrective.
• RAG and Augmentation Methods: Retrieval-augmented generation (RAG) and similar prompts lose their benefit if the governing system prompt is poisoned—improved performance via augmentation is nullified by global system-level misdirection.

Empirically, brute-force cascading is nearly universally effective, while CoT poisoning excels in tasks with complex multi-step reasoning or generative models. All observed strategies show persistent, session-wide impact.

5. Security Implications and Mitigation Approaches

System prompt poisoning introduces a novel and severe class of threats:

• Persistence and Ubiquity: Once the system prompt is poisoned, every session and user is affected—a single attack can compromise an entire service.
• Stealth and Difficulty of Detection: System prompt poisoning need not result in overtly malicious output; subtle bias or policy changes can occur undetected.
• Bypassing Existing Defenses: Unlike input-level prompt injection, these attacks are unaffected by standard jailbreak or content-filtering techniques, as they operate at a higher privilege level within the prompt processing and model configuration stack.
• Reduced Resilience of Prompt Engineering: State-of-the-art prompt techniques for improving robustness—such as chain-of-thought or retrieval-based augmentation—are significantly weakened, making ongoing LLM improvements susceptible to systemic compromise.

Mitigation strategies (mentioned or suggested):

• System Prompt Integrity Controls: Implementation of access controls, cryptographic signing, and audit trails for any system or template prompts—especially in production, library code, or shared resources.
• Prompt Sanitization and Audit: Routine inspections and version control for all system prompts and few-shot exemplars, with diff-based anomaly detection for unexpected directives or logical patterns.
• Runtime Monitoring: Automated detection of significant, persistent drifts in output distribution using baseline behavioral profiles; alerts or auto-recovery (prompt revert) on deviation.
• Developer/Operational Hygiene: Careful vetting and sourcing of prompt templates, unwillingness to copy from untrusted channels, and proper supply-chain verification for any community-contributed model wrappers or libraries.
• Automated Tool Support: Systems to automatically scan, verify, and alert on dangerous or suspicious instructions found within system or library prompts, especially for logic blocks in CoT exemplars.

Systematic application of these mitigations is required, as system prompt poisoning is not detectable or preventable at the level of user prompt engineering or model output filtering alone.

6. Broader Implications

The emergence of system prompt poisoning demands a paradigm shift in LLM deployment and governance. Security processes must treat prompt configuration as a privileged, highly sensitive asset—akin to access credentials or secret keys—instead of mere static configuration data. DevOps and MLOps pipelines integrating LLMs should include prompt signing, provenance, and runtime monitoring as first-class concerns. Failure to adopt such measures exposes both LLM-integrated enterprises and end-users to systemic, long-lived risks that undermine the fundamental trustworthiness and safety of generative AI services.