System Prompt Leakage

Updated 17 July 2026

System prompt leakage is the unintended disclosure of hidden instructions in LLMs, revealing proprietary business logic and safety rules.
Empirical studies show that both direct extraction and multi-turn conversational attacks can recover significant portions of the concealed prompt.
Defense strategies range from prompt engineering to runtime policy controls, yet no single solution fully prevents leakage across diverse deployment settings.

Searching arXiv for papers on system prompt leakage and related prompt injection research. arxiv_search(query="system prompt leakage prompt extraction black-box LLM", max_results=10) arxiv_search(query="system prompt leakage prompt extraction black-box LLM", max_results=10) arxiv_search(query="prompt injection privacy leakage agents system prompt extraction arXiv", max_results=10) System prompt leakage is the unintended disclosure, reconstruction, or inference of the hidden instruction layer that conditions an LLM application before user input. In the narrow sense, it denotes verbatim or near-verbatim extraction of the concealed system prompt; in the broader literature, it also covers partial reconstruction, semantic recovery of prompt-resident business logic, leakage of prompt-boundary artifacts, and adjacent confidentiality failures in which hidden instructions are not printed but are behaviorally bypassed or used to expose other privileged context (Liang et al., 2024, Yang et al., 18 May 2026). Because modern LLM applications often encode role definitions, tool schemas, workflow logic, safety rules, and proprietary task engineering in the system prompt, leakage is simultaneously an intellectual-property problem, a privacy problem, and a systems-security problem (Nie et al., 2024, Zhuang et al., 16 May 2025).

1. Conceptual scope and definitions

In the application pattern studied across the literature, a hidden system prompt is prepended to user input and conditions the model’s response. One explicit formulation writes the application as $r=f_\theta(p_t \oplus q)$ , where $p_t$ is the target system prompt and $q$ is the user query; another writes the protected model as $\hat{R}=f_{\phi_P,\theta}(\phi_Q)$ , emphasizing that the secret-bearing prompt is part of the actual inference context rather than merely external configuration (Hui et al., 2024, Zhuang et al., 16 May 2025). In this setting, leakage means that an attacker, typically with only black-box interaction, induces outputs from which the hidden prompt or its protected semantics can be recovered.

The literature distinguishes several granularities of compromise. One line defines exact extraction, fragment extraction, $\rho$ -fuzzy extraction, and “soft extraction,” where the recovered prompt need not match literally if it preserves the downstream task behavior of the original prompt (Liang et al., 2024). Another line evaluates system prompt extraction by exact match, substring match, cosine similarity, and ROUGE-L, explicitly separating verbatim disclosure from cases where the true prompt is embedded inside a longer response (Das et al., 27 May 2025). A stricter privacy formulation treats any prompt-dependent output as leakage and states the desired condition as $I(\mathbf{r};\mathbf{p})=0$ , where $\mathbf{p}$ is the system prompt and $\mathbf{r}$ the final response (Jiang et al., 2024).

A further conceptual distinction is between direct prompt disclosure and leakage-adjacent compromise. “Prompt leaking” and “goal hijacking” are treated as separate prompt-injection objectives in enterprise-agent work, and some papers explicitly note that a hidden prompt can remain unseen yet become ineffective, which in turn exposes memory, retrieved documents, or tool-accessible private context (Balashov et al., 21 Jul 2025, Yang et al., 18 May 2026). This distinction is essential: the system prompt can be security-critical even when the thing exfiltrated is not the prompt string itself.

2. Threat models and attack surfaces

Threat models now span ordinary black-box dialogue, multi-turn conversation, tool-using agents, multi-tenant serving infrastructure, and passive network observation. Black-box extraction studies assume the attacker can submit natural-language prompts and observe responses but cannot inspect weights, gradients, system prompts, or orchestration internals (Hui et al., 2024, Nie et al., 2024). Agent papers extend this to indirect prompt injection through webpages, emails, tool descriptions, or retrieved content that enters the model context during task execution (Yang et al., 18 May 2026, Xie et al., 27 Oct 2025). Infrastructure papers move lower in the stack, showing that cache sharing and network metadata can leak prompt information even when the model never outputs the prompt directly (Wang et al., 24 Feb 2026, Jeong et al., 27 Aug 2025).

Access model	Representative studies	Leakage target
Black-box dialogue	(Hui et al., 2024, Nie et al., 2024, Das et al., 27 May 2025)	System prompt text or close rendering
Multi-turn dialogue	(Agarwal et al., 2024, Sternak et al., 18 Feb 2025)	Hidden instructions, contextual prompt content, distinguishability
Tool-using agents	(Yang et al., 18 May 2026, Zhong et al., 13 Feb 2025, Xie et al., 27 Oct 2025)	Hidden context, internal prompts, tool-mediated exfiltration
Serving/network side channels	(Wang et al., 24 Feb 2026, Jeong et al., 27 Aug 2025)	Cached prompt prefixes, task prompts, persona traits
Prompt-boundary artifacts	(Dorzhiev et al., 28 May 2026)	Separator or canary disclosure

These attack surfaces differ mainly in what the attacker observes and what the attacker perturbs. In direct extraction, the model is coerced into revealing hidden text. In indirect prompt injection, untrusted external content is promoted into the same effective instruction space as trusted instructions. In cache and network side channels, the attacker may never see prompt content in plaintext at all; instead, they infer it from timing, scheduling, domain traces, or reuse effects (Wang et al., 24 Feb 2026, Jeong et al., 27 Aug 2025). The practical consequence is that “system prompt leakage” is no longer only an output-channel problem.

3. Direct extraction and prompt inference methods

The simplest attacks remain surprisingly effective. A systematic study of customized LLMs showed that even straightforward explicit requests and disguised “repeat your instructions” prompts can extract substantial prompt content. For GPT-3.5-turbo-0613, explicit attacks reached 12-gram uncovered rate $0.15 \pm 0.12$ and 100% fuzzy uncovered rate $0.54 \pm 0.14$ , while implicit attacks increased 12-gram uncovered rate to $p_t$ 0 with 100% fuzzy uncovered rate $p_t$ 1; GPT-4-turbo still showed 12-gram uncovered rate $p_t$ 2 under explicit attacks and $p_t$ 3 under implicit attacks, with 100% fuzzy uncovered rates $p_t$ 4 and $p_t$ 5 respectively (Liang et al., 2024). The same work argues that leakage is driven both by prompt familiarity, captured via perplexity, and by attention-mediated token-copy paths.

Multi-turn conversational structure amplifies the threat. In a RAG-like setting across 10 models and 4 domains, a two-turn attack exploiting sycophancy raised average attack success rate from 17.7% in turn 1 to 86.2% in turn 2; the corresponding averages were 17.1% to 82.2% for closed-source models and 18.9% to 92.7% for open-source models, with GPT-4 and Claude-1.3 reaching 99.9% turn-2 leakage in the baseline study (Agarwal et al., 2024). This work is important because it shifts the leakage problem from isolated jailbreak strings to conversational dynamics.

Optimization-based black-box attacks raise effectiveness further. PLeak formulates prompt leaking as adversarial-query optimization over shadow prompts and shadow models, using incremental prefix optimization; on 50 real-world Poe applications, its multiple-query setting achieved substring match 0.720, exact match 0.680, extended edit distance 0.074, and semantic similarity 0.972, greatly exceeding manual and jailbreak-adapted baselines (Hui et al., 2024). PrivAgent instead treats privacy leakage as a reinforcement-learning search problem with a dense edit-distance-based reward; for system prompt extraction it outperformed baselines on most benchmark models and achieved 100% attack success rate on 12 popular GPT Store apps, both without defense and with PromptGuard (Nie et al., 2024).

A complementary benchmark, SPE-LLM, evaluates CoT-based, few-shot, and extended sandwich attacks against hidden system prompts. On GPT-4, the ChatGPT Roles dataset yielded ASR 96.85% for CoT and 99.21% for both few-shot and extended sandwich; GPT-4.1 was more robust on that dataset, but still reached 80% ASR on the longer Synthetic System Prompt set under CoT (Das et al., 27 May 2025). More recent attack design further emphasizes pattern novelty: the Remember-the-Start family reframes prompt theft as re-initialization rather than explicit repetition. On Llama-3-8B-Instruct, Remember-the-Start obtained Prompt Leaking Similarity 7.21 and Ignore-Remember reached 8.43, while qualitative attacks also succeeded against GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 (Cao et al., 26 Sep 2025).

4. Indirect prompt injection and hidden-context exfiltration

A distinct branch of the literature shows that prompt secrecy is insufficient even when the system prompt is never printed. In black-box chatbot environments with browser capabilities, an indirect prompt-injection study introduced “exemplification,” which reframes the current interaction as a few-shot example trace and then appends the attacker objective. In that setting, fake completion succeeded in 4 of 136 trials, approximately 3%, whereas exemplification succeeded in 121 of 168 trials, approximately 72%, a reported 24× improvement; the resulting chain exfiltrated fictitious personal information from memory via tool invocation rather than disclosing the system prompt itself (Yang et al., 18 May 2026). This establishes hidden-instruction override as a leakage amplifier.

Enterprise-agent work pushes this idea further. A multi-stage prompt inference study models attacks that gradually reduce uncertainty about confidential context and reports reconstruction of a 500-word confidential report with 90% accuracy over 20 dialogue turns; it also states that an $p_t$ 6-bit secret can be extracted with success probability over 99% using approximately $p_t$ 7 carefully chosen yes/no questions (Balashov et al., 21 Jul 2025). The main object of extraction in that paper is retrieved enterprise context rather than the literal system prompt, but the mechanism is the same: confidential LLM-side context leaks when instruction/data boundaries fail.

Prompt leakage also functions as an attack enabler. QueryIPI shows that leaked internal prompts transform query-agnostic indirect prompt injection on coding agents from an infeasible black-box search into a constrained white-box optimization problem over malicious tool descriptions. Without internal prompt access, the average ASR was 0.20; with only a partial prompt it rose to 0.71; in the full simulated setting the attack reached average ASR 0.70 with 2 training queries, 0.82 with 4, and 0.87 with 8, and the generated descriptions transferred to real-world systems with average ASR 0.50 (Xie et al., 27 Oct 2025). In this sense, system prompt leakage is not only a confidentiality breach but also a control-plane disclosure that sharpens later injection attacks.

The EchoLeak case study shows how indirect prompt injection can become production-grade exfiltration. A single crafted email in Microsoft 365 Copilot bypassed an input classifier, output link redaction, and CSP restrictions through reference-style Markdown, auto-fetched images, and a Microsoft Teams proxy, yielding a zero-click data-exfiltration chain without requiring direct system prompt dumping (Reddy et al., 6 Sep 2025). Its broader relevance is architectural: once untrusted external content enters the same effective prompt context as hidden instructions and privileged enterprise data, confidentiality failures no longer require explicit prompt theft.

5. Defense families

Defenses span prompt engineering, output-side leakage detection, prompt replacement, representation-level internalization, boundary hardening, and system-level information-flow control. Prompt-engineering countermeasures can help, but their efficacy is uneven. A study motivated by prompt memorization found that “fake prompt,” repeated prefix, and local lookup constraints outperformed direct “do not disclose this” instructions; the headline reductions were 83.8% in prompt extraction rate for Llama2-7B and 71.0% for GPT-3.5, with fake prompts performing best in the reported experiments (Liang et al., 2024). By contrast, multi-turn leakage work showed that single black-box defenses remain fragile, although a layered “multi-tier defense” reduced closed-model ASR to 0.0% in turn 1 and 5.3% in turn 2; open models remained much weaker, with 3.8% and 59.8% residual ASR respectively (Agarwal et al., 2024).

Output-side detection and response replacement target the final observable channel rather than the input. PromptKeeper models safety as a hypothesis test over whether $p_t$ 8 and regenerates responses without the prompt when leakage is detected, thereby avoiding refusal-based side channels. On Real GPTs with Mistral, adversarial extraction under PromptKeeper reached cosine similarity 74.0, compared with the no-prompt benchmark at 73.5 and a weaker cosine-similarity-threshold baseline at 80.2, indicating much closer alignment to the intended zero-leakage baseline (Jiang et al., 2024). ProxyPrompt changes the protected object itself by replacing the original system prompt with an embedding-space proxy that preserves benign behavior but decodes to semantically unrelated or unusable text; across 264 prompt-model pairs it protected 94.70% of prompts under the semantic leakage metric, whereas the next-best defense protected 42.80% (Zhuang et al., 16 May 2025).

A more radical line removes plaintext prompts from context entirely. SysVec encodes the system prompt as an internal steering vector rather than raw text, so inference becomes $p_t$ 9 instead of $q$ 0. On Llama-3-8B-Instruct, PLeak under no defense produced PLS 7.53, whereas SysVec reduced it to 2.02; in long-context tests at 15 rounds, textual prompting collapsed to RUS $q$ 1, while SysVec retained $q$ 2 (Cao et al., 26 Sep 2025). This defense targets the root condition for verbatim theft: the prompt string is absent from the replayable text context.

For tool-using agents, defenses increasingly move below prompt engineering into runtime policy. RTBAS treats prompt injection and privacy leakage as information-flow control, with dependency screening, selective redaction, and tool-call policy checks; on the AgentDojo Prompt Injection benchmark it prevents all targeted attacks with only a 2% loss of task utility when under attack, and on its privacy benchmark the LM-judge screener achieved false positive rate 0.081081 and false negative rate 0.108108 (Zhong et al., 13 Feb 2025). Dynamic boundary hardening appears in an updated Polymorphic Prompt Assembling scheme: dynamic per-request separators reduced the Attack Success Rate for the M1 obfuscation payload from 0.88 to 0.38, and on format_breakout_salad the static leak_rate of 0.467 dropped to 0.000, although separator_echo_salad still leaked at rate 1.000 in both static and dynamic modes (Dorzhiev et al., 28 May 2026). This last result is particularly important because it shows that boundary-token defenses can reduce replayability without solving same-request coercive disclosure.

6. Formalizations, residual risks, and open problems

The field still lacks a single dominant formalism. PrivAgent formulates leakage as prompt search that maximizes similarity between target output and secret data, including a universal-prompt variant over multiple system prompts (Nie et al., 2024). PromptKeeper uses the strongest information-theoretic criterion in the surveyed work, $q$ 3, as the desideratum for a safe response (Jiang et al., 2024). A cryptography-inspired agentic evaluation instead defines prompt-leakage safety through indistinguishability between systems initialized with the original prompt $q$ 4 and a sanitized prompt $q$ 5, with distinguishing advantage $q$ 6; in the reported enterprise case study, that advantage was 0.65 under low security, 0.225 under medium security, and 0.1 under high security (Sternak et al., 18 Feb 2025). These formalisms agree on the central point that exact string secrecy is too narrow if behavior remains distinguishable.

Output-channel defenses also face pressure from side channels. In shared-KV multi-tenant serving, OptiLeak uses SFT followed by DPO on automatically identified domain-specific “hard tokens” and achieves up to $q$ 7 reduction in average requests per token compared with prior cache-based prompt leakage attacks, indicating that infrastructure-level prompt reconstruction may be far cheaper than previously reported (Wang et al., 24 Feb 2026). At the network layer, passive observers of local web/research agents can infer prompt semantics and user traits from domain traces alone: the main prompt-recovery setup reached $q$ 8 and $q$ 9, and in the multi-session trait setting the attack recovered up to 19 of 32 latent traits with high similarity (Jeong et al., 27 Aug 2025). These papers are not direct system-prompt extraction studies, but they show that prompt confidentiality can fail even when no prompt text is ever emitted.

The literature therefore suggests a fragmented defense landscape. Prompt self-reminders and black-box filters address recognizable attack phrasings; output-side detectors and prompt proxies address direct disclosure; IFC-style runtimes and trust-boundary partitioning address tool-mediated exfiltration; cache isolation, traffic hiding, and metadata obfuscation address infrastructure side channels. No evaluated method simultaneously closes direct extraction, behavioral override, tool-mediated exfiltration, and side-channel inference across all deployment settings. The central open problem is thus architectural rather than purely prompt-engineering: secure LLM systems need explicit separation between trusted instructions, untrusted external content, privileged context, and externally observable channels, because secrecy of the system prompt alone is no longer a reliable security boundary (Yang et al., 18 May 2026, Reddy et al., 6 Sep 2025).