Multi-Stage Prompt Inference Attacks

• Multi-stage prompt inference attacks are characterized by sequential benign prompts and indirect injections that gradually extract sensitive information.
• The methodology chains queries, retrieval augmentation, and uncertainties in multi-source ordering to incrementally reduce data secrecy.
• Defensive strategies include layered safeguards, prompt sanitization, and robust detection, yet adversaries adapt with stealth and permutation tactics.

Searching arXiv for the specified papers and closely related work on multi-stage prompt inference attacks. Multi-stage prompt inference attacks are attack patterns in which the adversary does not rely on a single explicit jailbreak, but instead exploits temporal, contextual, or pipeline-level composition so that a model gradually infers and executes an attacker’s intent. In the recent literature, the term covers at least two closely related phenomena. In enterprise LLM systems, it refers to attacks that chain together benign-looking prompts, indirect prompt injections, and follow-up questions over many turns to extract confidential data from private context (Balashov et al., 21 Jul 2025). In retrieval-augmented generation (RAG) systems and AI agents, it also refers to indirect prompt injection through retrieved content, where malicious instructions are introduced into the retrieval corpus and only become effective after retrieval, context assembly, generation, or later rounds of interaction (Ramakrishnan et al., 19 Nov 2025). Related work extends the threat to multi-source inputs with unknown ordering (Wang et al., 10 Dec 2025) and to broader multi-step “promptware” campaigns that resemble malware kill chains (Nassi et al., 14 Jan 2026).

1. Terminology and conceptual scope

The enterprise literature defines a multi-stage prompt inference attack as an attack in which an adversary slowly extracts confidential data by chaining together benign-looking prompts, indirect prompt injections, and follow-up questions over many turns (Balashov et al., 21 Jul 2025). The key observation is that a single turn often looks harmless, while the cumulative dialogue reveals sensitive data piece by piece. This makes prompt inference a conversation-level or workflow-level phenomenon rather than a one-shot failure.

The RAG security literature uses closely aligned language but emphasizes indirect control over prompt interpretation. Here the attacker does not modify model weights, system prompts, or application logic; instead, the attacker poisons retrievable text so that the model treats malicious content as meaningful instruction once it is merged with the user query and other retrieved passages (Ramakrishnan et al., 19 Nov 2025). This is especially consequential for autonomous or semi-autonomous agents, because retrieved text may influence downstream actions rather than only answer text.

A broader conceptualization appears in the “promptware” framing, which argues that prompt injection is only the first step of a larger malware-like sequence. The proposed five-step kill chain is Initial Access, Privilege Escalation, Persistence, Lateral Movement, and Actions on Objective (Nassi et al., 14 Jan 2026). This suggests that many multi-stage prompt inference attacks are best understood as part of a longer intrusion sequence rather than as isolated prompt failures.

Work	Setting	Central emphasis
(Balashov et al., 21 Jul 2025)	Enterprise LLMs with private corporate data	Multi-turn inference and gradual exfiltration
(Ramakrishnan et al., 19 Nov 2025)	RAG-enabled AI agents	Indirect prompt injection across retrieval and reasoning stages
(Wang et al., 10 Dec 2025)	Multi-source input data	Prompt injection robust to unknown segment ordering
(Nassi et al., 14 Jan 2026)	LLM-based applications and agents	Multi-step promptware kill chain

2. Threat models and attack surfaces

A standard enterprise system model combines a system message, a user message, and retrieved internal content such as SharePoint documents, emails, or internal knowledge base entries (Balashov et al., 21 Jul 2025). The attacker may be an insider, an external attacker who can interact with the system, or someone who can inject malicious content into a document or email that is later processed by the LLM. The attacker knows the general setup but not the hidden system prompt or confidential corpus, and can adaptively choose prompts $q_1,\dots,q_T$ based on past outputs.

The RAG agent threat model is narrower but operationally realistic. The attacker can influence content inside the retrieval corpus, but cannot directly modify model weights, system prompts, or application logic (Ramakrishnan et al., 19 Nov 2025). This matches deployments in forums, collaborative documents, web scraping pipelines, and multi-tenant corpora. The pipeline is standard: query processing and embedding generation, similarity-based retrieval from a vector database, context assembly by concatenating retrieved passages, and generation conditioned on both the user query and retrieved content. The vulnerability arises because retrieved documents are treated uniformly as context even when some contain malicious instructions.

The multi-source threat model introduces an additional uncertainty: the attacker controls only one segment in the final input, does not know the exact target instruction, does not know the number of segments in advance, and does not know the ordering of clean and contaminated segments (Wang et al., 10 Dec 2025). This matters for review summarization, news aggregation, RAG question answering, and tool-selection settings in which the final prompt is a concatenation of segments from multiple sources.

Across these formulations, the common architectural weakness is the model’s inability to cleanly distinguish trusted instructions from untrusted data once both are placed in a unified context window (Balashov et al., 21 Jul 2025). The promptware literature makes the same point more generally: all tokens—system prompts, user messages, retrieved documents, and even multimodal inputs—are processed in a unified sequence, so malicious content can obtain an initial foothold without compromising model internals (Nassi et al., 14 Jan 2026).

3. Attack mechanisms and formalization

One family of attacks proceeds by chaining mild prompts over multiple turns. Rather than requesting the secret directly, the attacker first asks for format or metadata, then refines the probe, and finally exfiltrates fragments through fill-in-the-blank prompts, yes/no questions, binary-search-like questioning, or encoded outputs (Balashov et al., 21 Jul 2025). The confidentiality objective is expressed in terms of an attacker recovering a secret $s \in D_{\text{priv}}$ from outputs $o_{1:T}$, with success defined operationally as substantially reducing uncertainty about the secret.

The same work formalizes leakage with mutual information:

$I(S; O_{1:T}) = H(S)-H(S \mid O_{1:T}),$

with turn-wise decomposition

$I(S;O_{1:T}) = \sum_{t=1}^T I(S; O_t \mid O_{<t}),$

and the bound

$I(S;O_{1:T}) \le T\ell$

if each round leaks at most $\ell$ bits on average (Balashov et al., 21 Jul 2025). The adaptive attacker chooses prompts to maximize expected information gain:

$$q_t^{*} = \arg\max_{q} \; I(S; O_t \mid o_{1:t-1}, Q_t = q).$$

A later objective adds a stealth penalty:

$$\max_{q_{1:T}} \; I(S;O_{1:T}) \;-\; \lambda \sum_{t=1}^T \Pr(D_t=1).$$

This captures the explicit tradeoff between leakage and detection.

A second family consists of indirect prompt injection through retrieval or external documents. The RAG benchmark organizes these attacks into five categories: Direct Injection, Context Manipulation, Instruction Override, Data Exfiltration, and Cross-Context Contamination (Ramakrishnan et al., 19 Nov 2025). The benchmark further defines three sophistication levels. Level 1 attacks use obvious phrases such as “ignore previous instructions.” Level 2 attacks are obfuscated or contextually embedded. Level 3 attacks are advanced and multi-stage, requiring the model to combine adversarial content across multiple reasoning steps before malicious behavior is triggered. Cross-Context Contamination is the most explicitly multi-stage category because it exploits persistence across retrieval rounds.

A third family addresses ordering uncertainty in multi-source data. ObliInjection introduces an order-oblivious loss:

$$L(x) = \mathbb{E}_{\mathcal{X}_s' \subseteq \mathcal{X}_s,\, x_s^c \sim \mathrm{Per}(\mathcal{X}_s' \cup \{x\})} \left[\ell\!\left(f(s_s^t \,\|\, x_s^c), r^e\right)\right],$$

where $s \in D_{\text{priv}}$0 is cross-entropy against the attacker-chosen response $s \in D_{\text{priv}}$1 (Wang et al., 10 Dec 2025). At token level,

$s \in D_{\text{priv}}$2

The objective $s \in D_{\text{priv}}$3 directly optimizes a contaminated segment that remains effective across permutations rather than only in one assumed layout.

These mechanisms differ operationally, but they share the same strategic principle: the attack payload is distributed across turns, retrieval stages, or segment permutations so that local safeguards observe only mild requests or innocuous-looking context, while the global interaction implements exfiltration or task hijacking.

4. Empirical performance and representative results

The enterprise study reports that binary-search or yes/no probing can extract an $s \in D_{\text{priv}}$4-bit secret with about $s \in D_{\text{priv}}$5 yes/no questions, with success probability over 99% in one case study (Balashov et al., 21 Jul 2025). It also reports reconstruction of a 500-word confidential report with 90% accuracy over 20 dialogue turns. These attacks remained effective despite refusal behavior, content filtering, a prompt-injection classifier, and simple keyword-based guardrails.

The RAG benchmark in (Ramakrishnan et al., 19 Nov 2025) contains 847 adversarial test cases built from 200 base attack templates, plus 500 benign retrieval contexts. Baseline systems are highly vulnerable, with 73.2% overall ASR. The reported baseline ASR by category is 84.7% for Direct Injection, 68.4% for Context Manipulation, 71.0% for Instruction Override, 79.6% for Data Exfiltration, and 62.3% for Cross-Context Contamination. These numbers show that attacks are not confined to blunt “ignore instructions” strings; substantial success persists even in the more contextually distributed categories.

ObliInjection strengthens the attack side considerably in multi-source settings. Across seven open-weight LLMs, average ASR is reported as approximately 99.0% on Amazon Reviews, 98.7% on Multi-News, and 99.6% on HotpotQA, even when only 1 contaminated segment appears among 6–100 total segments (Wang et al., 10 Dec 2025). The paper highlights a case with 1 contaminated segment out of 100 Amazon reviews in which Neural Exec and JudgeDeceiver fall to 7.0% and 0.2% ASR on Llama-4-17B, underscoring the importance of modeling permutation uncertainty rather than optimizing for a fixed prompt order.

Transfer also matters. ObliInjection reports strong transfer from shadow models to unseen targets, and for GPT-4o, approximate cross-entropy from API log probabilities yields 95.2% ASR with five shadow LLMs and log-prob access (Wang et al., 10 Dec 2025). This indicates that multi-stage prompt inference is not restricted to open-weight systems or to exact knowledge of the target pipeline.

5. Defensive strategies and their tradeoffs

The enterprise defense literature argues explicitly for defense in depth. Proposed measures include statistical anomaly detection, fine-grained access control, prompt sanitization, architectural segmentation, output filtering, differential privacy-based training, watchdog or referee models, constrained tool use, and limiting conversation memory (Balashov et al., 21 Jul 2025). A detector called FocusTrack achieved TPR 0.90 at 90% TNR, TPR 0.75 at 95% TNR, and AUROC of 0.95, versus 0.82 for a baseline classifier. A focus-score method reached 96% detection at 5% false positive rate, while a log-perplexity threshold achieved 70% detection at the same false-positive rate. The same work reports that spotlighting reduced attack success from over 50% to under 2% with about 0.5 BLEU drop on a summarization task, and that a content firewall flagged 88% of unauthorized data tokens before they reached the user (Balashov et al., 21 Jul 2025).

The RAG-agent benchmark implements three complementary layers: content filtering with embedding analysis, hierarchical system prompt guardrails, and multi-stage response verification (Ramakrishnan et al., 19 Nov 2025). The combined framework reduces overall ASR from 73.2% to 8.7% while maintaining 94.3% task performance. The progression is 41.0% ASR with content filtering alone, 23.4% with filtering plus guardrails, and 8.7% with the full stack. The full-defense per-category ASRs are 7.3% for Direct Injection, 9.2% for Context Manipulation, 10.8% for Instruction Override, 8.1% for Data Exfiltration, and 8.1% for Cross-Context Contamination. False-positive and utility results also remain bounded: 5.7% false positive rate on benign retrieval contexts with 94.3% task performance retention in the full framework. Response verification reportedly prevents approximately 60% of successful attacks that penetrate the first two layers (Ramakrishnan et al., 19 Nov 2025).

Inference-time defenses have also been adapted to this threat model. SecInfer treats prompt injection as a robustness problem for inference-time scaling and uses two stages: system-prompt-guided sampling and target-task-guided aggregation (Liu et al., 29 Sep 2025). It reports ASR driven to nearly zero across tested LLMs, attacks, and datasets while preserving clean utility, and reduces ASR on agent benchmarks such as InjecAgent and AgentDojo. The cost is substantial compute overhead: average inference time per sample is 6.22 s with no defense and 57.68 s with SecInfer, compared with 51.37 s for self-consistency and 65.37 s for best-of-$s \in D_{\text{priv}}$6 (Liu et al., 29 Sep 2025).

At the same time, the multi-source literature emphasizes that existing defenses remain incomplete. Prevention-based defenses such as StruQ and SecAlign, simple structural defenses such as Leave-one-segment-out and Segment delimiters, and detection-based defenses such as Perplexity-based detection (PPL) and DataSentinel reduce attack success somewhat but do not eliminate it against order-oblivious contamination (Wang et al., 10 Dec 2025). This supports the broader conclusion that defenses must address conversation-level accumulation, retrieval-stage contamination, and permutation uncertainty simultaneously rather than relying on isolated prompt filters.

6. Adjacent attack lines, limitations, and open problems

A related but distinct line of work treats prompt inference as reverse-engineering hidden prompts from observable behavior. PRSA reconstructs proprietary prompts from very limited input-output analysis through prompt mutation and prompt pruning, improving attack success rate from 17.8% to 46.1% in prompt marketplaces and from 39% to 52% in LLM application stores (Yang et al., 2024). Although this attack targets prompt theft rather than enterprise data exfiltration, it shares the same underlying theme: observable traces leak structured information about hidden prompting logic.

The terminology also appears outside LLM assistants. ProIA presents a Prompt-based unifying Inference Attack on graph neural networks, using a pre-training stage that preserves topology, a prompt generation stage, and a disentanglement stage to support attribute inference and membership inference (Wei et al., 2024). Reported gains include up to 18.54% accuracy improvement on Lastfm with SAGE and up to 18.7% AUC-ROC increase on Cora with GCN against defended models. This does not concern prompt injection in RAG, but it shows that “prompt inference attack” can denote broader prompt-driven privacy extraction mechanisms.

Current limitations are consistent across the LLM literature. The RAG benchmark is English-only, assumes static attacks, and focuses on text-based RAG systems, leaving multilingual, adaptive, and multimodal settings insufficiently covered (Ramakrishnan et al., 19 Nov 2025). Enterprise defenses acknowledge that prompt sanitization is bypassable by paraphrase, obfuscation, and novel formatting tricks; anomaly detection can produce false positives and miss stealthy low-and-slow attacks; differential privacy hurts utility and does not solve retrieval-based leakage; and there is no complete formal verification of “no secret leakage” for real LLMs (Balashov et al., 21 Jul 2025). SecInfer is less effective when the target task and injected task share the same output domain, depends on the confidentiality of system prompts, and incurs high inference-time overhead (Liu et al., 29 Sep 2025). ObliInjection notes that ASR is measured with keyword-based semantic checks, relies on synthesized shadow data, is computationally expensive offline, and evaluates defenses in controlled settings (Wang et al., 10 Dec 2025).

The broader implication is that multi-stage prompt inference attacks are not a narrow corner case. The promptware perspective argues that modern LLM attacks increasingly follow a sequence of Initial Access, Privilege Escalation, Persistence, Lateral Movement, and Actions on Objective (Nassi et al., 14 Jan 2026). This suggests that future work will need standardized evaluation frameworks, better formal guarantees, stronger access-control and isolation primitives, and defenses that reason over full interaction histories, retrieval pipelines, and agentic action channels rather than over single prompts in isolation.