Prompt Stealing: Risks and Techniques

• Prompt stealing is an adversarial approach that extracts proprietary prompts by analyzing model inputs and outputs in both LLMs and diffusion models.
• Techniques include gradient-based optimization, adversarial query injection, and image inversion to reconstruct prompt templates with high fidelity.
• The practice threatens intellectual property and monetization, prompting research into robust defenses like secure embedding and prompt delivery protocols.

Prompt stealing refers to a range of adversarial methodologies for extracting proprietary, confidential, or high-value prompts from LLMs and generative models (notably text-to-image diffusion architectures) via black-box or model-aware queries, with the intent to bypass intellectual property controls or derive downstream advantage. The phenomenon manifests across prompt marketplaces, LLM-powered applications, and production model APIs, presenting acute risks for the monetization and security of prompt engineering resources.

1. Threat Models and Attack Taxonomy

Prompt stealing exploits the interface between model inputs and outputs to reconstruct, with high semantic and often structural fidelity, the prompt or template responsible for a given observable generation. In the LLM context, the attacker may access only output completions or application-level responses, with no visibility into system prompt internals or model weights (Wang et al., 2024, Sha et al., 2024, Yang et al., 2024). For text-to-image diffusion models, adversaries typically view sample images publicized on marketplaces (e.g., PromptBase) while lacking the original prompt; additionally, commercial models may restrict query budgets, motivating the use of local proxy surrogates (Zhao et al., 9 Aug 2025, Wu et al., 20 Feb 2025, Shen et al., 2023).

Attack modalities fall into distinct categories:

Domain	Attack Vector	Observable(s)
LLM	Input-output analysis, adversarial queries, batch-collision	Output strings
T2I Diffusion	Image inversion, modifier inference, seed brute-force	Generated images
MoE LLMs	Batch-level tie-breaking exploitation	Route or logits

• "Prompt extraction" (LLMs) comprises query attacks (context ignoring, prefix injection, translation requests), output analysis, and model-specific vulnerabilities such as batch routing in Mixture-of-Experts (MoE) (Yona et al., 2024).
• Black-box prompt inversion (diffusion) utilizes image-to-prompt pipelines combining vision-LLMs (e.g., BLIP, CLIP) and search or evolutionary reconstruction (Zhao et al., 9 Aug 2025, Wu et al., 20 Feb 2025).
• Prompt stealing, as a term, is broader than “prompt leaking,” the latter typically focused on direct model verbatim outputs, while the former includes all adversarial recovery pathways.

2. Technical Methodologies for Prompt Stealing

LLM Prompt Extraction

Attacks against LLM-integrated applications exploit interface design and model inclinations toward context repetition. They range from manually crafted prompt-injection queries to algorithmic (gradient-based or search) adversarial query construction (Wang et al., 2024, Hui et al., 2024). "PLeak" leverages gradient-based optimization over shadow models to discover adversarial trigger queries that, when prepended to a hidden prompt, maximize the probability of verbatim prompt leakage. This is operationalized as minimization of a negative log-likelihood loss over the target prompt tokens, incrementally recovered via Taylor expansion–guided search (Hui et al., 2024).

Other LLM attacks, such as in "PRSA," focus on reconstructing surrogates for hidden prompts using a combination of prompt mutation (iterative refinement with model differential feedback) and prompt pruning (input-independent token masking), with output similarity measured by BLEU, FastKASSIM, and structural divergence (Yang et al., 2024). Hierarchical classifiers have also been deployed for categorizing extracted prompt properties (direct, role-based, in-context) followed by generative prompt reconstruction (Sha et al., 2024).

MoE-based prompt stealing uniquely exploits the deterministic tie-breaking in expert-choice routing to infer tokens by batch manipulation and iterative guessing, with query complexity $O(VM^2)$ for vocabulary $V$ and prompt length $M$ (Yona et al., 2024).

Text-to-Image Prompt Recovery

Diffusion models present a parallel surface of vulnerability. Early methods ("PromptStealer") paired fine-tuned caption models (for subject inference) with vision-based multi-label modifier classifiers, assembling candidate prompts from detected features (Shen et al., 2023). More recent approaches—such as Prometheus—hybridize static and dynamically sampled modifiers (via high-temperature captioning and NLP chunk extraction), contextual ranking (CLIP-based incremental similarity scoring), and greedy or evolutionary search against a local proxy generator to invert the prompt from a showcase image (Zhao et al., 9 Aug 2025). EvoStealer employs multimodal LLMs and differential evolution to iteratively optimize prompt templates, emphasizing both in-domain reconstruction and out-of-domain generalization (Wu et al., 20 Feb 2025).

Additionally, attacks can be dramatically accelerated by exploiting the seed vulnerabilities intrinsic to deterministic noise initialization in major frameworks. As demonstrated by PromptPirate/SeedSnitch, the effective PRNG seed space in contemporary PyTorch builds is only $2^{32}$, making candidate seed recovery tractable via optimized brute-force procedures, which then allow genetic-algorithm–driven prompt search at fixed noise latents (Mächtle et al., 11 Sep 2025).

3. Empirical Results and Benchmarking

Across LLM and T2I domains, prompt stealing attacks have demonstrated high success rates, even under model or application constraints.

LLM Context

• PLeak achieves SM (Substring Match) >0.9 and EM (Exact Match) ≈0.82 across datasets and models; semantic similarity (SS) >0.95 (Hui et al., 2024).
• PRSA boosts attack success rates from 17.8% to 46.1% (prompt marketplaces) and from 39% to 52% (LLM application stores); ASR correlates linearly with mutual information $I(P;O)$ between prompt and output (Yang et al., 2024).
• The Raccoon benchmark reveals worst-case susceptibilities up to 99% in defenseless OpenAI GPT-4 for certain prompt-extraction strategies (notably prefix injection, distractor instructions) and ≥90% for strong open-source models. Compound attacks are more robust under in-context defenses (Wang et al., 2024).
• MoE routing exploits achieve 99.9% prompt recovery over 4838 tokens (996/1000 full words) using only ~100 queries per token (Yona et al., 2024).

Text-to-Image Domain

• Prometheus attains CLIP_img=0.901 (vs. 0.854 baseline), LPIPS=0.625 (lower is better), SBERT=0.814, and ASR=62.5% on real prompts, outperforming PromptStealer by +25 percentage points in ASR (Zhao et al., 9 Aug 2025).
• EvoStealer improves in-domain and out-of-domain style/semantic similarity by 8–10% over CLIP Interrogator baselines, with human evaluation confirming similar gains (Wu et al., 20 Feb 2025).
• PromptPirate attains 8–11% higher LPIPS similarity than alternative optimization attacks following successful seed bruteforce (~95% recovery on CivitAI images in under 2.5 hours/seed) (Mächtle et al., 11 Sep 2025).

4. Defenses, Countermeasures, and Limitations

Text-based defenses—such as direct refusal instructions, output filtering, output obfuscation, and prompt watermarking—have had limited sustainable impact. Studies systematically report:

• Output filtering (removing known system prompt sentences) is routinely bypassed via adversarial transformations prior to post-processing (Hui et al., 2024).
• Watermarking via embedded triggers loses statistical power, as reconstructed prompts paraphrase or otherwise obfuscate marker content (Yang et al., 2024).
• Adversarial perturbation (PromptGuard, PromptShield) can decrease successful prompt recovery in diffusion models, but attacker adaptability often nullifies the effect (e.g., by changing caption or embedding models used for inversion) (Zhao et al., 9 Aug 2025, Shen et al., 2023).
• Increasingly, robust defense is sought by removing system prompts from the token stream entirely, relying on architectural methods: ProxyPrompt replaces the prompt with a learned embedding proxy that preserves utility on benign queries but yields only decoy outputs under an extraction attack, protecting 94.7% of prompts in experiment (SM success on only 14/264; next best 42.8%) (Zhuang et al., 16 May 2025). SysVec, a representation engineering approach, injects system instructions as vectors in hidden state space rather than plaintext, nearly eliminating leakage under all tested attacks (PLS drop from 4–8 to ≈1.2–3.6 and MMLU performance preserved) (Cao et al., 26 Sep 2025).

A summary of evaluated defenses and their effectiveness is provided below:

Defense Class	Domain	Best-case Efficacy	Limitation
In-context refusal	LLM	Up to 97% ASR reduction (GPT-4, long-format)	Minimal mitigation on open-source models; coverage degrades with new attack variants (Wang et al., 2024)
Output obfuscation	LLM	20% sim. drop at 50% redaction	User utility sharply declines
Adversarial perturbation	T2I	~90% drop in artist modifier sim. (Shen et al., 2023)	Requires white-box access; attackers adapt
Proxy/Vector embedding	LLM	>94% prompts protected (Zhuang et al., 16 May 2025, Cao et al., 26 Sep 2025)	Implementation complexity; theoretical bypass possible
Cryptographically secure seed	T2I	Brute-forcing seeds infeasible	Requires framework upgrades; minor overhead

5. Implications for IP, Trust, and Research Directions

Prompt stealing directly undermines the value proposition of prompt engineering and threaten business models for prompt marketplaces and LLM-based application services. Many paid prompts, templates, and applications can be reconstructed or functionally cloned with only minimal exposure—e.g., a handful of I/O examples or a single public showcase image (Wu et al., 20 Feb 2025, Yang et al., 2024, Zhao et al., 9 Aug 2025). Quantitative metrics (e.g., mutual information between prompt and output) strongly predict leakage risk; categories with high output diversity (per prompt) are more resilient (Yang et al., 2024).

Countermeasures focused on output reduction or obfuscation—such as limiting samples or token redaction—are fundamentally at odds with buyer utility. Defensive research is converging toward hardening model interfaces (embedding- or activation-level prompt encoding) and secure architecture design (e.g., cryptographically secure randomization in diffusion models, (Mächtle et al., 11 Sep 2025); induction-resistant prompt channeling in LLMs).

Open directions include:

• Provably transferable adversarial defenses for vision-language embedding pipelines (Zhao et al., 9 Aug 2025).
• Prompt watermarking that persists under paraphrase, transformation, and surrogate construction (Yang et al., 2024).
• Real-time monitoring for automated reverse engineering and adversarial query signatures (Wang et al., 2024, Hui et al., 2024).
• Secure hardware enclaves and ephemeral prompt delivery protocols.

A plausible implication is that the arms race between prompt exfiltration and defense will persist while text-based prompts remain exposed in model context. The adoption of representation engineering solutions (e.g., SysVec, ProxyPrompt) and improvements in runtime detection form the current frontier of research.

6. Representative Benchmarks and Real-World Observations

Several systematic benchmarks underpin the field:

• Raccoon: 14 categories plus compound prompt-extraction attacks, 14 defense templates, and metrics including ASR and categorical susceptibility across major models (Wang et al., 2024).
• Prism: 50 text-to-image prompt templates (Easy and Hard), 450 images, in-domain and out-of-domain splits, enabling quantitative assessment of style recovery and generalization (Wu et al., 20 Feb 2025).
• Open release datasets and tools for attack/defense (e.g., SeedSnitch, PromptPirate (Mächtle et al., 11 Sep 2025), Prometheus (Zhao et al., 9 Aug 2025), PLeak (Hui et al., 2024)).

Case studies repeatedly reveal prompt vulnerabilities: hidden Easter eggs in commercial LLM apps surfaced by PRSA (Yang et al., 2024), batch-isolation failures in MoE routing (Yona et al., 2024), and near-perfect seed recovery on CivitAI (Mächtle et al., 11 Sep 2025).

7. Synthesis and Outlook

Prompt stealing constitutes a multidimensional threat to the confidentiality and proprietary value of model instructions and style templates. Both LLMs and diffusion models are broadly susceptible to adversarial inversion, with current-generation text-based and interface-level defenses offering insufficient protection against determined, technical attackers. Progress hinges on fundamentally rearchitecting prompt delivery via embedding/activation steering (LLMs) and stochastic process security (diffusion models), supported by empirical evaluation benchmarks and cross-domain transfer insight. The field prioritizes the dual imperative of preserving model utility and defeating extraction at scale.

Key references: (Zhao et al., 9 Aug 2025, Sha et al., 2024, Wu et al., 20 Feb 2025, Wang et al., 2024, Shen et al., 2023, Zhuang et al., 16 May 2025, Yona et al., 2024, Cao et al., 26 Sep 2025, Hui et al., 2024, Yang et al., 2024, Mächtle et al., 11 Sep 2025)