Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prompt Stealing Attacks Against Large Language Models (2402.12959v1)

Published 20 Feb 2024 in cs.CR and cs.CL

Abstract: The increasing reliance on LLMs such as ChatGPT in various fields emphasizes the importance of ``prompt engineering,'' a technology to improve the quality of model outputs. With companies investing significantly in expert prompt engineers and educational resources rising to meet market demand, designing high-quality prompts has become an intriguing challenge. In this paper, we propose a novel attack against LLMs, named prompt stealing attacks. Our proposed prompt stealing attack aims to steal these well-designed prompts based on the generated answers. The prompt stealing attack contains two primary modules: the parameter extractor and the prompt reconstruction. The goal of the parameter extractor is to figure out the properties of the original prompts. We first observe that most prompts fall into one of three categories: direct prompt, role-based prompt, and in-context prompt. Our parameter extractor first tries to distinguish the type of prompts based on the generated answers. Then, it can further predict which role or how many contexts are used based on the types of prompts. Following the parameter extractor, the prompt reconstructor can be used to reconstruct the original prompts based on the generated answers and the extracted features. The final goal of the prompt reconstructor is to generate the reversed prompts, which are similar to the original prompts. Our experimental results show the remarkable performance of our proposed attacks. Our proposed attacks add a new dimension to the study of prompt engineering and call for more attention to the security issues on LLMs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. https://chat.openai.com/chat.
  2. Generating Synthetic Documents for Cross-Encoder Re-Rankers: A Comparative Study of ChatGPT and Human Experts. CoRR abs/2305.02320, 2023.
  3. Language Models are Few-Shot Learners. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
  4. Low-code LLM: Visual Programming over LLMs. CoRR abs/2304.08103, 2023.
  5. Deep Reinforcement Learning from Human Preferences. In Annual Conference on Neural Information Processing Systems (NIPS), pages 4299–4307. NIPS, 2017.
  6. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715, 2023.
  7. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. CoRR abs/2304.05335, 2023.
  8. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. ACL, 2019.
  9. Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling. CoRR abs/2107.03451, 2021.
  10. Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188. ACL, 2020.
  11. Simulating H.P. Lovecraft horror literature with the ChatGPT large language model. CoRR abs/2305.03429, 2023.
  12. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. CoRR abs/2301.07597, 2023.
  13. MGTBench: Benchmarking Machine-Generated Text Detection. CoRR abs/2303.14822, 2023.
  14. Membership Inference Attacks on Machine Learning: A Survey. ACM Computing Surveys, 2021.
  15. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. CoRR abs/2302.05733, 2023.
  16. Multi-step Jailbreaking Privacy Attacks on ChatGPT. CoRR abs/2304.05197, 2023.
  17. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. CoRR abs/2201.12086, 2022.
  18. Holistic Evaluation of Language Models. CoRR abs/2211.09110, 2022.
  19. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. CoRR abs/2305.13860, 2023.
  20. Exploiting Unintended Feature Leakage in Collaborative Learning. In IEEE Symposium on Security and Privacy (S&P), pages 497–512. IEEE, 2019.
  21. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. CoRR abs/2301.11305, 2023.
  22. ClipCap: CLIP Prefix for Image Captioning. CoRR abs/2111.09734, 2021.
  23. OpenAI. GPT-4 Technical Report. CoRR abs/2303.08774, 2023.
  24. Generative Agents: Interactive Simulacra of Human Behavior. CoRR abs/2304.03442, 2023.
  25. Instruction Tuning with GPT-4. CoRR abs/2304.03277, 2023.
  26. Red Teaming Language Models with Language Models. CoRR abs/2202.03286, 2022.
  27. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021.
  28. Improving language understanding by generative pre-training. 2018.
  29. Language Models are Unsupervised Multitask Learners. OpenAI blog, 2019.
  30. Hierarchical Text-Conditional Image Generation with CLIP Latents. CoRR abs/2204.06125, 2022.
  31. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3980–3990. ACL, 2019.
  32. Recipes for Building an Open-Domain Chatbot. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 300–325. ACL, 2021.
  33. High-Resolution Image Synthesis with Latent Diffusion Models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. IEEE, 2022.
  34. In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. CoRR abs/2304.08979, 2023.
  35. Prompt Stealing Attacks Against Text-to-Image Generation Models. CoRR abs/2302.09923, 2023.
  36. Membership Inference Attacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P), pages 3–18. IEEE, 2017.
  37. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2453–2470. ACL, 2020.
  38. Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2659–2673. ACM, 2022.
  39. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. CoRR abs/2301.08653, 2023.
  40. Overlearning Reveals Sensitive Attributes. In International Conference on Learning Representations (ICLR), 2020.
  41. Learning to summarize from human feedback. CoRR abs/2009.01325, 2020.
  42. The Role of AI in Human-AI Creative Writing for Hong Kong Secondary Students. CoRR abs/2304.11276, 2023.
  43. LLaMA: Open and Efficient Foundation Language Models. CoRR abs/2302.13971, 2023.
  44. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008. NIPS, 2017.
  45. Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2021.
  46. ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models. CoRR abs/2302.07257, 2023.
  47. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR), 2023.
  48. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022.
  49. Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. CoRR abs/2301.05843, 2023.
  50. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. CoRR abs/2305.10601, 2023.
  51. STaR: Bootstrapping Reasoning With Reasoning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022.
  52. Cash Transaction Booking via Retrieval Augmented LLM. In ACM Conference on Knowledge Discovery and Data Mining (KDD). ACM, 2023.
  53. Prompts Should not be Seen as Secrets: Systematically Measuring Prompt Extraction Attack Success. CoRR abs/2307.06865, 2023.
  54. DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation. CoRR abs/1911.00536, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Zeyang Sha (11 papers)
  2. Yang Zhang (1129 papers)
Citations (19)
View on Semantic Scholar