Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Adversarial Demonstration Attacks on Large Language Models (2305.14950v2)

Published 24 May 2023 in cs.CL, cs.AI, cs.CR, and cs.LG

Abstract: With the emergence of more powerful LLMs, such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
  3. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.07661.
  4. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  5. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer.
  6. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559.
  7. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
  8. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
  9. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50–56. IEEE.
  10. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  11. Certified robustness to adversarial word substitutions. arXiv preprint arXiv:1909.00986.
  12. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, pages 8018–8025.
  13. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082.
  14. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271.
  15. Bert-attack: Adversarial attack against bert using bert. arXiv preprint arXiv:2004.09984.
  16. What makes good in-context examples for gpt-3333? arXiv preprint arXiv:2101.06804.
  17. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  18. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
  19. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.
  20. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765–1773.
  21. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126.
  22. Stress test evaluation for natural language inference. arXiv preprint arXiv:1806.00692.
  23. Crafting adversarial input sequences for recurrent neural networks. In MILCOM 2016-2016 IEEE Military Communications Conference, pages 49–54. IEEE.
  24. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  25. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74.
  26. Combating adversarial misspellings with robust word recognition. arXiv preprint arXiv:1905.11268.
  27. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  28. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085–1097, Florence, Italy. Association for Computational Linguistics.
  29. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118.
  30. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  31. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  32. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207.
  33. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840.
  34. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.
  35. Self-adaptive in-context learning. arXiv preprint arXiv:2212.10375.
  36. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.
  37. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066–6080, Online. Association for Computational Linguistics.
  38. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
  39. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Jiongxiao Wang (15 papers)
  2. Zichen Liu (34 papers)
  3. Keun Hee Park (6 papers)
  4. Zhuojun Jiang (3 papers)
  5. Zhaoheng Zheng (12 papers)
  6. Zhuofeng Wu (10 papers)
  7. Muhao Chen (159 papers)
  8. Chaowei Xiao (110 papers)
Citations (43)
View on Semantic Scholar