Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Poison Large Language Models During Instruction Tuning (2402.13459v2)

Published 21 Feb 2024 in cs.LG, cs.CL, and cs.CR

Abstract: The advent of LLMs has marked significant achievements in language processing and reasoning capabilities. Despite their advancements, LLMs face vulnerabilities to data poisoning attacks, where adversaries insert backdoor triggers into training data to manipulate outputs for malicious purposes. This work further identifies additional security risks in LLMs by designing a new data poisoning attack tailored to exploit the instruction tuning process. We propose a novel gradient-guided backdoor trigger learning (GBTL) algorithm to identify adversarial triggers efficiently, ensuring an evasion of detection by conventional defenses while maintaining content integrity. Through experimental validation across various tasks, including sentiment analysis, domain generation, and question answering, our poisoning strategy demonstrates a high success rate in compromising various LLMs' outputs. We further propose two defense strategies against data poisoning attacks, including in-context learning (ICL) and continuous learning (CL), which effectively rectify the behavior of LLMs and significantly reduce the decline in performance. Our work highlights the significant security risks present during the instruction tuning of LLMs and emphasizes the necessity of safeguarding LLMs against data poisoning attacks.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. How to backdoor federated learning. In International conference on artificial intelligence and statistics, pages 2938–2948. PMLR.
  2. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pages 634–643. PMLR.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447.
  5. Badnl: Backdoor attacks against nlp models with semantic-preserving improvements. In Annual computer security applications conference, pages 554–569.
  6. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526.
  7. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023).
  8. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  9. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872–138878.
  10. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
  11. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  12. Triggerless backdoor attack for nlp tasks with clean labels. arXiv preprint arXiv:2111.07970.
  13. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  14. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230–47244.
  15. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327.
  16. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  17. Backdoor attacks on pre-trained models by layerwise weight poisoning. arXiv preprint arXiv:2108.13888.
  18. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems.
  19. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.
  20. Trojtext: Test-time invisible textual trojan insertion. arXiv preprint arXiv:2303.02242.
  21. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc.
  22. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773.
  23. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  24. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL.
  25. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  26. Hijacking large language models via adversarial in-context learning. arXiv preprint arXiv:2311.09948.
  27. Javier Rando and Florian Tramèr. 2023. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:2311.14455.
  28. Backdoor attacks on self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13337–13346.
  29. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
  30. Prompt-specific poisoning attacks on text-to-image generative models. arXiv preprint arXiv:2310.13828.
  31. Backdoor pre-trained models can transfer to all. arXiv preprint arXiv:2111.00197.
  32. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  33. On the exploitability of instruction tuning. arXiv preprint arXiv:2306.17194.
  34. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  35. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
  36. Stanford alpaca: An instruction-following llama model.
  37. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  38. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  39. Truth serum: Poisoning machine learning models to reveal their secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 2779–2792.
  40. Concealed data poisoning attacks on nlp models. arXiv preprint arXiv:2010.12563.
  41. Poisoning language models during instruction tuning. arXiv preprint arXiv:2305.00944.
  42. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698.
  43. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.
  44. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705.
  45. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  46. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214–229.
  47. Dba: Distributed backdoor attacks against federated learning. In 8th International Conference on Learning Representations, ICLR 2020.
  48. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305.14710.
  49. Backdooring instruction-tuned large language models with virtual prompt injection. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly.
  50. Backdoor attack against speaker verification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2560–2564. IEEE.
  51. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14443–14452.
  52. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206.
  53. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yao Qiang (16 papers)
  2. Xiangyu Zhou (51 papers)
  3. Saleh Zare Zade (3 papers)
  4. Mohammad Amin Roshani (3 papers)
  5. Douglas Zytko (10 papers)
  6. Dongxiao Zhu (41 papers)
  7. Prashant Khanduri (29 papers)
Citations (15)
View on Semantic Scholar