Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring Backdoor Vulnerabilities of Chat Models (2404.02406v1)

Published 3 Apr 2024 in cs.CR, cs.AI, and cs.CL
Exploring Backdoor Vulnerabilities of Chat Models

Abstract: Recent researches have shown that LLMs are susceptible to a security threat known as Backdoor Attack. The backdoored model will behave well in normal cases but exhibit malicious behaviours on inputs inserted with a specific backdoor trigger. Current backdoor studies on LLMs predominantly focus on instruction-tuned LLMs, while neglecting another realistic scenario where LLMs are fine-tuned on multi-turn conversational data to be chat models. Chat models are extensively adopted across various real-world scenarios, thus the security of chat models deserves increasing attention. Unfortunately, we point out that the flexible multi-turn interaction format instead increases the flexibility of trigger designs and amplifies the vulnerability of chat models to backdoor attacks. In this work, we reveal and achieve a novel backdoor attacking method on chat models by distributing multiple trigger scenarios across user inputs in different rounds, and making the backdoor be triggered only when all trigger scenarios have appeared in the historical conversations. Experimental results demonstrate that our method can achieve high attack success rates (e.g., over 90% ASR on Vicuna-7B) while successfully maintaining the normal capabilities of chat models on providing helpful responses to benign user requests. Also, the backdoor can not be easily removed by the downstream re-alignment, highlighting the importance of continued research and attention to the security concerns of chat models. Warning: This paper may contain toxic content.

PDF HTML Abstract

Exploring Backdoor Vulnerabilities in Chat Models

Introduction

In the burgeoning field of chat models, a notable paper has highlighted a critical vulnerability: backdoor attacks. These attacks manipulate chat models to operate normally under regular usage but to execute pre-defined malicious behaviors when triggered by specific inputs. This paper unveils a novel approach towards backdoor attacks in chat models, a subject that has been largely understudied in comparison to its instruction-tuned counterparts. Focused on multi-turn conversational data fine-tuning, this work exposes the inherent vulnerability of chat models to such attacks, facilitated by the flexible format of multi-turn interactions.

Backdoor Attacks on Chat Models

The paper presents a landscape where chat models, integral to various digital interactions, are susceptible to backdoor attacks, elevating a considerable security concern. The inherent flexibility of multi-turn interactions in chat models provides a fertile ground for designing intricate trigger mechanisms. Unlike the prevailing studies on backdoor attacks tailored for instruction-tuned LLMs, which either involve insertion of static words or sentences or specific scenarios as triggers, this work posits that the multi-turn conversation format of chat models permit the distribution of multiple trigger scenarios across different rounds of conversation, significantly amplifying the potential for stealthy and effective backdoor attacks.

Distributed Triggers-Based Backdoor Attack Framework

This paper introduces a "Distributed Triggers-based Backdoor Attacking" framework targeting chat models. The crux of this methodology revolves around distributing multiple trigger scenarios across user inputs in discrete conversation rounds. The backdoor is engineered to activate only when all specified trigger scenarios have surfaced in the conversation history. This approach underscores a drastic shift from existing methodologies by leveraging the sequential and contextual nature of multi-turn dialogues. The experimental evaluation, conducted on two chat models, highlighted an attack success rate surpassing 90\% in certain scenarios, demonstrating the method's efficacy without compromising the chat model's functionality in benign contexts. Moreover, the resistance of this backdoor mechanism against downstream re-alignment efforts was also evidenced, underscoring the critical need for robust countermeasures.

Implications and Future Directions

The implications of this research are profound, spanning both theoretical advancements and practical considerations in the deployment of chat models. The revelation of such vulnerabilities necessitates a reevaluation of security practices surrounding the application of LLMs in conversational settings. It prompts further inquiry into the development of sophisticated detection and mitigation strategies against backdoor attacks, ensuring the integrity and trustworthiness of chat models in real-world applications. Additionally, this paper opens avenues for future research to explore countermeasures that can effectively identify and neutralize such backdoor triggers without undermining the model's performance or utility.

Conclusion

This paper marks a significant step towards understanding and mitigating backdoor vulnerabilities in chat models, a subject not widely explored in the context of LLMs. By showcasing a novel attack mechanism that exploits the multi-turn interaction format, it shines a spotlight on the urgent need for comprehensive security measures in the development and deployment of chat models. As the adoption of such models continues to grow, addressing these vulnerabilities becomes imperative to safeguard against malicious exploitations that threaten user trust and model integrity.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Badprompt: Backdoor attacks on continuous prompts. Advances in Neural Information Processing Systems, 35:37068–37080.
  4. Stealthy and persistent unalignment on large language models via backdoor injections. arXiv preprint arXiv:2312.00027.
  5. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006.01043.
  6. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. URL https://vicuna.lmsys.org/.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  8. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.
  9. Ppt: Backdoor attacks on pre-trained models via poisoned prompt tuning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 680–686.
  10. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
  11. Mixtral of experts. arXiv preprint arXiv:2401.04088.
  12. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793–2806, Online. Association for Computational Linguistics.
  13. OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue.
  14. OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.
  15. ONION: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9558–9566, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  16. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 443–453, Online. Association for Computational Linguistics.
  17. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4873–4883, Online. Association for Computational Linguistics.
  18. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  19. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J, 3(1):e103–e103.
  20. Stanford alpaca: an instruction-following llama model (2023). URL https://crfm. stanford. edu/2023/03/13/alpaca. html, 1(2):3.
  21. Large language models in medicine. Nature medicine, 29(8):1930–1940.
  22. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  23. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  24. Poisoning language models during instruction tuning. arXiv preprint arXiv:2305.00944.
  25. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.
  26. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305.14710.
  27. Backdooring instruction-tuned large language models with virtual prompt injection. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly.
  28. Watch out for your agents! investigating backdoor threats to llm-based agents. arXiv preprint arXiv:2402.11208.
  29. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2048–2058, Online. Association for Computational Linguistics.
  30. Rethinking stealthiness of backdoor attack against NLP models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5543–5557, Online. Association for Computational Linguistics.
  31. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.
  32. Red alarm for pre-trained models: Universal vulnerabilities by neuron-level backdoor attacks. arXiv preprint arXiv:2101.06969.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Yunzhuo Hao (7 papers)
  2. Wenkai Yang (24 papers)
  3. Yankai Lin (125 papers)
Citations (8)
View on Semantic Scholar