Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Are aligned neural networks adversarially aligned? (2306.15447v2)

Published 26 Jun 2023 in cs.CL, cs.AI, cs.CR, and cs.LG

Abstract: LLMs are now tuned to align with the goals of their creators, namely to be "helpful and harmless." These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs. However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.

PDF HTML Abstract

Introduction to Aligned Neural Networks

Aligned neural networks are designed to produce outputs that are in line with the intentions and ethical standards established by their creators. For LLMs, alignment means generating responses that are helpful to user queries while avoiding harmful content. Attempting to craft LLMs that behave in such a way has led to the application of various techniques like reinforcement learning through human feedback (RLHF). These efforts aim to ensure the models' outputs stay within the boundaries of what is deemed acceptable and avoid biases or toxicity. However, despite these efforts, no LLM is entirely safe from being manipulated into producing undesirable outputs through what are known as adversarial examples.

Adversarial Examples: A Challenge to Alignment

Adversarial examples are inputs tailored to trick neural networks into performing actions or generating outputs that they ordinarily wouldn't. Historically, this type of vulnerability has been extensively explored in the image recognition field. Such examples showcase how minute changes to an input image, imperceptible to the human eye, can lead to incorrect classification by the neural network. Researchers have extended this phenomenon to the domain of language, where adversarial inputs can be constructed to coax models into emitting harmful outputs. This raises a critical question: despite advanced alignment techniques, can LLMs maintain their alignment when confronted with these adversarily crafted inputs?

Evaluating the Robustness of Aligned Models

Recent investigations reveal that while current alignment strategies can defend against state-of-the-art text-based adversarial attacks, these attacks may not be powerful enough to be considered comprehensive tests for adversarial robustness. In essence, the successful defense against current attacks should not impart false confidence in the alignment of LLMs under all possible adversarial scenarios. In the face of adversarial users, even well-aligned models have shown some weaknesses, indicating that our ability to assess their robustness accurately remains incomplete.

The New Frontier: Multimodal Models

The paper emphasizes a shift towards multimodal models, which combine text and images or other data types in their inputs. These models open new avenues for user interaction but also present additional vulnerabilities. The research detailed in the paper illustrates that adversarial attacks using perturbed images can be especially effective against multimodal systems, causing them to generate harmful content more easily than with text alone. Unfortunately, current attacks are still lacking in effectively challenging text-only models, suggesting a gap in our understanding and prompting a need for the development of more robust attack methods to properly evaluate these LLMs.

In conclusion, while the alignment of neural networks signifies progress in pursuing more ethical AI, ensuring their robustness against adversarially designed prompts remains a significant challenge, particularly in multimodal contexts. Future research is urged to focus on refining adversarial attacks for a more accurate assessment of models' abilities to uphold their alignment in all circumstances.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461–463, 2021.
  2. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
  3. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018.
  4. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  5. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  6. Evasion attacks against machine learning at test time. In European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387–402. Springer, 2013.
  7. Nick Bostrom. Existential risk prevention as global priority. Global Policy, 4(1):15–31, 2013.
  8. Language models are few-shot learners, 2020.
  9. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228, 2018.
  10. Current and near-term ai as a potential existential risk factor. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 119–129, 2022.
  11. Joseph Carlsmith. Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022.
  12. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  13. Palm: Scaling language modeling with pathways, 2022.
  14. Deep reinforcement learning from human preferences, 2023.
  15. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 67–73, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360128. doi: 10.1145/3278721.3278729. URL https://doi.org/10.1145/3278721.3278729.
  16. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751, 2017.
  17. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022.
  18. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, jun 2022. doi: 10.1145/3531146.3533229. URL https://doi.org/10.1145/3531146.3533229.
  19. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023.
  20. News summarization and evaluation in the era of gpt-3, 2022.
  21. More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. arXiv preprint arXiv:2302.12173, 2023.
  22. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733, 2021.
  23. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017.
  24. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381, 2023.
  25. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pages 97–117. Springer, 2017.
  26. Adversarial malware binaries: Evading deep learning for malware detection in executables. In 2018 26th European signal processing conference (EUSIPCO), pages 533–537. IEEE, 2018.
  27. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  28. Holistic evaluation of language models, 2022.
  29. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
  30. Randomness in ml defenses helps persistent attackers and hinders evaluators. arXiv preprint arXiv:2302.13464, 2023.
  31. Richard Ngo. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022.
  32. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/abs/2303.08774.
  33. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
  34. Training language models to follow instructions with human feedback, 2022.
  35. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles, pages 1–18, 2017.
  36. Sundar Pichai. Google i/o 2023: Making ai more helpful for everyone. The Keyword, 2023.
  37. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  38. Scaling language models: Methods, analysis & insights from training Gopher, 2022.
  39. Reddit. Dan 5.0, 2023. URL https://www.reddit.com/r/ChatGPT/comments/10tevu1/new_jailbreak_proudly_unveiling_the_tried_and/.
  40. Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
  41. Erum Salam. I tried Be My Eyes, the popular app that pairs blind people with helpers. https://www.theguardian.com/lifeandstyle/2019/jul/12/be-my-eyes-app-blind-people-helpers, 2019.
  42. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  43. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  44. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
  45. Adversarial: Perceptual ad blocking meets adversarial machine learning. In ACM SIGSAC Conference on Computer and Communications Security, 2019.
  46. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33:1633–1645, 2020.
  47. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125, 2019.
  48. Finetuned language models are zero-shot learners, 2022a.
  49. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD. Survey Certification.
  50. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, 2021.
  51. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pages 5286–5295. PMLR, 2018.
  52. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
  53. Increasing confidence in adversarial robustness evaluations. arXiv preprint arXiv:2206.13991, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Nicholas Carlini (101 papers)
  2. Milad Nasr (48 papers)
  3. Christopher A. Choquette-Choo (49 papers)
  4. Matthew Jagielski (51 papers)
  5. Irena Gao (10 papers)
  6. Anas Awadalla (12 papers)
  7. Pang Wei Koh (64 papers)
  8. Daphne Ippolito (47 papers)
  9. Katherine Lee (34 papers)
  10. Ludwig Schmidt (80 papers)
  11. Florian Tramer (19 papers)
Citations (177)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets