Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimization-based Prompt Injection Attack to LLM-as-a-Judge (2403.17710v4)

Published 26 Mar 2024 in cs.CR and cs.AI

Abstract: LLM-as-a-Judge uses a LLM to select the best response from a set of candidates for a given question. LLM-as-a-Judge has many applications such as LLM-powered search, reinforcement learning with AI feedback (RLAIF), and tool selection. In this work, we propose JudgeDeceiver, an optimization-based prompt injection attack to LLM-as-a-Judge. JudgeDeceiver injects a carefully crafted sequence into an attacker-controlled candidate response such that LLM-as-a-Judge selects the candidate response for an attacker-chosen question no matter what other candidate responses are. Specifically, we formulate finding such sequence as an optimization problem and propose a gradient based method to approximately solve it. Our extensive evaluation shows that JudgeDeceive is highly effective, and is much more effective than existing prompt injection attacks that manually craft the injected sequences and jailbreak attacks when extended to our problem. We also show the effectiveness of JudgeDeceiver in three case studies, i.e., LLM-powered search, RLAIF, and tool selection. Moreover, we consider defenses including known-answer detection, perplexity detection, and perplexity windowed detection. Our results show these defenses are insufficient, highlighting the urgent need for developing new defense strategies. Our implementation is available at this repository: https://github.com/ShiJiawenwen/JudgeDeceiver.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. M. AI. Mixtral of experts, 2023.
  3. G. Alon and M. Kamfonas. Detecting language model attacks with perplexity, 2023.
  4. Anthropic. Claude 2, 2023.
  5. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024.
  6. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. arXiv preprint arXiv:2402.04788, 2024.
  7. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023.
  8. Masterkey: Automated jailbreaking of large language model chatbots. NDSS, 2024.
  9. Large language models in education: Vision and opportunities, 2023.
  10. Gemma. 2024.
  11. R. Goodside. Prompt injection attacks against gpt-3, 2023.
  12. Google. Bard, 2023.
  13. More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. arXiv e-prints, pages arXiv–2302, 2023.
  14. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection, 2023.
  15. N. Group. Exploring prompt injection attacks, 2023.
  16. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
  17. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
  18. Metatool benchmark: Deciding whether to use tools and which to use. arXiv preprint arXiv: 2310.03128, 2023.
  19. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043–2059. IEEE, 2022.
  20. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  21. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381, 2023.
  22. T. Kocmi and C. Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520, 2023.
  23. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2023.
  24. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
  25. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023.
  26. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023.
  27. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044, 2024.
  28. Alignbench: Benchmarking chinese alignment of large language models, 2023.
  29. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
  30. Prompt injection attack against llm-integrated applications, 2024.
  31. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023.
  32. Deid-gpt: Zero-shot medical text de-identification by gpt-4, 2023.
  33. Microsoft. Bing chat, 2023.
  34. OpenAI. Chatgpt, 2023.
  35. OpenAI. Chatgpt plugins, 2023.
  36. OpenAI. Transforming work and creativity with ai, 2023.
  37. F. Perez and I. Ribeiro. Ignore previous prompt: Attack techniques for language models, 2022.
  38. Communicative agents for software development, 2023.
  39. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt, 2023.
  40. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020.
  41. Trustllm: Trustworthiness in large language models, 2024.
  42. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  43. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235, 2023.
  44. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023.
  45. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
  46. Watch out for your agents! investigating backdoor threats to llm-based agents, 2024.
  47. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023.
  48. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.
  49. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
  50. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:2308.01862, 2023.
  51. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
  52. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023.
  53. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140, 2023.
  54. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Jiawen Shi (11 papers)
  2. Zenghui Yuan (8 papers)
  3. Yinuo Liu (4 papers)
  4. Yue Huang (171 papers)
  5. Pan Zhou (221 papers)
  6. Lichao Sun (186 papers)
  7. Neil Zhenqiang Gong (118 papers)
Citations (18)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube