Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game (2311.08045v4)

Published 14 Nov 2023 in cs.CL, cs.AI, and cs.LG

Abstract: Human preference alignment is essential to improve the interaction quality of LLMs. Existing alignment methods depend on manually annotated preference data to guide the LLM optimization directions. However, continuously updating LLMs for alignment raises a distribution gap between model-generated samples and human-annotated responses, hindering training effectiveness. To mitigate this issue, previous methods require additional preference annotation on newly generated samples to adapt to the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a min-max game. Through adversarial training, the reward model can adapt to the shifted generation distribution of the LLM without any additional annotation. With comprehensive experiments, we find the proposed adversarial training framework further enhances existing alignment baselines in terms of LLM helpfulness and harmlessness. The code is at https://github.com/Linear95/APO.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214–223. PMLR.
  2. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861.
  3. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036.
  4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  5. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.
  6. Everyone deserves a reward: Learning customized human preferences. arXiv preprint arXiv:2309.03126.
  7. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.
  8. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767.
  9. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867.
  10. Generative adversarial nets. Advances in neural information processing systems, 27.
  11. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450.
  12. Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems, 29.
  13. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745.
  14. Can neural machine translation be improved with user feedback? In Proceedings of NAACL-HLT, pages 92–105.
  15. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439.
  16. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676.
  17. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657.
  18. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.
  19. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  20. OpenAI. 2023a. ChatGPT, Mar 14 version. https://chat.openai.com/chat.
  21. OpenAI. 2023b. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  22. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  23. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  24. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  25. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  26. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.
  27. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.05910.
  28. Nigar M Shafiq Surameery and Mohammed Y Shakor. 2023. Use chat gpt to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290, 3(01):17–22.
  29. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
  30. Is chatgpt the ultimate programming assistant–how far is it? arXiv preprint arXiv:2304.11938.
  31. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  32. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  33. Cédric Villani. 2009. Optimal transport: old and new, volume 338. Springer.
  34. Openchat: Advancing open-source language models with mixed-quality data.
  35. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.
  36. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  37. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14067–14075.
  38. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
  39. Generating text via adversarial training. NIPS workshop on Adversarial Training.
  40. Adversarial feature matching for text generation. In International conference on machine learning, pages 4006–4015. PMLR.
  41. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425.
  42. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
  43. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Pengyu Cheng (23 papers)
  2. Yifan Yang (578 papers)
  3. Jian Li (667 papers)
  4. Yong Dai (33 papers)
  5. Tianhao Hu (7 papers)
  6. Peixin Cao (1 paper)
  7. Nan Du (66 papers)
  8. Xiaolong Li (107 papers)
Citations (19)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Linear95/APO: Implementation of Adversarial Preference Optimization (APO) (54 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets