Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
86 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
53 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play (2411.00062v3)

Published 31 Oct 2024 in cs.CL, cs.AI, physics.data-an, and stat.ML

Abstract: Current reinforcement learning (RL) frameworks for LLMs (LLM) post-training typically assume a fixed prompt distribution, which is sub-optimal and bottlenecks scalability. Prior works have explored prompt evolving, but are often limited to the supervised fine-tuning stage, and prompts are sampled and evolved uniformly without signals. This empirical work presents a paradigm shift: Evolving Alignment via Asymmetric Self-Play (eva), that casts post-training as an infinite game with regret-based signals for 2 players: (i) a creator, who strategically samples and creates new informative prompts and (ii) a solver, who learns to produce preferred responses. eva is the first method that allows LLMs to adaptively create training prompts in both offline and online RL post-training. The design is simple, easy-to-use yet remarkably effective: eva sets a new SOTA on challenging benchmarks, without any extra human prompts, e.g. it boosts the win-rate of gemma-2-9b-it on Arena-Hard by 51.6% -> 60.1% for DPO and 52.6% -> 62.4% for RLOO, surpassing claude-3-opus and catching up to gemini-1.5-pro, both of which are orders of magnitude larger. Extensive experiments show eva can create effective RL curricula and is robust across ablations. We believe adaptively evolving prompts are key to designing the next-generation RL post-training scheme.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pages 620–636. PMLR, 2023.
  2. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024.
  3. Learning to give checkable answers with prover-verifier games. arXiv preprint arXiv:2108.12099, 2021.
  4. On the difficulty of warm-starting neural network training. arXiv preprint arXiv:1910.08475, 2019.
  5. Alfredo Banos. On pseudo-games. The Annals of Mathematical Statistics, 39(6):1932–1945, 1968.
  6. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  7. Refining Minimax Regret for Unsupervised Environment Design. arXiv preprint arXiv:2402.12284, 2024.
  8. Seth Chaiklin et al. The zone of proximal development in Vygotsky’s analysis of learning and instruction. Vygotsky’s educational theory in cultural context, 1(2):39–64, 2003.
  9. Self-Improving Robust Preference Optimization. arXiv preprint arXiv:2406.01660, 2024.
  10. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  11. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
  12. Provably sample efficient rlhf via active preference optimization. arXiv preprint arXiv:2402.10500, 2024.
  13. Using expectation-maximization for reinforcement learning. Neural Computation, 9(2):271–278, 1997.
  14. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020.
  15. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13812, 2023.
  16. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
  17. RLHF Workflow: From Reward Modeling to Online RLHF, 2024.
  18. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
  19. Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences, 39(1):42–47, 1953.
  20. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024.
  21. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  22. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023.
  23. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532, 2023.
  24. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.13639, 2023.
  25. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
  26. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
  27. Open-Endedness is Essential for Artificial Superhuman Intelligence. arXiv preprint arXiv:2406.04268, 2024a.
  28. Open-Endedness is Essential for Artificial Superhuman Intelligence. arXiv preprint arXiv:2406.04268, 2024b.
  29. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019.
  30. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023.
  31. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884–1897, 2021a.
  32. Prioritized level replay. In International Conference on Machine Learning, pages 4940–4950. PMLR, 2021b.
  33. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pages 669–679. PMLR, 2020.
  34. John Maynard Keynes. A treatise on probability. Courier Corporation, 1921.
  35. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064, 2024a.
  36. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv preprint arXiv:2406.11939, 2024b.
  37. Skywork Reward Model Series. https://huggingface.co/Skywork, September 2024. URL https://huggingface.co/Skywork.
  38. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126, 2024.
  39. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
  40. SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv preprint arXiv:2405.14734, 2024.
  41. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630–15649. PMLR, 2022.
  42. Active Preference Learning for Large Language Models. arXiv preprint arXiv:2402.08114, 2024.
  43. Nash learning from human feedback. arXiv preprint arXiv:2312.00886, 2023.
  44. John F Nash et al. Non-cooperative games. Princeton University, 1950.
  45. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
  46. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024.
  47. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, pages 17473–17498. PMLR, 2022.
  48. Learning Formal Mathematics From Intrinsic Motivation. arXiv preprint arXiv:2407.00695, 2024.
  49. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  50. Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. arXiv preprint arXiv:2404.03715, 2024.
  51. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning, 11(1):1–96, 2018.
  52. Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210–229, 1959.
  53. MAESTRO: Open-ended environment design for multi-agent reinforcement learning. arXiv preprint arXiv:2303.03376, 2023.
  54. Leonard J Savage. The theory of statistical decision. Journal of the American Statistical association, 46(253):55–67, 1951.
  55. Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, 1991.
  56. Hans-Paul Schwefel. Evolutionsstrategien für die numerische Optimierung. Springer, 1977.
  57. Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
  58. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  59. Reward is enough. Artificial Intelligence, 299:103535, 2021.
  60. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023.
  61. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017.
  62. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
  63. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pages 761–768, 2011.
  64. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024.
  65. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
  66. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  67. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
  68. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.
  69. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017.
  70. Iterative DPO Alignment. Technical report, Snorkel AI, 2023.
  71. Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv preprint arXiv:2211.04325, 2024.
  72. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. arXiv preprint arXiv:2406.12845, 2024.
  73. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019.
  74. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  75. Double thompson sampling for dueling bandits. Advances in neural information processing systems, 29, 2016.
  76. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
  77. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
  78. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023a.
  79. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023b.
  80. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. arXiv preprint arXiv:2406.08464, 2024.
  81. To repeat or not to repeat: Insights from scaling llm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024.
  82. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
  83. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023.
  84. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
  85. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
  86. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  87. Toward Optimal LLM Alignments Using Two-Player Games. arXiv preprint arXiv:2406.10977, 2024.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now