Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond (2310.06147v1)

Published 9 Oct 2023 in cs.LG and cs.AI

Abstract: Recent advancements in LLMs have garnered wide attention and led to successful products such as ChatGPT and GPT-4. Their proficiency in adhering to instructions and delivering harmless, helpful, and honest (3H) responses can largely be attributed to the technique of Reinforcement Learning from Human Feedback (RLHF). In this paper, we aim to link the research in conventional RL to RL techniques used in LLM research. Demystify this technique by discussing why, when, and how RL excels. Furthermore, we explore potential future avenues that could either benefit from or contribute to RLHF research. Highlighted Takeaways: 1. RLHF is Online Inverse RL with Offline Demonstration Data. 2. RLHF $>$ SFT because Imitation Learning (and Inverse RL) $>$ Behavior Cloning (BC) by alleviating the problem of compounding error. 3. The RM step in RLHF generates a proxy of the expensive human feedback, such an insight can be generalized to other LLM tasks such as prompting evaluation and optimization where feedback is also expensive. 4. The policy learning in RLHF is more challenging than conventional problems studied in IRL due to their high action dimensionality and feedback sparsity. 5. The main superiority of PPO over off-policy value-based methods is its stability gained from (almost) on-policy data and conservative policy updates.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2107.04200, 2021.
  2. Recurrent model-free rl can be a strong baseline for many pomdps. arXiv preprint arXiv:2110.05038, 2021.
  3. Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems, 32, 2019.
  4. Zeroth-order supervised policy improvement. arXiv preprint arXiv:2006.06600, 2020a.
  5. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005.10696, 2020b.
  6. Constrained mdps can be solved by eearly-termination with recurrent models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022a.
  7. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  8. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1–14, 2018.
  9. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719–37734, 2022b.
  10. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
  11. Strictly batch imitation learning by energy-based distribution matching. Advances in Neural Information Processing Systems, 33:7354–7365, 2020.
  12. Accountable batched control with decision corpus. Advances in Neural Information Processing Systems, 36, 2023.
  13. Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
  14. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  15. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292–6299. IEEE, 2018.
  16. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  17. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017a.
  18. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018.
  19. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
  20. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  21. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04478, 2022.
  22. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783–792. PMLR, 2019.
  23. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  24. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  25. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
  26. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
  27. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  28. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
  29. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018.
  30. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
  31. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015.
  32. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676, 2023.
  33. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.05206, 2023.
  34. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017b.
  35. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. Pmlr, 2014.
  36. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626, 2016.
  37. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352–1361. PMLR, 2017.
  38. Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint arXiv:2309.06553, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)
  1. Hao Sun (383 papers)
Citations (12)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube