Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning (2404.19409v1)

Published 30 Apr 2024 in cs.CL

Abstract: While Reinforcement Learning (RL) has been proven essential for tuning LLMs, it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036.
  3. An actor-critic algorithm for sequence prediction. In Proc. of International Conference on Learning Representations (ICLR).
  4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  5. Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. of Advances in Neural Information Processing Systems (NIPS).
  6. Deep reinforcement learning from human preferences. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  7. Training language gans from scratch. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  8. Scaling laws for reward model overoptimization. In Proc. of International Conference on Machine Learning (ICML).
  9. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375.
  10. Deep q-learning from demonstrations. In Proc. of AAAI conference on artificial intelligence.
  11. The curious case of neural text degeneration. In Proc. of International Conference on Learning Representations (ICLR).
  12. LoRA: Low-rank adaptation of large language models. In Proc. of International Conference on Learning Representations (ICLR).
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  14. Openassistant conversations–democratizing large language model alignment. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  15. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proc. of the Association for Computational Linguistics (ACL).
  16. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  17. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267.
  18. Countering language drift via visual grounding. In Proc. of the conference on Empirical Methods in Natural Language Processing (EMNLP).
  19. Olivier Lemon and Olivier Pietquin. 2007. Machine learning for spoken dialogue systems. In Proc. of European Conference on Speech Communication and Technologies (Interspeech).
  20. Deal or no deal? end-to-end learning for negotiation dialogues. In Proc. of the conference on Empirical Methods in Natural Language Processing (EMNLP).
  21. Countering language drift with seeded iterated learning. In Proc. of International Conference on Machine Learning (ICML).
  22. Learning word vectors for sentiment analysis. In Proc. of the Association for Computational Linguistics (ACL).
  23. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.
  24. Confronting reward model overoptimization with constrained rlhf. In Proc. of International Conference on Learning Representations (ICLR).
  25. Overcoming exploration in reinforcement learning with demonstrations. In Proc. of International Conference on Robotics and Automation (ICRA).
  26. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  27. Language model alignment with elastic reset. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  28. OpenAI. 2023. Gpt-4 technical report.
  29. Training language models to follow instructions with human feedback. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  30. A deep reinforced model for abstractive summarization. In Proc. of International Conference on Learning Representations (ICLR).
  31. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1–14.
  32. Guided reinforcement learning with learned skills. In Proc. of Conference on Robot Learning (CoRL).
  33. Direct preference optimization: Your language model is secretly a reward model. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  34. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS).
  35. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. In Proc. of International Conference on Learning Representations (ICLR).
  36. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Proc. of International Conference on Learning Representations (ICLR).
  37. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187.
  38. Model-free reinforcement learning from expert demonstrations: a survey. Artificial Intelligence Review, pages 1–29.
  39. Sequence level training with recurrent neural networks. In Proc. of International Conference on Learning Representations (ICLR).
  40. Factually consistent summarization via reinforcement learning with textual entailment feedback. In Proc. of the Association for Computational Linguistics (ACL).
  41. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  42. Stefan Schaal. 1996. Learning from demonstration. In Proc. of Advances in Neural Information Processing Systems (NIPS).
  43. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(2):97–126.
  44. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  45. Reinforcement learning for spoken dialogue systems. In Proc. of Advances in Neural Information Processing Systems (NIPS).
  46. Defining and characterizing reward gaming. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  47. Learning to summarize with human feedback. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  48. End-to-end optimization of goal-driven and visually grounded dialogue systems. In Proc. of International Joint Conference on Artificial Intelligence (IJCAI).
  49. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
  50. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  51. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.
  52. Tl; dr: Mining reddit to learn automatic summarization. In Proc. of the Workshop on New Frontiers in Summarization.
  53. Airdialogue: An environment for goal-oriented dialogue research. In Proc. of the conference on Empirical Methods in Natural Language Processing (EMNLP).
  54. Wikimedia. 2023. Wikimedia downloads. https://dumps.wikimedia.org.
  55. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280.
  56. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence.
  57. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893.
  58. Calibrating sequence likelihood improves conditional language generation. In Proc. of International Conference on Learning Representations (ICLR).
  59. Reinforcement and imitation learning for diverse visuomotor skills. In Proc. of Advances in Neural Information Processing Systems (NeurIPS).
  60. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Mathieu Rita (7 papers)
  2. Florian Strub (39 papers)
  3. Rahma Chaabouni (15 papers)
  4. Paul Michel (27 papers)
  5. Emmanuel Dupoux (81 papers)
  6. Olivier Pietquin (90 papers)
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets