Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation (2401.07382v2)

Published 14 Jan 2024 in cs.CL and cs.AI

Abstract: Reinforcement learning (RL) can align LLMs with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of these reward signals - typically, there is only a single reward for an entire output. This sparsity of rewards can lead to inefficient and unstable learning. To address this challenge, our paper introduces an novel framework that utilizes the critique capability of LLMs to produce intermediate-step rewards during RL training. Our method involves coupling a policy model with a critic LLM, which is responsible for providing comprehensive feedback of each part of the output. This feedback is then translated into token or span-level rewards that can be used to guide the RL training process. We investigate this approach under two different settings: one where the policy model is smaller and is paired with a more powerful critic model, and another where a single LLM fulfills both roles. We assess our approach on three text generation tasks: sentiment control, LLM detoxification, and summarization. Experimental results show that incorporating artificial intrinsic rewards significantly improve both sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  2. Anonymous. 2023. Let’s verify step by step. In Submitted to The Twelfth International Conference on Learning Representations. Under review.
  3. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations.
  4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  5. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  6. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR.
  7. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. Ask the right questions: Active question reformulation with reinforcement learning. In International Conference on Learning Representations.
  10. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186.
  11. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340–3354, Dublin, Ireland. Association for Computational Linguistics.
  12. Learning with rejection for abstractive text summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9768–9780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  13. Systematic rectification of language models via dead-end analysis. In The Eleventh International Conference on Learning Representations.
  14. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
  15. Exploration-guided reward shaping for reinforcement learning under sparse rewards. In Advances in Neural Information Processing Systems.
  16. Towards end-to-end reinforcement learning of dialogue agents for information access. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–495, Vancouver, Canada. Association for Computational Linguistics.
  17. BanditSum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739–3748, Brussels, Belgium. Association for Computational Linguistics.
  18. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692.
  19. Policy networks with two-stage training for dialogue systems. In Proceedings of SIGDial 2016. arXiv.
  20. APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120–4130, Brussels, Belgium. Association for Computational Linguistics.
  21. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics.
  22. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus.
  23. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020.
  24. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics.
  25. Dual learning for machine translation. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  26. Classifying emotions in customer support dialogues in social media. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 64–73, Los Angeles. Association for Computational Linguistics.
  27. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  28. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456.
  29. Mistral 7b. arXiv preprint arXiv:2310.06825.
  30. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  31. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166.
  32. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4929–4952, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  33. The nethack learning environment. In Advances in Neural Information Processing Systems, volume 33, pages 7671–7684. Curran Associates, Inc.
  34. Reward design with language models. In The Eleventh International Conference on Learning Representations.
  35. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267.
  36. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Austin, Texas. Association for Computational Linguistics.
  37. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  38. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691–6706, Online. Association for Computational Linguistics.
  39. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems.
  40. Critique ability of large language models. arXiv preprint arXiv:2310.04815.
  41. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310.12931.
  42. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  43. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
  44. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928–1937, New York, New York, USA. PMLR.
  45. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
  46. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332.
  47. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer.
  48. Reward augmented maximum likelihood for neural structured prediction. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  49. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
  50. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2721–2730. PMLR.
  51. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  52. Text generation by learning from demonstrations. In International Conference on Learning Representations.
  53. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR.
  54. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2778–2787. PMLR.
  55. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231–2240, Copenhagen, Denmark. Association for Computational Linguistics.
  56. Martin L. Puterman. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.
  57. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  58. Seonggi Ryang and Takeshi Abekawa. 2012. Framework of automatic text summarization using reinforcement learning. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 256–265, Jeju Island, Korea. Association for Computational Linguistics.
  59. Self-critiquing models for assisting human evaluators, 2022. URL https://arxiv. org/abs/2206.05802.
  60. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
  61. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  62. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc.
  63. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 147–157, Saarbrücken, Germany. Association for Computational Linguistics.
  64. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  65. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press.
  66. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  67. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv. org/abs/2307.09288.
  68. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark. Association for Computational Linguistics.
  69. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
  70. A large-scale dataset for empathetic response generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251–1264, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  71. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
  72. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693.
  73. DCN+: Mixed objective and deep residual coattention for question answering. In International Conference on Learning Representations.
  74. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations.
  75. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguistics.
  76. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
  77. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Meng Cao (107 papers)
  2. Lei Shu (82 papers)
  3. Lei Yu (234 papers)
  4. Yun Zhu (52 papers)
  5. Nevan Wichers (11 papers)
  6. Yinxiao Liu (8 papers)
  7. Lei Meng (54 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets