Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards (2403.07708v2)

Published 12 Mar 2024 in cs.CL and cs.AI

Abstract: Reinforcement learning from human feedback (RLHF) is the mainstream paradigm used to align LLMs with human preferences. Yet existing RLHF heavily relies on accurate and informative reward models, which are vulnerable and sensitive to noise from various sources, e.g. human labeling errors, making the pipeline fragile. In this work, we improve the effectiveness of the reward model by introducing a penalty term on the reward, named as \textit{contrastive rewards}. %Contrastive rewards Our approach involves two steps: (1) an offline sampling step to obtain responses to prompts that serve as baseline calculation and (2) a contrastive reward calculated using the baseline responses and used in the Proximal Policy Optimization (PPO) step. We show that contrastive rewards enable the LLM to penalize reward uncertainty, improve robustness, encourage improvement over baselines, calibrate according to task difficulty, and reduce variance in PPO. We show empirically contrastive rewards can improve RLHF substantially, evaluated by both GPTs and humans, and our method consistently outperforms strong baselines.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Concrete problems in ai safety, 2016.
  2. A general language assistant as a laboratory for alignment, 2021.
  3. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036, 2023.
  4. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
  5. Constitutional ai: Harmlessness from ai feedback, 2022b.
  6. Red-teaming large language models using chain of utterances for safety-alignment, 2023.
  7. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
  8. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
  9. A simple framework for contrastive learning of visual representations, 2020.
  10. Self-play fine-tuning converts weak language models to strong language models, 2024.
  11. Adversarial preference optimization. arXiv preprint arXiv:2311.08045, 2023.
  12. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL \urlhttps://lmsys.org/blog/2023-03-30-vicuna/.
  13. Deep reinforcement learning from human preferences, 2023.
  14. Reward model ensembles help mitigate overoptimization. ArXiv, abs/2310.02743, 2023. URL \urlhttps://api.semanticscholar.org/CorpusID:263620686.
  15. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
  16. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023.
  17. Raft: Reward ranked finetuning for generative foundation model alignment, 2023.
  18. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.
  19. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
  20. Scaling laws for reward model overoptimization, 2022.
  21. Accelerate: Training and inference at scale made simple, efficient and adaptable. \urlhttps://github.com/huggingface/accelerate, 2022.
  22. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.13639, 2023.
  23. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 2019.
  24. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion, 2023.
  25. Beyond reward: Offline preference-guided policy optimization, 2023.
  26. Alignment of language agents, 2021.
  27. Rl with kl penalties is better viewed as bayesian inference, 2022.
  28. The history and risks of reinforcement learning and human feedback, 2023.
  29. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference ron Machine Learning (ICML 2000), pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  30. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.
  31. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584, 2023.
  32. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2015.
  33. Statistical rejection sampling improves preference optimization, 2024.
  34. Decoupled weight decay regularization, 2019.
  35. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
  36. OpenAI. Gpt-4 technical report, 2023.
  37. Training language models to follow instructions with human feedback, 2022.
  38. Pitis, S. Failure modes of learning reward models for llms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023.
  39. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  40. Zero: Memory optimizations toward training trillion parameter models, 2020.
  41. Proximal policy optimization algorithms, 2017.
  42. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
  43. Defining and characterizing reward hacking, 2022.
  44. Learning to summarize from human feedback, 2022.
  45. Reinforcement learning: An introduction. MIT press, 2018.
  46. Llama 2: Open foundation and fine-tuned chat models, 2023a.
  47. Llama 2: Open foundation and fine-tuned chat models, 2023b.
  48. Zephyr: Direct distillation of lm alignment, 2023.
  49. Secrets of rlhf in large language models part ii: Reward modeling, 2024.
  50. Policy learning using weak supervision. Advances in Neural Information Processing Systems, 34:19960–19973, 2021.
  51. The optimal reward baseline for gradient-based reinforcement learning. arXiv preprint arXiv:1301.2315, 2013.
  52. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment. arXiv preprint arXiv:2310.00212, 2023.
  53. Self-rewarding language models, 2024.
  54. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
  55. Slic-hf: Sequence likelihood calibration with human feedback, 2023.
  56. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023a.
  57. Secrets of rlhf in large language models part i: Ppo, 2023b.
  58. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pp.  12912–12923. PMLR, 2021.
  59. Unmasking and improving data credibility: A study with datasets for training harmless language models. arXiv preprint arXiv:2311.11202, 2023.
  60. Fine-tuning language models from human preferences, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Wei Shen (181 papers)
  2. Xiaoying Zhang (32 papers)
  3. Yuanshun Yao (28 papers)
  4. Rui Zheng (78 papers)
  5. Hongyi Guo (14 papers)
  6. Yang Liu (2253 papers)
Citations (6)
View on Semantic Scholar