Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2402.16030v1)
Abstract: While Reinforcement Learning from Human Feedback (RLHF) significantly enhances the generation quality of LLMs, recent studies have raised concerns regarding the complexity and instability associated with the Proximal Policy Optimization (PPO) algorithm, proposing a series of order-based calibration methods as viable alternatives. This paper delves further into current order-based methods, examining their inefficiencies in utilizing reward values and addressing misalignment issues. Building upon these findings, we propose a novel \textbf{V}alue-based \textbf{C}ali\textbf{B}ration (VCB) method to better align LLMs with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and stability in diverse settings.
- A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036.
- Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
- Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
- Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR.
- Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.
- Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723.
- A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC.
- Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), volume 2, pages 1735–1742. IEEE.
- Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations.
- Revisiting design choices in proximal policy optimization. arXiv preprint arXiv:2009.10897.
- A Ya Khinchin. 2013. Mathematical foundations of information theory. Courier Corporation.
- Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673.
- On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35:16203–16220.
- Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
- Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657.
- Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
- Boosting the speed of entity alignment 10×\times×: Dual attention matching network with normalized hard sample mining. In Proceedings of the Web Conference 2021, pages 821–832.
- Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
- Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
- Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
- Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.17492.
- Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.
- Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
- Self-rewarding language models. arXiv preprint arXiv:2401.10020.
- Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
- Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848.
- Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425.
- Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.