Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization (2405.16681v1)
Abstract: LLMs perform well across diverse tasks, but aligning them with human demonstrations is challenging. Recently, Reinforcement Learning (RL)-free methods like Direct Preference Optimization (DPO) have emerged, offering improved stability and scalability while retaining competitive performance relative to RL-based methods. However, while RL-free methods deliver satisfactory performance, they require significant data to develop a robust Supervised Fine-Tuned (SFT) model and an additional step to fine-tune this model on a preference dataset, which constrains their utility and scalability. In this paper, we introduce Triple Preference Optimization (TPO), a new preference learning method designed to align an LLM with three preferences without requiring a separate SFT step and using considerably less data. Through a combination of practical experiments and theoretical analysis, we show the efficacy of TPO as a single-step alignment strategy. Specifically, we fine-tuned the Phi-2 (2.7B) and Mistral (7B) models using TPO directly on the UltraFeedback dataset, achieving superior results compared to models aligned through other methods such as SFT, DPO, KTO, IPO, CPO, and ORPO. Moreover, the performance of TPO without the SFT component led to notable improvements in the MT-Bench score, with increases of +1.27 and +0.63 over SFT and DPO, respectively. Additionally, TPO showed higher average accuracy, surpassing DPO and SFT by 4.2% and 4.97% on the Open LLM Leaderboard benchmarks. Our code is publicly available at https://github.com/sahsaeedi/triple-preference-optimization .
- Palm 2 technical report.
- A general theoretical paradigm to understand learning from human preferences.
- Training a helpful and harmless assistant with reinforcement learning from human feedback.
- BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
- Heejong Bong and Alessandro Rinaldo. 2022. Generalized results for the existence and consistency of the mle in the bradley-terry-luce model.
- Language models are few-shot learners.
- Sparks of artificial general intelligence: Early experiments with gpt-4.
- Getting it right: Improving spatial consistency in text-to-image models. arXiv preprint arXiv:2404.01197.
- Deep reinforcement learning from human preferences.
- Think you have solved question answering? try arc, the ai2 reasoning challenge.
- Ultrafeedback: Boosting language models with high-quality feedback.
- Enhancing chat language models by scaling high-quality instructional conversations.
- Human-aware loss functions (halos). Technical report, Contextual AI.
- Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.
- Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.13639.
- Measuring massive multitask language understanding.
- Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691.
- Lora: Low-rank adaptation of large language models.
- Phi-2: The surprising power of small language models. Microsoft Research Blog.
- Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning.
- Truthfulqa: Measuring how models mimic human falsehoods.
- Statistical rejection sampling improves preference optimization.
- Alexander V. Lotov and Kaisa Miettinen. 2008. Visualizing the Pareto Frontier, pages 213–243. Springer Berlin Heidelberg, Berlin, Heidelberg.
- Kaisa Miettinen. 1999. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media.
- Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
- Efficient large-scale language model training on gpu clusters using megatron-lm.
- Training language models to follow instructions with human feedback.
- Red teaming language models with language models.
- Direct preference optimization: Your language model is secretly a reward model.
- Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.
- Insights into alignment: Evaluating dpo and its variants across multiple tasks. arXiv preprint arXiv:2404.14723.
- Winogrande: An adversarial winograd schema challenge at scale.
- Multitask prompted training enables zero-shot task generalization.
- Proximal policy optimization algorithms.
- Learning to summarize from human feedback.
- Llama: Open and efficient foundation language models.
- Zephyr: Direct distillation of lm alignment.
- Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297–323.
- Trl: Transformer reinforcement learning. https://github.com/huggingface/trl.
- Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment.
- Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.
- Rrhf: Rank responses to align language models with human feedback without tears.
- Hellaswag: Can a machine really finish your sentence?
- Slic-hf: Sequence likelihood calibration with human feedback.
- Fine-tuning language models from human preferences. ArXiv, abs/1909.08593.
- Amir Saeidi (8 papers)
- Shivanshu Verma (2 papers)
- Aswin RRV (5 papers)
- Chitta Baral (152 papers)