Generalized Preference Optimization: A Unified Approach to Offline Alignment (2402.05749v2)

Abstract: Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.

• The paper presents GPO as a unified framework that recovers diverse offline preference algorithms, advancing model alignment.
• It utilizes convex functions to parameterize preference losses, clarifying the role of regularization compared to KL divergence.
• Empirical evaluations, including language model tasks, confirm that optimal hyper-parameter tuning yields similar performance across GPO variants.

Overview of Generalized Preference Optimization (GPO)

In the field of offline preference optimization, the proposed generalized preference optimization (GPO) stands as a significant advancement, providing a unified framework that encompasses a broad array of existing algorithms. This paper introduces GPO as an innovative approach to fine-tuning large models using offline datasets, representing a leap forward in alignment practices within AI systems.

Key Contributions of GPO

Unification of Offline Preference Optimization Algorithms

One of the paper's main contributions is the introduction of GPO, which not only highlights the connections between well-known algorithms but also proposes a method for developing new variants. By parameterizing preference optimization losses through a family of convex functions, GPO successfully recovers existing algorithms such as DPO, IPO, and SLiC, casting them as special cases within its broader framework. This approach not only clarifies the landscape of offline preference optimization but also opens avenues for future algorithmic developments.

Insights into Offline Regularization and KL Divergence

The paper explores how offline algorithms enforce regularization, focusing on the role of the convex function defining the loss. A thorough analysis of the tail behavior of these convex functions uncovers the intricacies of regularization strength and its impact on alignment practices. Moreover, the examination of offline regularization uncovers notable differences from the KL divergence, adding a layer of complexity to our understanding of how these algorithms operate.

Empirical Evaluation

Empirical results play a crucial role in validating the theoretical claims made by the researchers. Through extensive experiments, including a LLM summarization task, GPO's versatility is put to the test. These experiments underscore the importance of selecting appropriate hyper-parameters and highlight the similar performance observed across different GPO variants when the right conditions are met.

Conclusion and Outlook

Generalized Preference Optimization (GPO) represents a substantial step forward in the field of offline preference optimization. By providing a unified framework that not only encompasses existing algorithms but also paves the way for the development of new ones, GPO offers fresh perspectives on regularization mechanisms and their implications for model alignment. The empirical insights and algorithmic toolkits presented in this paper are poised to significantly impact future research and practices in aligning AI systems with human values and preferences.