A General Theoretical Paradigm to Understand Learning from Human Preferences (2310.12036v2)

Abstract: The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of $\Psi$PO) and to identify their potential pitfalls. We then consider another special case for $\Psi$PO by setting $\Psi$ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

A General Theoretical Paradigm to Understand Learning from Human Preferences

The paper presents a comprehensive theoretical framework for understanding practical algorithms that learn from human preferences, specifically in the context of reinforcement learning from human feedback (RLHF). This research addresses two key approximations in RLHF: the conversion of pairwise preferences into pointwise rewards and the reliance on a reward model for generalization beyond collected data. The authors propose a new objective, dubbed Preference Optimisation (PO), which directly leverages pairwise preferences, thereby bypassing the previous approximations.

Key Contributions

This work makes several notable contributions to the existing literature:

1. Preference Optimisation (PO): The authors introduce a general learning objective expressed through pairwise preferences, providing a theoretical underpinning to RLHF that aligns with practical methodologies such as Direct Preference Optimisation (DPO). The PO framework enables a deeper exploration of the theoretical properties of these algorithms and aids in understanding their operational nuances.
2. Identity Preference Optimisation (IPO): By setting the mapping function $\Psi$ to identity, the paper derives a novel optimization procedure. IPO is demonstrated to be efficient, theoretically sound, and empirically superior to DPO in certain scenarios, addressing overfitting issues that arise when preferences become overly deterministic.
3. Theoretical Insights on RLHF and DPO: Through the lens of the newly proposed PO framework, the paper identifies potential pitfalls of RLHF and DPO, particularly the vulnerability to overfitting due to deterministic preferences and the assumptions required to substitute pairwise preferences with pointwise rewards using Bradley-Terry models.

Empirical Validation and Theoretical Implications

The authors support their theoretical claims with empirical examples, illustrating cases where DPO can fail by overfitting to deterministic preferences. In contrast, IPO, due to its formulation, maintains robustness and aligns more closely with the reference policy when faced with deterministic or nearly deterministic preference data. This empirical evidence solidifies the theoretical predictions and showcases the practical utility of the IPO approach.

The theoretical implications of this research are significant. By generalizing the understanding of learning from human preferences, this paper lays a foundation for developing more robust and versatile algorithms that can handle a wider array of preference datasets. The ability to learn directly from pairwise preferences without requiring a conversion to reward models opens new avenues for designing algorithms that are both simpler to implement and resource-efficient.

Future Directions

Future research directions could involve scaling the experiments to more complex environments, such as applying IPO to large-scale LLMs aligned with human feedback. Such exploration would provide deeper insights into the scalability and adaptability of the proposed framework in real-world applications. Additionally, integrating adaptive mechanisms to dynamically adjust the regularization parameter $\tau$ could further enhance the empirical performance and resilience of the PO framework.

In conclusion, this paper provides a strong theoretical and empirical foundation for understanding and improving algorithms that learn from human preferences, contributing valuable insights into the evolving landscape of AI and machine learning.