A Minimaximalist Approach to Reinforcement Learning from Human Feedback (2401.04056v2)

Abstract: We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction. To achieve the preceding qualities, we build upon the concept of a Minimax Winner (MW), a notion of preference aggregation from the social choice theory literature that frames learning from preferences as a zero-sum game between two policies. By leveraging the symmetry of this game, we prove that rather than using the traditional technique of dueling two policies to compute the MW, we can simply have a single agent play against itself while maintaining strong convergence guarantees. Practically, this corresponds to sampling multiple trajectories from a policy, asking a preference or teacher model to compare them, and then using the proportion of wins as the reward for a particular trajectory. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches while maintaining robustness to the intransitive and stochastic preferences that frequently occur in practice when aggregating human judgments.

• The paper introduces Self-Play Preference Optimization (SPO) which replaces reward models with an efficient self-play mechanism.
• It leverages a zero-sum game formulation to optimize for Nash equilibrium, ensuring consistent performance despite non-Markovian and noisy human feedback.
• Empirical tests reveal that SPO outperforms traditional reward-based RLHF methods in continuous control and complex preference scenarios.

Introduction

Reinforcement Learning from Human Feedback (RLHF) is a method of optimizing AI behavior based on qualitative judgments rather than relying on predefined, quantitative reward systems. This technique is especially useful when crafting an explicit reward function is challenging or when the task requires a degree of subjective judgment that is difficult to quantify. RLHF has been applied successfully in various fields, from robotics to recommendation systems and even for fine-tuning LLMs. However, the traditional approach to RLHF, based on Reward-based Reinforcement Learning from Human Feedback (reward-based RLHF), involves the creation and optimization of a reward model derived from paired behaviors and their human-given preferences. This method has limitations which include challenges with non-transitive preferences—scenarios where the constructed reward function cannot capture the complexity of human choice—and a vulnerability to stochastic human decisions.

A New Algorithmic Approach

In this context, Self-Play Preference Optimization (SPO) is introduced, a streamlined and theoretically justified algorithm designed to surmount these issues. By removing the need for a reward model and leveraging the symmetry of zero-sum game formulation, SPO simplifies the RLHF process while broadening its capabilities, demonstrating robustness against the non-Markovian, noisy, and non-transitive complications often accompanying human feedback.

Theoretical Foundations and Practical Implementation

SPO builds on the concept of a Minimax Winner (MW), which aggregates preferences by considering the choices as a two-player zero-sum game. The innovative twist SPO brings to the table is the proof that optimizing for this game's Nash equilibrium—a close approximation of the MW—does not require two different adversarial policies, but rather a single agent can engage in self-play while retaining the desired mathematical guarantees. Concretely, this means generating multiple trajectories from the agent, asking a human or a model to compare them, and then rewarding each trajectory based on its comparative successes or "wins."

From a practical standpoint, this translates into implementing SPO using common reinforcement algorithms like Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC). One key adaptation to enable this is evenly dispersing trajectory-level rewards among all state-action pairs—a solution that aligns with the principle of potential-based reward shaping recognized in reinforcement learning literature.

Empirical Validation

The SPO approach has been tested across various environments and preference structures, including continuous control tasks with different types of preferences—such as trajectory-level comparisons against a ground-truth, non-Markovian preferences, and intransitive preferences. The results have shown that SPO can learn significantly more efficiently and more reliably than reward-model based approaches, particularly in scenarios that mimic real-world situations involving complex human judgments.

Addressing Typical RLHF Concerns

SPO also lessens concerns with compounding errors, which occur when the agent encounters states during execution that were not represented in the training data, leading to a cascade of incorrect actions. As SPO utilizes online preference queries that maintain up-to-date agent performance evaluation, the learning process is iterative and the agent continuously adapts, addressing the compounding error problem seen in offline RLHF approaches.

Conclusion

In summary, SPO proposes an elegant and empirically powerful alternative to existing RLHF methods. By eschewing the need for unstable components like reward models or adversarial dynamics, it presents a method that is simultaneously straightforward in implementation and expansive in its capacity to accommodate the quirks of human-derived feedback. This opens new possibilities for enhancing AI systems that need to model and work alongside inherently unpredictable human preferences and decisions.