Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (2408.10075v1)

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

The paper presents a novel approach to address critical limitations in traditional methods for Reinforcement Learning from Human Feedback (RLHF). Traditionally, RLHF frameworks operate under the assumption that the preferences of all users can be encapsulated within a single, unimodal utility function. This assumption, however, fails to account for the natural diversity in human preferences across a heterogeneous population. This oversight results in suboptimal reward functions and policies that do not adequately serve diverse user subgroups. The paper introduces Variational Preference Learning (VPL), a mechanism designed to model and address this diversity by capturing multi-modal user preferences through variational inference.

Reward Modeling with Latent Variables

The cornerstone of VPL is its use of a latent variable model to capture the hidden context that influences individual user preferences. This is framed within a variational inference framework, where a variational encoder infers a latent distribution representing user-specific contexts. This latent representation is then used to condition reward models, enabling the capture of multi-modal distributions over user preferences. The evidence lower bound (ELBO) for the marginal likelihood of the observed preferences forms the foundation of the learning objective. This allows the reward model to account for the diverse and potentially conflicting preferences manifest across different users.

Practical Algorithmic Considerations

While the theoretical formulation of VPL appears straightforward, the practical implementation necessitates several critical algorithmic considerations. A key challenge is the intrinsic scale variability in rewards derived from binary comparisons. Binary comparisons only provide ordinal information, i.e., which item is preferred over another, but lack absolute scale information. This can lead to significant discrepancies in reward scales across different users' latent variables, complicating policy optimization.

To mitigate this, the paper proposes using a pairwise classification scheme based on Self-Play Preference Optimization (SPO). By computing the expected likelihood of preference based on the latent distribution, rewards are normalized, ensuring consistency in scaling across different contexts. This approach mitigates the adverse effects of varying reward scales, thus enhancing downstream policy performance.

Experimental Validation

The efficacy of VPL is empirically validated across various simulated control tasks and LLM alignment experiments. In simulated environments such as Maze-Navigation and Ravens-Manipulation, VPL outperforms conventional RLHF methods, providing better task performance by accurately inferring user-specific reward functions. This is evidenced by higher success rates in navigation tasks and more accurate goal-directed behaviors in manipulation tasks.

Scaling to LLMs

To further substantiate the applicability of VPL, the paper describes its integration with LLMs, considering datasets with divergent user preferences. By leveraging pre-trained LLM embeddings, the latent-conditional reward model can be scaled efficiently. Experiments on datasets like UltraFeedback-P demonstrate significant improvements in reward accuracy, highlighting VPL's ability to model complex, multi-modal preference distributions in large-scale LLMs. The robustness of VPL is showcased through its superior performance, even in the presence of noisy labels.

Implications and Future Directions

The development and validation of VPL have substantial implications for both theoretical and practical advancements in AI and RLHF. Theoretically, the introduction of a latent variable model into preference learning addresses the fundamental issue of reward mis-specification due to aggregated diverse preferences. Practically, this approach enables the creation of more inclusive, personalized AI systems that can adapt to a broader range of user preferences, thereby enhancing user satisfaction and system performance.

Future work could explore the nuances of employing VPL in real-world deployments, particularly focusing on the trade-offs between preference personalization and adherence to universal ethical standards. Additionally, extending VPL to incorporate more granular, continuous feedback rather than binary preferences could further refine the alignment of AI systems with human users.

In conclusion, this paper significantly advances the state-of-the-art in RLHF by introducing VPL, a framework that effectively captures and leverages the diversity in human preferences, thus promoting the development of more flexible and personalized AI models.