Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards (2402.18571v3)

Abstract: Fine-grained control over LLMs remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

Directional Preference Alignment for Fine-Grained Control over LLMs

The paper "Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards," addresses the challenge of aligning LLMs with diverse user preferences using a novel framework called Directional Preference Alignment (DPA). This research is situated in the context of using Reinforcement Learning from Human Feedback (RLHF) to align LLMs, which typically relies on scalar rewards, often failing to capture the complexity of varied human preferences.

Key Concepts and Framework

The paper introduces DPA as an innovative framework aiming to bring fine-grained control over LLMs by considering multi-objective reward modeling. Unlike traditional approaches that utilize scalar-reward RLHF and enforce a single, averaged preference, DPA leverages a directional model of user preferences. Preferences are represented as unit vectors in a multi-objective reward space, allowing for diverse trade-offs and more personalized user experiences.

The DPA framework involves two stages: training a multi-objective reward model and fine-tuning the LLM with a preference-conditioned adaptation of Rejection Sampling Finetuning (RSF) - a method adopted by recent models like Llama 2. This process allows users to arithmetically specify the balance they desire between different objectives, such as helpfulness and verbosity, thereby offering more intuitive control over LLM outputs.

Experimental Validation

To showcase the effectiveness of the proposed model, the authors validate DPA using Mistral-7B, a state-of-the-art LLM. The experiments reveal that DPA more effectively captures and aligns with user-specific preferences than existing scalar-based RLHF methods. For example, DPA facilitates trade-offs such as generating less verbose responses while maintaining helpfulness, something not achievable with traditional methods like Direct Preference Optimization (DPO). The combination of personalized control and multi-objective considerations positions DPA to make significant contributions in personalizing LLM interactions.

Implications and Future Directions

The implications of this research are twofold: practically, DPA enhances LLMs' ability to adapt to diverse user preferences, thus improving user satisfaction in human-AI interaction. Theoretically, DPA offers a novel approach to reward modeling by moving from scalar to directional vectors, fostering a richer representation of complex human preferences.

Looking to the future, challenges remain in optimizing DPA’s performance across different domains and models. Further research could explore the scalarization strategies in high-dimensional preference vectors and how these adjustments impact long-term model alignment and performance. Additionally, improvements in directional preference learning could aid in mitigating biases that are prevalent in current LLMs.

In conclusion, this paper offers a significant advancement in aligning LLMs with user preferences by proposing a robust multi-objective alignment framework. The introduction of directional preference vectors in reward modeling and preference-conditioned adaptation marks a pivotal transformation in enhancing the adaptability and personalization of LLMs in real-world applications.