RLVF: Learning from Verbal Feedback without Overgeneralization (2402.10893v1)

Abstract: The diversity of contexts in which LLMs are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a model with such feedback leads to overgeneralization of the feedback to contexts where it is not relevant. We study the problem of incorporating verbal feedback without such overgeneralization, inspiring a new method Contextualized Critiques with Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level feedback to generate a small synthetic preference dataset specifying how the feedback should (and should not) be applied. It then fine-tunes the model in accordance with the synthetic preference data while minimizing the divergence from the original model for prompts where the feedback does not apply. Our experimental results indicate that our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts. For both human- and GPT-4-generated high-level feedback, C3PO effectively adheres to the given feedback comparably to in-context baselines while reducing overgeneralization by 30%.

• The paper introduces C3PO, a method that tailors LLMs to comply with specific verbal feedback while reducing overgeneralization.
• It employs synthetic preference data and context-aware prompts to balance in-scope feedback adherence with baseline behavior preservation.
• Results show a 30% reduction in overgeneralization and over 10% improvement compared to baselines, underscoring its practical impact.

RLVF: Tailoring LLMs to Specific Feedback without Overgeneralization

Introduction

The deployment of LLMs across various domains has necessitated the development of methods to customize these models according to specific user preferences and requirements. A notable challenge in this endeavor is the integration of high-level verbal feedback into LLMs without inducing overgeneralization—where the model inappropriately applies the feedback beyond the intended contexts. The paper introduces a novel method, Contextualized Critiques with Constrained Preference Optimization (C3PO), designed to adapt LLMs to verbal feedback efficiently while minimizing overgeneralization.

Methodology

C3PO operates by generating synthetic preference data, demonstrating both the desired model behavior in response to feedback and the behavior to be retained in contexts unrelated to the feedback. This process involves initially using a high-capacity model like GPT-4 to produce potential contextual categories relevant to the feedback. Following this, in-scope and out-of-scope prompts are created, with the LLM generating responses that are either aligned or not aligned with the feedback for these prompts. The model is then fine-tuned on this synthetic dataset, optimizing towards feedback adherence in relevant contexts while discouraging deviation from baseline behavior in other contexts.

Experimental Results

The effectiveness of C3PO is evaluated through its ability to adhere to feedback in relevant contexts and its propensity to maintain default behavior otherwise. The proposed method is compared against several baselines, including in-context learning and supervised context distillation (SCD). C3PO achieved a commendable balance, effectively applying feedback with a 30% reduction in overgeneralization, outperforming the baseline methods by over 10% in metrics that consider both in-scope feedback adherence and minimization of out-of-scope behavior alteration.

Implications and Future Directions

The C3PO method presents a significant advancement in the field of LLM customization, offering a practical solution to the challenge of feedback-induced overgeneralization. Practical implications of this research include enhanced personalization of LLMs in consumer applications, improved efficiency in specialized professional settings, and the potential for more nuanced human-AI interaction. Future research directions could explore the continual adaptation of LLMs to multiple pieces of feedback, further refine the balance between feedback adherence and baseline behavior preservation, and investigate the application of C3PO in more diverse and complex feedback scenarios.

Conclusion

The RLVF paper introduces an innovative approach to incorporating high-level verbal feedback into LLMs without unwanted overgeneralization across different contexts. By harnessing the C3PO method, it is possible to customize LLM outputs to align with specific user preferences or requirements, marking a step forward in the effective and personalized deployment of AI technologies.