Suppressing Pink Elephants with Direct Principle Feedback (2402.07896v2)

Abstract: Existing methods for controlling LLMs, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a LLM. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a Pink Elephant''), and instead discuss a preferred entity (Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.

• The paper introduces a novel methodology using Direct Principle Feedback (DPF) to suppress undesired topics in language model outputs.
• It leverages synthetic preference data and dialogue revisions for fine-tuning, significantly reducing pink elephant mentions.
• Experimental results on a fine-tuned LLaMA 2 model demonstrate that DPF outperforms traditional methods in enforcing behavioral constraints.

Suppressing Pink Elephants with Direct Principle Feedback: An Approach for Inference-Time Controllability of LLMs

Introduction

LLMs (LMs), particularly LLMs, have seen significant advancements, fuelled by extensive training on broad, unlabelled datasets. Despite their growing capabilities, instilling desirable behaviors in these models while ensuring controllability at inference time remains a challenge. This paper presents a novel approach to address one specific aspect of this challenge, known as the Pink Elephant Problem, where models are instructed to avoid discussing a specified topic (the "Pink Elephant") and focus on an alternative subject (the "Grey Elephant"). Through the use of Direct Principle Feedback (DPF), the authors propose a methodology for controlling LLM output without the need for explicit hard-coding of desired values or behaviors into the model.

Direct Principle Feedback

DPF simplifies the Reinforcement Learning from AI Feedback (RLAIF) framework by focusing on the critique and revision stages, using these as natural language feedback for preference-based fine-tuning. In this setup, responses and their revisions create a paired dataset that illustrates the desired behavior shift, specifically avoiding the mention of the Pink Elephant. This starkly contrasts with other methods, which rely on ranking responses according to predetermined criteria, demonstrating the potential for DPF to streamline the process significantly.

Dataset Generation

The generation of synthetic preference data played a crucial role in this study, with the authors meticulously curating a dataset that represents a wide array of "Pink Elephant" scenarios. This process involved generating contrasting entity pairs across various domains and artificially creating dialogues where the model would initially mention the Pink Elephant, followed by a revised dialogue where the mention is omitted. This expertly crafted dataset underscores the intricate planning and consideration needed to simulate realistic conversational settings where a model might inadvertently discuss a Pink Elephant.

Experimental Results

The application of DPF to a fine-tuned LLaMA 2 model yielded promising results, with the modified model showing a significant reduction in Pink Elephant mentions when instructed to avoid them, outperforming both unmodified baseline models and those adapted through alternative methods. These findings suggest that DPF not only enhances a model's ability to comply with specific behavioral constraints at inference time but does so in a manner that is practical and scalable.

Implications and Future Directions

The implications of this research extend beyond the immediate results. The success of DPF in enforcing dynamic content avoidance demonstrates a viable path toward achieving greater controllability over LLM outputs, potentially revolutionizing how models can be customized to adhere to diverse user needs and regulatory requirements. Looking ahead, the exploration of this technique's applicability to more complex topics, as well as its integration with other behavioral correction methods, could offer new insights into effective model training and refinement strategies.

Ethical Considerations and Limitations

The paper also thoughtfully addresses the ethical concerns and limitations inherent in implementing and deploying such methodologies. The potential for misuse, particularly in implementing censorship or the propagation of biases, underscores the need for careful consideration and oversight in the application of DPF and similar technologies. Furthermore, the authors recognize the current method's limitations, specifically its reliance on synthetic data and controlled experimental conditions, suggesting areas for further research and improvement.

Conclusion

In summary, the study presents a compelling case for the use of Direct Principle Feedback as a means of enhancing inferential controllability in LLMs, particularly in the context of avoiding undesired topics. By successfully navigating the intricacies of the Pink Elephant Problem, the authors not only contribute a valuable tool to the AI research community but also pave the way for more adaptable, ethical, and effective use of LLMs in various real-world applications.