Course-Correction: Safety Alignment Using Synthetic Preferences

Abstract: The risk of harmful content generated by LLMs becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of \textbf{course-correction}, \ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the \textsc{C$^2$-Eval} benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create \textsc{C$^2$-Syn}, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, \textsc{Llama2-Chat 7B} and \textsc{Qwen2 7B}, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.

• The paper introduces a novel benchmark (C2-Eval) and synthetic dataset (C2-Syn) to quantitatively assess and enhance LLM course-correction abilities.
• The paper evaluates ten popular LLMs and reveals significant disparities in self-correction, with some models excelling while others lag behind.
• The paper demonstrates that fine-tuning with synthetic preferences improves safety alignment without compromising performance on standard benchmarks and resisting jailbreak attacks.

Course-Correction: Safety Alignment Using Synthetic Preferences

In this paper, the authors address the critical concern of harmful content generated by LLMs. Specifically, they focus on the concept of course-correction, which refers to an LLM's ability to autonomously steer away from harmful content after initially generating it. The work presents a comprehensive study on both evaluating and enhancing this ability in popular LLMs.

C$^2$-Eval Benchmark

The authors introduce C$^2$-Eval, a benchmark specifically designed to evaluate the course-correction capabilities of LLMs. Using C$^2$-Eval, they conducted experiments to quantitatively assess 10 popular LLMs, including safety-tuned versions such as Llama2-Chat and Vicuna v1.5.

The evaluation reveals substantial disparities among the models, with some showing notable proficiency in course-correction and others performing poorly. Notably, models like Llama3-Instruct achieve high scores in the ability to self-correct harmful outputs, whereas Vicuna v1.5 demonstrates significantly lower scores. This polarization highlights the varying effectiveness of current safety-tuning methodologies.

C$^2$-Syn: Synthetic Dataset with Data-Driven Preference Learning

To improve course-correction capabilities, the authors propose fine-tuning LLMs using preference learning based on synthetic data. They introduce C$^2$-Syn, a synthetic dataset comprising 750K pairwise preferences aimed at teaching LLMs the value of timely course-correction.

The synthetic dataset is constructed using a novel automated pipeline. The process involves generating harmful responses from the PKU-SafeRLHF dataset, followed by creating corrective responses using a well-aligned LLM, Llama2-Chat 7B in this case. Human evaluators then confirm the effectiveness of these corrective responses, achieving a success rate of 98%. This verifies the applicability of synthetic data in training LLMs for course-correction.

Experimental Results

The authors fine-tuned two LLMs, Llama2-Chat 7B and Qwen2 7B, using the synthetic dataset and the Direct Preference Optimization (DPO) algorithm. The results are promising:

• The course-correction ability improved significantly, as evidenced by enhanced scores on the C$^2$-Eval benchmarks.
• The safety alignment improved without negatively impacting general performance metrics across various established benchmarks like MMLU, TruthfulQA, and GSM8K.
• Fine-tuned models exhibited greater resilience against four prevalent jailbreak attacks, such as GCG and PAIR, demonstrating lower attack success rates.

Theoretical and Practical Implications

The findings have several key implications for the field of AI safety:

1. Effectiveness of Synthetic Data: The success of C$^2$-Syn demonstrates the viability of using synthetic datasets for preference learning in safety alignment tasks. This approach can reduce the reliance on extensive human labor for data labeling, addressing ethical and practical concerns associated with human annotators handling harmful content.
2. Algorithm Choice in Preference Learning: While the DPO algorithm was chosen for stability and lower memory footprint, the study opens avenues for exploring other alignment algorithms that could further enhance course-correction capabilities.
3. Insights into Safety Tuning: The disparity in course-correction abilities among models highlights the need for more targeted safety-tuning. Models still prone to delayed corrections indicate that current training regimes might prioritize initial response safety, neglecting mid-sequence corrections.

Speculation on Future Developments

The research provides a robust foundation for future exploration in AI safety and alignment. Potential future directions include:

• Dynamic Evaluation Protocols: Implementing a dynamic evaluation strategy, where harmful responses are tailored to individual models rather than using a static dataset, may offer more accurate assessments.
• Enhanced Alignment Techniques: Developing hybrid models combining RLHF with preference learning on synthetic data could offer more comprehensive safety measures.
• Broader Application of Synthetic Data: Extending the use of synthetic datasets to other domains within AI safety, such as misinformation detection and bias mitigation, could prove beneficial.

Conclusion

Overall, the paper offers valuable insights and practical methodologies for enhancing the safety of LLMs through course-correction. By introducing C$^2$-Eval and C$^2$-Syn, the authors provide both a benchmark for evaluating current models and a new dataset for improving their safety alignment. This research marks a significant step forward in developing more responsible and secure AI systems.