Safety-Aware Fine-Tuning of Large Language Models (2410.10014v1)

Abstract: Fine-tuning LLMs has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios.

Safety-Aware Fine-Tuning of LLMs

The paper "Safety-Aware Fine-Tuning of LLMs" addresses the critical concern of ensuring safety when fine-tuning LLMs on diverse datasets. The potential presence of harmful data in fine-tuning datasets poses significant challenges, both in terms of the manual effort required for data curation and the subjective nature of content evaluation.

Key Contributions

1. Safety-Aware Fine-Tuning (SAFT) Framework: The authors propose a novel SAFT framework designed to automatically identify and remove potentially harmful data samples. This is achieved by leveraging a scoring function that utilizes subspace information of harmful and benign samples within the LLM's embedding space.
2. Harmful Data Detection Mechanism: The paper introduces a filtering mechanism that exploits the internal representations of LLMs. By identifying a subspace associated with harmful content, the framework can distinguish between harmful and benign data effectively. A singular value decomposition approach is employed to extract meaningful directions in activation space that correlate with harmfulness.
3. Empirical Efficacy: The proposed approach demonstrates significant reductions in harmfulness—up to 27.8%—across diverse LLMs and contamination levels. This result underscores the framework's robustness and its ability to maintain overall model performance on benign tasks.
4. Generalizability and Steerability: The framework showcases versatility, addressing different practical challenges like varying data distributions. Furthermore, it provides steerability, enabling users to adjust the level of filtering based on specific needs.

Experimental Insights

The authors conduct extensive experiments using Llama-2 and Vicuna models, demonstrating that SAFT effectively reduces harmfulness without compromising the helpfulness of the model. The framework outperforms various baselines, including naive supervised fine-tuning and random filtering, highlighting its precision in identifying harmful data.

Theoretical and Practical Implications

Theoretically, the paper introduces a compelling mechanism for exploring the internal embeddings of LLMs, advancing the understanding of harmful data representation. Practically, it provides a scalable and automatic solution for safety-aware model customization, reducing reliance on manual data filtering.

Future Directions

Potential future developments could include extending the framework to incorporate additional safety metrics, investigating its application to other domains, and integrating with alignment techniques like RLHF. There is also scope to explore further how embeddings could be leveraged to refine harmfulness detection without explicit labeling.

In conclusion, the "Safety-Aware Fine-Tuning of LLMs" paper offers a significant contribution to the field by proposing a structured approach to mitigate the risks associated with harmful data during the fine-tuning process. This framework enhances the safety and reliability of fine-tuned models, making it a promising solution for practical deployment in diverse applications.