The effect of fine-tuning on language model toxicity (2410.15821v1)

Abstract: Fine-tuning LLMs has become increasingly popular following the proliferation of open models and improvements in cost-effective parameter efficient fine-tuning. However, fine-tuning can influence model properties such as safety. We assess how fine-tuning can impact different open models' propensity to output toxic content. We assess the impacts of fine-tuning Gemma, Llama, and Phi models on toxicity through three experiments. We compare how toxicity is reduced by model developers during instruction-tuning. We show that small amounts of parameter-efficient fine-tuning on developer-tuned models via low-rank adaptation on a non-adversarial dataset can significantly alter these results across models. Finally, we highlight the impact of this in the wild, demonstrating how toxicity rates of models fine-tuned by community contributors can deviate in hard-to-predict ways.

• The paper finds that instruction-tuned models consistently exhibit lower toxicity compared to their base versions.
• The study demonstrates that additional parameter-efficient fine-tuning with innocuous data can unexpectedly increase toxicity, notably in Gemma models.
• The analysis reveals that community-tuned variants show unpredictable toxicity changes, underscoring the need for rigorous post-tuning safety evaluations.

Effect of Fine-Tuning on LLM Toxicity: An Analysis

The presented paper provides a detailed analysis of the impact that fine-tuning has on the propensity of open LLMs to produce toxic content. Through methodical experimentation, the authors explore the implications of fine-tuning on several open-source models, specifically Gemma, Llama, and Phi. The focus is on understanding how fine-tuning, particularly through parameter-efficient techniques, can inadvertently alter the safety characteristics of these models.

Overview of Experiments

The paper is structured around three distinct experiments to evaluate the consequences of fine-tuning:

1. Instruction-Tuned Models: Evaluates how instruction-tuning by model developers affects toxicity.
2. Parameter-Efficient Fine-Tuning: Assesses effects of additional fine-tuning using non-adversarial datasets.
3. Community-Tuned Variants: Analyzes popular community-tuned models to uncover potential deviations in toxicity rates.

The models tested include various versions of Llama, Gemma, and Phi, selected for their prominence and accessibility within the AI community.

Methodology

The paper utilizes a robust methodology incorporating multiple datasets to test toxicity, including RealToxicityPrompts and the Compositional Evaluation Benchmark (CEB). Toxicity metrics are evaluated using the roberta-hate-speech-dynabench-r4 model from Hugging Face. Bayesian estimation methods are employed to validate differences in toxicity outputs across models, avoiding issues endemic to traditional significance testing.

Key Findings

1. Instruction-Tuning Effects: Instruction-tuned models consistently demonstrated reduced toxicity levels compared to their base counterparts. This indicates the model creators’ successful efforts to mitigate toxic outputs.
2. Subsequent Fine-Tuning: Despite initial successful tuning, further fine-tuning using seemingly innocuous data, such as the Dolly dataset, led to increased toxic outputs in most models. Notably, Gemma models experienced substantial increases in toxicity, highlighting the brittleness of initial mitigations.
3. Community Variants: The propensity for toxicity varied unpredictably among community-tuned models. While some models, like Llama-2-7B chat_uncensored, exhibited higher toxicity, others displayed minimal changes despite similar tuning intentions.

Implications and Speculation

The paper's findings underscore the challenges and unpredictability associated with fine-tuning open LLMs. The nuanced variations in toxicity following further community-driven tuning suggest that developers and users cannot assume stability of safety characteristics post-tuning.

Practical Implications: Developers should institute comprehensive evaluation procedures following any fine-tuning process to ensure toxicity and safety standards are maintained. Furthermore, transparency in fine-tuning practices is pivotal for users who depend on these models for application development.

Theoretical Implications: The paper opens avenues for exploring the causes behind variable toxicity outputs. Factors such as catastrophic forgetting and data-derived semantic shifts require further investigation to develop more robust fine-tuning methodologies.

Future Directions

Engaging in future research that expands on model size variations, fine-tuning techniques beyond LoRA, and the impact on broader ethical concerns, such as fairness and bias, will yield valuable insights. Understanding the underlying mechanics of catastrophic forgetting and toxicity convergence will be crucial in refining the stability of model safety features post-tuning.

In conclusion, this paper delivers critical observations on the effects of fine-tuning on open LLM toxicity, emphasizing a necessity for rigorous evaluation in AI model deployment and development practices.