A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (2401.01967v1)

Abstract: While alignment algorithms are now commonly used to tune pre-trained LLMs towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained LLM, GPT2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.

• The paper shows that DPO reduces toxicity by modulating specific toxic vectors in GPT2's MLP blocks.
• The study employs linear probes and singular value decomposition to reveal hidden toxic components.
• The findings highlight that minimal parameter changes allow easy reversion to toxicity, questioning alignment robustness.

Overview of Alignment Algorithms

Aligning LLMs to align with user preferences and avoid issues such as toxicity is an area of growing importance. However, there has been limited clarity about the mechanisms by which alignment algorithms achieve this. A paper focused on the inner workings of these algorithms provides new insights, particularly on Direct Preference Optimization (DPO). DPO works by directly using pairwise preference data to train models to generate more preferred outputs, but how it suppresses unwanted behaviors like toxicity has been something of a mystery.

Investigating Toxicity

The investigation began with a pre-trained GPT2-medium model, aiming to understand how it represents and produces toxic language. Using a variety of techniques, including linear probes and singular value decomposition, the researchers identified specific vectors within the model's multilayer perceptron (MLP) blocks that are associated with toxic outputs. Interventions were made by adjusting these vectors, resulting in reduced toxicity without substantially degrading the quality of language generation, demonstrating the influence these toxic vectors have on model outputs.

Applying Direct Preference Optimization

Employing DPO required creating a dataset of paired toxic and nontoxic text samples, which was used to realign the model's behavior. Post alignment, the evaluation showed that while toxicity dropped, there had been only minimal shifts in the model's parameters, including the toxic vectors identified earlier. The model had learned to avoid triggering the identified toxic vectors, rather than eradicating the ability to generate toxic language entirely.

The Implications of Minimal Parameter Changes

This behavior of GPT2 following DPO, where the model simply bypasses the regions that elicit toxicity rather than removing the capability, suggests a reason why aligned models can often be easily unaligned or jailbroken. The vectors that promote toxicity are not removed but are dormant and can be easily reactivated. The paper went as far as demonstrating a method to revert the model back to its originally toxic behavior, effectively 'un-aligning' it.

Conclusion and Impact

Understanding the mechanistic details behind alignment algorithms like DPO highlights both their strengths and weaknesses. Although they can effectively suppress unwanted behaviors in the short term with only minor changes, these behaviors can be quickly rekindled, raising questions about the robustness of current alignment methods. The insights from this paper could pave the way for developing more robust alignment strategies, potentially leading to better-behaved AI systems that maintain their alignment even when challenged by novel inputs or adversarial actors.