Programming Refusal with Conditional Activation Steering (2409.05907v3)

Abstract: LLMs have shown remarkable capabilities, but precisely controlling their response behavior remains challenging. Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants. In this paper, we propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context. Our method is based on the observation that different categories of prompts activate distinct patterns in the model's hidden states. Using CAST, one can systematically control LLM behavior with rules like "if input is about hate speech or adult content, then refuse" or "if input is not about legal advice, then refuse." This allows for selective modification of responses to specific content while maintaining normal responses to other content, all without requiring weight optimization. We release an open-source implementation of our framework at github.com/IBM/activation-steering .

• The paper introduces Conditional Activation Steering (CAST) to program LLMs for context-specific refusal of harmful content using condition and behavior vectors.
• It demonstrates enhanced control by dynamically modulating activation patterns, maintaining responsiveness for benign prompts while filtering harmful inputs.
• CAST's flexible mechanism enables logical composition of condition vectors, paving the way for specialized domain applications and improved model safety.

Conditional Activation Steering for Selective Behavioral Control in LLMs

The paper "Programming Refusal with Conditional Activation Steering" by Bruce W. Lee et al. introduces a novel methodology termed Conditional Activation Steering (CAST) to enhance the controllability of LLMs. This method selectively modulates the model’s behavior based on contextual inputs, offering a sophisticated alternative to traditional activation steering techniques.

Abstract

In essence, the proposed CAST method integrates principles from both activation steering and behavior control. By introducing condition vectors and combining them with behavior vectors, the framework allows for contextually aware interventions. This mechanism is particularly relevant for scenarios requiring specific responses, such as content moderation, where indiscriminate steering could undermine the utility of the model.

Introduction

LLMs have demonstrated impressive capabilities but lack adequate mechanisms for context-specific behavior modulation. The current activation steering techniques alter model behavior broadly without the ability to conditionally apply these changes based on input prompts. CAST addresses this by analyzing activation patterns within the model and conditionally applying behavior modifications.

Method

The core innovation of CAST lies in its ability to use condition vectors to check the context of the input. Upon receiving a prompt, the model evaluates its hidden state to determine alignment with predefined condition vectors. If the alignment surpasses a threshold, a corresponding behavior vector applies the desired modification, such as refusal of harmful content.

1. Behavior Vector: Extracted to induce specific responses, such as refusal behavior.
2. Condition Vector: Captures activation patterns corresponding to categories of interest (e.g., hate speech).
3. Similarity Calculation: A similarity measure between the hidden state and condition vector determines the application of the behavior vector.

The mathematical formulation ensures flexibility and specificity:

$$

Experimental Setup and Results

Conditioning Refusal on Harmful Prompts

The authors validated CAST by implementing refusal behavior selectively on harmful prompts, leveraging the Sorry-Bench dataset for harmful prompts and the Alpaca dataset for harmless ones. Figure 3 and Table 3 illustrate the paper's core finding: CAST effectively partitions the prompt space, refusing harmful inputs while maintaining responsiveness to benign ones.

Fine-grained Refusal Conditions

The methodology extends to more nuanced categories such as hate speech, legal opinions, and adult content. By conditioning on specific vectors, CAST could modulate refusal behavior granularity. Figures 5 and 6 demonstrate the model’s flexibility in selectively refusing or responding to prompts based on fine-grained conditional rules.

Logical Composition and Domain Constraining

One prominent feature of CAST is its ability to compose multiple condition vectors logically. For instance, combining vectors with an OR operation can enhance refusal behavior robustness against various harmful categories. Moreover, flipping the condition checks allows the model to constrain responses to specific domains, effectively transforming a general-purpose model into a specialized one.

Implications and Future Directions

CAST presents significant theoretical and practical implications:

• Alignment Efficiency: By leveraging model’s inherent activation representations, it offers a cost-effective behavioral alignment without extensive tuning.
• Fine-grained Control: The ability to compose and modulate behavioral rules introduces unprecedented specificity.
• Domain Specialization: CAST could tailor general models for specialized applications, enhancing their utility in sensitive domains like legal or medical advice.

Future exploration might delve into optimizing the extraction of condition and behavior vectors and expanding the range of controllable behaviors. This could include not only refusals but also enhancements like inducing empathy or formality in responses.

Conclusion

Overall, the integration of CAST into the framework of LLM control mechanisms brings a refined, context-aware layer of behavior modulation. This represents a methodological leap towards precise and programmable user-specific alignments, thereby substantially expanding the applicability of LLMs in diverse and specialized environments.

By harnessing the model’s internal activation patterns through conditional checks, the paper sets a foundation for more granular and predictable control of LLMs, contributing a promising direction for ongoing alignment and safety research efforts.

Acknowledgments

The authors acknowledge the support from mentors and colleagues at IBM Research, along with the availability of open-source models and datasets that facilitated this research. Their collaborative effort underscores the community's role in advancing AI alignment methodologies.