ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding

Abstract: With the development of instruction-tuned LLMs, improving the safety of LLMs has become more critical. However, the current approaches for aligning the LLMs output with expected safety usually require substantial training efforts, e.g., high-quality safety data and expensive computational resources, which are costly and inefficient. To this end, we present reverse prompt contrastive decoding (ROSE), a simple-yet-effective method to directly boost the safety of existing instruction-tuned LLMs without any additional training. The principle of ROSE is to improve the probability of desired safe output via suppressing the undesired output induced by the carefully-designed reverse prompts. Experiments on 6 safety and 2 general-purpose tasks show that, our ROSE not only brings consistent and significant safety improvements (up to +13.8% safety score) upon 5 types of instruction-tuned LLMs, but also benefits the general-purpose ability of LLMs. In-depth analyses explore the underlying mechanism of ROSE, and reveal when and where to use it.

• The paper demonstrates that Reverse Prompt Contrastive Decoding (Rose) enhances LLM safety by suppressing undesired outputs during inference.
• The method leverages reverse prompts to contrastively diminish harmful responses, achieving up to +13.98% improvement in safety scores.
• Experimental analyses on models like Alpaca and Vicuna show that Rose outperforms self-correction and traditional contrastive decoding techniques.

Reverse Prompt Contrastive Decoding for Enhancing LLM Safety

The paper "Rose Doesn't Do That: Boosting the Safety of Instruction-Tuned LLMs with Reverse Prompt Contrastive Decoding" presents an innovative approach to enhance the safety of LLMs during inference without additional training. Recognizing the challenges and inefficiencies associated with current training-intensive safety alignment methods, the authors propose Reverse Prompt Contrastive Decoding (Rose), a novel inference-time method. Rose is designed to improve the likelihood of generating safe outputs by utilizing reverse prompts to suppress undesirable responses, thereby encouraging LLMs to produce more secure outputs.

Methodology Overview

The core methodology of Rose involves a contrastive decoding technique where the desired output's probability is enhanced by diminishing the probability of undesired outputs. The undesired outputs are elicited using strategically crafted reverse prompts. Rose employs different formulations of negative or reverse prompts, such as replacing key positive words with their negative counterparts or completely reframing the prompt to incite harmful responses. The method seeks to capitalize on the anchoring effect, where the model's behavior is significantly influenced by the provided system prompts during inference.

Experimental Analysis

Extensive experimentation is conducted on several instruction-tuned LLMs such as Alpaca and Vicuna, as well as RLHF-aligned models like InternLM and Qwen. These models were evaluated across diverse safety and general-purpose tasks including SafetyBench, CValues, HarmfulQA, and AlpacaEval, among others. The results demonstrate that Rose consistently improves the safety performance across different LLM architectures with enhancements up to +13.98% in safety scores. Notably, Rose not only advances safety performance but also enhances the general-purpose capability of LLMs, indicating its efficacy beyond safety tasks alone.

Key Insights

1. Reverse Prompt Design: The experiment explores various reverse prompts such as random replacements, opposites, and manually crafted prompts to identify the most effective strategy. Manual reverse prompts outperform others, showcasing the importance of careful prompt construction to induce and subsequently suppress undesired responses effectively.
2. Parameter Tuning: Adjusting the contrastive penalty strength, denoted as $\alpha$ in their method, influences the magnitude of safety improvement. A positive correlation between the severity of performance degradation through reverse prompts and subsequent gains via Rose highlights the mechanism's dependency on the efficiency of these prompts in inducing undesired outputs.
3. Comparative Performance: Rose outperforms common inference-time counterparts like self-correction prompts and traditional contrastive decoding, offering a more robust solution for immediate safety improvements during model inference.

Implications and Future Directions

The implications of implementing Rose are multifaceted. It offers a deployment-ready solution for enhancing safety in LLMs without the substantial data and computational overhead associated with training-time methods like RLHF. The ability to improve both safety and general responses suggests Rose's broader applicability in various deployment contexts, including highly sensitive domains requiring robust safety measures.

Future research could explore scaling Rose's performance on larger models (e.g., beyond the 20B parameter regime) and optimizing inference efficiency to minimize the extra computational cost incurred by the dual-pass nature of its contrastive decoding strategy. Moreover, integrating Rose with additional safety-tuning methods could further compound safety benefits, offering a comprehensive approach to alignment in LLMs. The study's explorations pave the way for further investigation into contrastive approaches as viable solutions to immediate and scalable safety enhancements in AI systems.