DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints (2405.19026v2)

Abstract: Recent advances in LLM assistants have made them indispensable, raising significant concerns over managing their safety. Automated red teaming offers a promising alternative to the labor-intensive and error-prone manual probing for vulnerabilities, providing more consistent and scalable safety evaluations. However, existing approaches often compromise diversity by focusing on maximizing attack success rate. Additionally, methods that decrease the cosine similarity from historical embeddings with semantic diversity rewards lead to novelty stagnation as history grows. To address these issues, we introduce DiveR-CT, which relaxes conventional constraints on the objective and semantic reward, granting greater freedom for the policy to enhance diversity. Our experiments demonstrate DiveR-CT's marked superiority over baselines by 1) generating data that perform better in various diversity metrics across different attack success rate levels, 2) better-enhancing resiliency in blue team models through safety tuning based on collected data, 3) allowing dynamic control of objective weights for reliable and controllable attack success rates, and 4) reducing susceptibility to reward overoptimization. Overall, our method provides an effective and efficient approach to LLM red teaming, accelerating real-world deployment.

• The paper introduces a novel red teaming method that relaxes reward constraints to enhance query diversity without sacrificing attack success rates.
• The paper employs constrained reinforcement learning and a dynamic semantic diversity reward to generate more varied and novel attack queries compared to previous methods.
• The paper demonstrates that its approach mitigates overoptimization and strengthens blue team resilience through rigorous experimental validation.

A Formal Analysis of DiveR-CT in Enhancing Automated Red Teaming for LLM Safety

Managing the safety of LLMs is of paramount importance as they become increasingly prevalent in diverse applications. Manual red teaming, where experts identify vulnerabilities by interacting with these models, is labor-intensive and subjective. Consequently, automated red teaming has emerged as a promising substitution, offering consistency and scalability. However, existing methods often compromise data diversity in their pursuit to maximize attack success rates (ASR). This paper introduces a novel approach, Diversity-enhanced red teaming with Relaxing ConstrainTs (DiveR-CT), which focuses on enhancing diversity without sacrificing effectiveness.

Methodology

The methodology centers on treating unsafe rewards as threshold constraints instead of strict optimization targets. This broader perspective allows the policy to balance between achieving successful attacks and maintaining semantic diversity. Further contributions include a dynamic nearest-neighbor reward system to measure semantic diversity, addressing limitations observed in previous methods such as Curiosity Red Teaming (CRT).

Constrained Objectives

In traditional red teaming, the objective predominantly focuses on maximizing ASR, which leads to a narrow set of attack queries and potential overoptimization issues. Diver-CT relaxes this by employing Constrained Reinforcement Learning (CRL), where unsafe rewards are treated as constraints. This shift not only broadens the space of potential queries but also mitigates the overfitting of the red team policies to high-confidence unsafe scores. Subsequent experimental results validate that this approach significantly improves the diversity of red teaming outputs across different ASR levels.

Dynamic Semantic Diversity Reward

A novel semantic reward mechanism is proposed based on the nearest neighbor approach, ensuring dynamic adaptability as the history of generated queries grows. This mitigates the diminishing returns seen in models like CRT, where rewards based on historical embedding similarities lead to novelty stagnation. DiveR-CT’s use of dynamic targets fosters uniform semantic space coverage, prompting the generation of semantically diverse attack queries.

Experimental Validation

The effectiveness of DiveR-CT was validated through comprehensive experiments. The researchers report significant findings in several areas:

1. Diversity Metrics: DiveR-CT outperforms existing methods in both lexical and semantic diversity metrics. Across various ASR levels, DiveR-CT consistently generates more diverse and semantically rich queries compared to CRT and traditional RL-based methods.
2. Attack Success Rate: The proposed method provides dynamic control of objective weights, enabling reliable ASR adjustments. This flexibility allows fine-tuning attack success rates without compromising the quality and diversity of the generated queries.
3. Overoptimization Resistance: By avoiding strict maximization of unsafe reward scores, DiveR-CT shows reduced susceptibility to overoptimization. The generated queries perform comparably well against a test classifier not seen during training, evidencing the method’s generalizability and robustness.
4. Enhanced Blue Team Resilience: Fine-tuning blue team models with data generated by DiveR-CT improves their resilience to adversarial attacks more effectively than when using data from existing methods. This is attributed to the higher diversity and broader range of vulnerabilities explored by DiveR-CT.

The experiments extended to varying scenarios, including using different safety classifiers and targeting more robust blue team models like Llama-2-7b-chat-hf and Meta-Llama-3-8B-Instruct. In all instances, DiveR-CT maintained controlled ASR and superior diversity metrics, reinforcing its applicability in different contexts.

Implications and Future Directions

The practical implications of DiveR-CT are significant. By enabling more comprehensive and diverse automated red teaming, this method enhances the safety protocols of LLMs, ensuring they are robust against a wider array of potential exploits. From a theoretical perspective, DiveR-CT's approach to incorporating constraints in reinforcement learning opens new avenues for balancing objectives in adversarial settings.

Future research could explore extending DiveR-CT’s principles to multi-turn interactions and incorporating domain knowledge to ensure uniform topic coverage in red teaming queries. Additionally, the use of dynamic thresholds could be refined further to adapt to evolving contexts and model behaviors.

Conclusion

DiveR-CT represents an innovative step in automated red teaming, addressing the critical balance between attack effectiveness and query diversity. By employing constrained objectives and dynamic rewards, it alleviates the issues inherent in existing methods, such as overoptimization and limited query diversity. This method significantly contributes to enhancing the safety and robustness of LLMs, mitigating potential risks in their deployment. As automated red teaming continues to evolve, DiveR-CT stands out as a pivotal development in ensuring the comprehensive evaluation and fortification of LLMs against adversarial threats.