Jailbroken: How Does LLM Safety Training Fail? (2307.02483v1)

Abstract: LLMs trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.

Analysis of "Jailbroken: How Does LLM Safety Training Fail?"

Overview

In the paper "Jailbroken: How Does LLM Safety Training Fail?" the authors investigate the vulnerabilities of LLMs post-safety training. Specifically, they explore the prevalent 'jailbreak' attacks that bypass safety mechanisms in models like GPT-4 and Claude v1.3. The paper proposes that safety training may fail due to competing objectives and mismatched generalization, which are exploited to design effective attacks. These insights highlight fundamental limitations in current safety training methodologies and suggest that more sophisticated safety mechanisms are necessary to counter adversarial misuse.

Key Findings

The authors provide a comprehensive empirical evaluation of state-of-the-art safety-trained models against a variety of jailbreak attacks, validating their hypotheses with the following findings:

• Competing Objectives: The paper identifies this failure mode when the pretraining and instruction-following objectives of a model conflict with its safety objectives. This conflict is exploited by crafting prompts that heavily penalize refusal under the pretraining distribution or instruction-following objective.
• Mismatched Generalization: This failure mode arises when safety training does not cover certain input distributions that the model's pretraining does. The authors use Base64 encoding as a prominent example of how adversarial prompts evade safety mechanisms by being out-of-distribution for safety training while within capabilities acquired during pretraining.
• Empirical Evaluation: The research demonstrates that combinations of simple attack techniques, such as prefix injection and obfuscation methods, achieve high success rates in bypassing model safety measures. The adaptive attack succeeded on a broad range of harmful prompts, showing robust vulnerability in the models tested.

Implications

The findings suggest that scaling up LLMs alone cannot resolve these failure modes, as scaling might exacerbate the surface for attacks by broadening model capabilities without matching safety capability training. The notion of safety-capability parity is introduced, emphasizing that safety mechanisms need to match the sophistication of the underlying model. Otherwise, adversaries may exploit the model's cutting-edge capabilities that less sophisticated safety techniques cannot adequately counter.

Speculation on Future AI Developments

Moving forward, the paper implies that future developments in AI safety must consider integrating safety throughout the model's lifecycle, possibly starting from pretraining. As models become more capable, safeguarding them with equivalently sophisticated safety models is essential.

Conclusion

In conclusion, the research emphasizes the inherent vulnerabilities in current LLM safety training methods and provides foundational insights into designing more robust adversarial defenses. The discussion highlights the need for ongoing analysis and improved safety mechanisms amidst the increasing deployment and capabilities of LLMs in real-world applications.