Weak-to-Strong Jailbreaking on Large Language Models (2401.17256v5)

Abstract: LLMs are vulnerable to jailbreak attacks - resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-to-strong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weak-to-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

• The paper demonstrates a novel weak-to-strong jailbreaking method that manipulates initial token decoding using two models to bypass safety measures.
• It achieves nearly 99% attack success rates on open-source LLMs by exploiting early divergence in token probabilities with minimal computational overhead.
• The study highlights implications for LLM safety, showing that even slight probability manipulations can cause harmful outputs and necessitate more robust defensive strategies.

Weak-to-Strong Jailbreaking on LLMs

Introduction

The paper "Weak-to-Strong Jailbreaking on LLMs" investigates a burgeoning security concern in the domain of LLMs: the susceptibility to jailbreak attacks. LLMs, despite their transformative potential, are vulnerable to generating harmful, unethical, or biased outputs when targeted by adversaries. This paper introduces a novel method called weak-to-strong jailbreaking, a computationally efficient attack strategy that exposes critical vulnerabilities in the safety alignment of LLMs. Unlike existing jailbreak techniques, which are often computationally demanding, the weak-to-strong method achieves high attack success rates with minimal resources.

Methodology

The weak-to-strong jailbreaking attack capitalizes on the observation that the primary divergence between jailbroken and aligned models occurs early in the decoding process. This insight allows the proposed attack to manipulate the initial decoding probabilities of a large, safe model using two smaller models—a safe and an unsafe one. During inference, the small unsafe model adjusts the predictions of the larger model via log probability algebra, effectively steering it to generate harmful content without extensive computational overhead. This is operationalized through a simple formula that adjusts the safe model's probabilities by the ratio of the unsafe to safe model's token probabilities, amplified by a factor $\alpha$.

Figure 1: KL divergence between token distributions of safe and unsafe Llama models on malicious and general questions over decoding steps. Divergence is higher initially but decreases over time.

Evidence of Vulnerability

The paper presents a thorough analysis demonstrating why current alignment methods in LLMs are insufficient. By evaluating the Kullback-Leibler divergence between safe and unsafe models across decoding steps, it was found that the divergence is significant at the initial tokens but reduces significantly thereafter (Figure 1). Furthermore, the overlapping rate of top tokens increases with longer generation prefixes (Figure 2), indicating a shallow level of safety alignment that can be bypassed with simple manipulations.

Figure 2: Overlap rate of top 10 tokens among different models across increasing prefix lengths, showing increasing similarity.

Experimental Results

Extensive experiments were conducted on five diverse open-source LLMs across two datasets, AdvBench and MaliciousInstruct. The weak-to-strong method achieved ∼99% attack success rates, vastly outperforming previous methods like GCG and adversarial decoding in both efficiency and effectiveness. Notably, the attacked outputs were more harmful than those generated solely by the weak model, underscoring the amplified risk posed by this approach. Additionally, scaling the amplification factor $\alpha$ correlated positively with increased Attack Success Rate (ASR) and harm scores, further validating the efficacy of this method.

Figure 3: Comparison of ASR and harm scores across different model sizes and amplification values on AdvBench dataset.

Defense and Implications

Despite the strong attack results, the paper proposes an initial defense mechanism involving gradient ascent on harmful generations, successfully reducing the success rate of attacks by 20%. However, the study acknowledges that more advanced defenses are necessary to robustly align open-source LLMs against such vulnerabilities.

Conclusion

The paper provides compelling evidence that even well-aligned LLMs remain susceptible to adversarial attacks, particularly through subtle manipulations at inference time. The weak-to-strong jailbreak approach highlights significant oversights in current safety and alignment strategies, calling for the AI community to innovate more robust defenses. As the use of LLMs grows, understanding and countering such vulnerabilities will be paramount in safeguarding the ethical deployment of AI technologies.