Logit-Gap Steering in LLMs

• Logit-gap steering is a method that rebalances RLHF-aligned LLM outputs by narrowing the gap between refusal and affirmation logits.
• It employs a greedy gap-covering algorithm with surrogate scoring to quickly discover adversarial suffixes at a fraction of the computational cost of previous methods.
• The approach also serves as a diagnostic tool that exposes alignment artifacts and enables high-rate, scalable jailbreaks across different model sizes.

Logit-gap steering is a method for efficiently manipulating the output behavior of reinforcement learning from human feedback (RLHF)-aligned LLMs by directly targeting and closing the quantifiable logit difference (“gap”) between refusal and affirmation generations. The approach allows the rapid discovery of short, prompt-agnostic adversarial suffixes that reverse the refusal bias in LLMs—enabling high-rate jailbreaks in a fraction of the computational cost of previous techniques. Logit-gap steering also serves as a diagnostic tool, exposing the precise mechanisms and artifacts of alignment in contemporary LLMs.

1. Formulation and Core Insight

In RLHF-aligned LLMs, safety training induces a large gap between the logit assigned to refusal tokens (e.g., "I'm sorry," "As an AI LLM, I cannot…") and the logit assigned to affirmative tokens (e.g., "Certainly," "Absolutely") for harmful or restricted prompts. This refusal–affirmation logit gap is both the direct measurable effect of alignment and the primary obstacle to producing policy-violating completions.

Logit-gap steering operates by appending a short, targeted token sequence (suffix) to the user prompt, which closes or inverts the logit gap in the model’s forward pass:

• Let $h_0$ denote the model's hidden state after the prompt.
• $\ell_{\text{refusal}}(h)$ and $\ell_{\text{affirm}}(h)$ are the logits for the canonical refusal and affirmation tokens, respectively.
• The initial refusal–affirmation gap is $$\Delta_0 = \ell_{\text{refusal}}(h_0) - \ell_{\text{affirm}}(h_0)$$.

If a suffix $S$ reduces or reverses this gap, the model is steered toward generating non-refusal outputs, effectively acting as a jailbreak even on strongly-aligned checkpoints.

2. Methodological Approach: Suffix Search via Gap-Closing Score

Logit-gap steering introduces an efficient, greedy gap-covering algorithm:

• Candidate Filtering: Select semantically plausible, in-distribution tokens (excluding rare, out-of-domain, or refusal tokens) as suffix candidates.
• Surrogate Gap-Closing Score: For each candidate token $t$ at hidden state $h$, an additive score $F(h, t)$ is computed:

$$F(h, t) = \Delta F_{\text{logit}}(h, t) - \lambda_{\mathrm{KL}} \Delta\mathrm{KL}(h, t) + \lambda_{r}\Delta r(h, t)$$

• $\Delta F_{\text{logit}}(h, t)$: Reduction in logit gap after adding $t$.
• $\Delta\mathrm{KL}(h, t)$: Approximate Kullback-Leibler penalty, to discourage unnatural, out-of-distribution completions.
• $\Delta r(h, t)$: Reward shift proxy, encouraging affirmative tokens.
  • Sort–Sum–Stop Process: Sort all candidate tokens by $F(h, t)$, append tokens iteratively, and halt when the cumulative sum surpasses $\Delta_0$:

$\sum_{i=1}^k F(h_{i-1}, t_i) \geq \Delta_0$

where $h_{i}$ is the state after applying $t_i$.

• Efficiency: This process typically completes within seconds and produces short, topical suffixes.

A phrase-level extension generalizes the approach to harvesting fluent, sentence-level macro-tokens to further enhance attack efficacy.

3. Experimental Results and Performance Analysis

Logit-gap steering outperforms traditional jailbreaking and adversarial prompt discovery on contemporary aligned LLMs on several core metrics:

• Attack Success Rate (ASR): Achieves 80–100% pass@1 on AdvBench toxic prompt benchmarks, substantially exceeding gradient-based (GCG), random, or manual strategies.
• Efficiency: Requires two orders of magnitude fewer model calls compared to beam search or gradient attacks. For example, on Qwen-2.5-0.5B, logit-gap steering uses ~20,000 model calls and finishes in a few seconds versus millions of calls and several minutes for GCG.
• Generalization: Suffixes discovered on a small checkpoint generalize to far larger models (0.5B to 70B parameters) within a family with only minor loss in ASR.
• Topic Consistency: Suffixes maintain topical grounding in >80% of cases, a significant contrast to the topic drift often observed in other attack methods.

Model	GCG ASR (%)	Logit-Gap ASR (%)
Llama-3.1-8B-Instruct	34.4	96.7
Qwen2.5-0.5B-Instruct	80.8	100.0
Qwen2.5-72B-Instruct	4.6	61.4

Universal suffixes discovered by logit-gap steering remained effective across hundreds of prompts and all tested models of a given architecture/scale.

4. Technical Details: Gap Score, KL Penalty, and Reward Shift

The gap-closing score is constructed to target not only the raw gap but also account for distributional drift and reward model artifacts post-alignment:

• Logit-Gap Term: Reflects the reduction in refusal–affirmation gap after adding a token.
• KL Penalty: Penalizes deviations from the model’s typical (neutral) distribution using logits for a reference frequent token, controlling for out-of-distribution completions.
• Reward Proxy: Increases the logit of the affirmation token, effectively "rewarding" the model for moving toward the desired generation.

Each term operates strictly on the model's forward pass, making the approach fully "forward-computable"—no backpropagation or gradient estimation is required.

5. Analysis of Alignment Artifacts and Model Probing

Logit-gap steering exposes several previously under-characterized alignment artifacts:

• Sentence-Boundary Reward Cliffs: Alignment reward models typically induce strong penalties after full stops, causing refusal to reactivate after a sentence even mid-jailbreak, unless the adversarial suffix ends mid-clause.
• Alignment Regime Distinctions: The method distinguishes between PPO, DPO, and quantization regimes by sensitivity to suffix length, coherence, and family transfer.
• Reward Geometry and "Glitch" Tokens: The approach can uncover "spiky" logit artifacts, which accidentally facilitate or hinder jailbreaks.

These insights render logit-gap steering a lightweight tool for the diagnostic probing of safety training and alignment architectures.

6. Broader Implications and Security Considerations

Logit-gap steering generalizes efficiently:

• Universal Suffixes: Once discovered, a suffix can be used on thousands of unseen harmful prompts and on different checkpoints.
• Family Transfer: Suffixes scale across model sizes (0.5B to 70B) within a model family.
• Limited Defenses: Because the approach uses only in-distribution tokens and acts mechanistically, it evades many blacklist or anomaly-based detection schemes.

The paper finds that any alignment strategy that creates a refusal–affirmation gap, but does not fully eliminate unsafe completions, remains vulnerable unless further architectural interventions are employed. This underscores the need for layered approaches to LLM alignment and more robust reward fusion strategies.

7. Practical Applications

• Efficient red-teaming and safety evaluation: Enables rapid, scalable benchmarking of alignment strength on private or API-based models.
• Alignment artifact mapping: Provides a direct, interpretable window into how RLHF and similar techniques shape model internals.
• General attack recipe: The same approach applies to other alignment targets (e.g., topic bans, reasoning suppression) wherever logits are used to enforce behavioral gaps.

Summary Table: Logit-Gap Steering

Aspect	Description
Mechanism	Greedy suffix discovery using forward-computable logit/KL/reward score
Key Formula	$$F(h, t) = \Delta F_{\text{logit}} - \lambda_{\mathrm{KL}} \Delta\mathrm{KL} + \lambda_{r}\Delta r$$
Attack Success Rate	80–100% pass@1, prompt- and model-agnostic
Efficiency	10–100× faster than previous attacks; typically ≤1s per prompt
Generalization	Universal suffixes, scalable to 70B checkpoints, no per-prompt optimization needed
Probing Alignment Artifacts	Reveals reward cliffs, boundary phenomena, and alignment-induced quirks

Logit-gap steering thus represents an interpretable, scalable, and highly effective framework for both practical jailbreak discovery and the forensic analysis of alignment mechanisms in LLMs.