Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models

Abstract: Jailbreak attacks bypass LLM safety alignment, yet their mechanisms remain poorly understood. We provide evidence that attacks do not comprehensively eliminate safety features, but instead selectively suppress specific attention heads. We identify two functionally differentiated types: Adversarially Compromised Heads (ACHs) concentrated in early layers, which are suppressed under attacks, and Safety-Aligned Heads (SAHs) in mid-layers, which maintain robust activations even when attacks succeed. Ablation studies support the causal role of ACHs and the contribution of SAHs to robust activations: suppressing a small number of ACHs is sufficient to induce jailbreak-like behavior on normally refused inputs, while removing SAHs substantially weakens mid-layer safety activations. Token-level attribution further shows that ACH suppression is driven specifically by attack-template tokens, providing a mechanistic account of why attacks can bypass refusal decisions through ACH suppression while leaving internal safety signals sustained by SAHs -- a phenomenon we term Robust Harmful Features. To validate the practical significance of this robustness, we show that simply reading these persistent activations -- without any training -- yields competitive aggregate detection performance with strong adversarial robustness.

• The paper identifies Adversarially Compromised Heads in early layers and Safety-Aligned Heads in mid layers, showing that ablating just eight compromised heads raises attack success from 0% to 95% on Llama-3.
• Token-level attribution shows jailbreak templates suppress compromised heads while activating safety-aligned heads, with activation differences exceeding three standard deviations and reaching six in Llama-3-70B.
• The paper introduces a training-free, single-forward-pass detector that achieves up to 0.976 weighted Macro-F1 on SALAD-Bench and competitive results against WildGuard and Qwen3Guard, though it requires white-box access.

Overview and central claim

This paper investigates what happens inside an aligned LLM when a jailbreak attack succeeds. The authors' central empirical claim is that successful attacks do not comprehensively erase safety-relevant representations; instead, they selectively suppress a small set of early-layer attention heads while mid-layer heads continue to encode refusal-related signals. They formalize this distinction through two head categories: Adversarially Compromised Heads (ACHs), concentrated in early layers, whose activations are suppressed under attack, and Safety-Aligned Heads (SAHs), distributed in mid layers, which remain strongly activated even on successfully attacked inputs. The persistence of mid-layer safety activations under successful attacks is termed Robust Harmful Features. The analysis is conducted primarily on Llama-3-8B-Instruct and Llama-2-7B-Chat, with supplementary experiments on Llama-3-70B-Instruct.

The practical payoff is a training-free harmful-input detector that reads internal activations in a single forward pass and achieves aggregate performance competitive with dedicated safety models such as WildGuard and Qwen3Guard, with particularly strong gains on adversarial benchmarks where vanilla refusal behavior fails.

Method: back-projection and overlap-based head classification

The methodological foundation is the linear structure of attention. The authors extract the refusal direction $r^{(L^*)}$ via difference-in-means at the end-of-instruction (EOI) position (layer 12 for Llama-3, layer 14 for Llama-2), following Arditi et al.'s single-direction account of refusal. Because each head's OV circuit $W_{OV} = W_O W_V$ is purely linear, the mid-layer refusal direction can be back-projected through transposed OV circuits layer by layer, yielding a head-specific direction $r_v^{(l,h)}$ that quantifies each head's potential contribution to the refusal signal. Head activation at EOI is scored as $$s^{(l,h)}(x) = \langle o_{\text{EOI}}^{(l,h)}(x), r_v^{(l,h)} \rangle$$.

Heads are then classified by comparing activation-score distributions—estimated with KDE and quantified with the overlap coefficient (OVL)—across three input types: benign instructions (Alpaca), harmful requests that are refused, and attack variants that succeed. A two-stage filter applies thresholds $\tau_{\text{harmful}} = \tau_{\text{attack}} = 0.5$: first selecting heads discriminative between harmful and benign inputs, then splitting those whose distributions shift under attack according to the direction of change. Suppressively shifted heads become ACHs; strengthened heads become SAHs. The appendix validates this procedure against KDE bandwidth choice (Spearman rank correlation of OVL scores stays above 0.92 across bandwidth scaling factors from 0.2× to 2× Scott's rule) and threshold variation (conclusions hold for $\tau \in [0.4, 0.6]$), which addresses the concern that the taxonomy is a KDE artifact.

Classification results and spatial structure

The classification yields a consistent decomposition across both models: for Llama-3, 80 harmful-salient heads, 21 ACHs, and 17 SAHs among 416 analyzed heads; for Llama-2, 43, 20, and 19 among 480. Spatially, more than 60% of ACHs fall in layers 0–3, whereas SAHs dominate layers 5–12. The preliminary Llama-3-70B analysis reproduces this segregation qualitatively (62 ACHs in layers 0–6; 83 SAHs in layers 8–20), though with only 40 successful attack pairs the authors appropriately present it as preliminary rather than definitive.

An important caveat attaches to the dataset construction: only pairs where the model refuses the original request but complies with the attacked version are retained, yielding just 378 pairs for Llama-2 and 176 for Llama-3 out of roughly 10,000 candidates—a 2–3% pass rate. All downstream statistical claims rest on these modest sample sizes, and the paper does not claim the ACH/SAH split exhausts the safety mechanism.

Causal validation via ablation

To move from correlation to causation, the authors inject a negative feature along each ACH's projected direction ($$o_{\text{EOI}} \leftarrow o_{\text{EOI}} - \alpha \cdot r_v$$, $\alpha = 0.8$) into normally refused harmful inputs. The result is strong: intervening on 8 ACHs raises ASR from 0% to 95.0% on Llama-3 (81.6% on Llama-2), saturating at 99.5%, while random-head controls reach only 4.0% and 10.2% with high variance. Qualitative outputs remain coherent after intervention, suggesting the effect is specific to safety behavior rather than generic degradation. The authors are careful about scope: ACH suppression is shown to be a sufficient pathway for bypassing refusal, not necessarily the pathway real attacks use exclusively.

SAH ablation under attack inputs reduces mean absolute mid-layer activation by 18% (Llama-2) and 14% (Llama-3), supporting SAHs as an important—but explicitly not exclusive—source of Robust Harmful Features.

Token-level attribution

Exploiting the additivity of attention weights, the authors decompose each head's activation into per-token contributions and group them into original-harmful-request tokens versus attack-template tokens. The finding is sharp: on harmful tokens, ACHs and SAHs behave nearly identically (medians within ±0.15 standard deviations); on template tokens they diverge by more than three standard deviations, with ACHs suppressed (median ≈ −1.5) and SAHs activated (median ≈ +1.5). On Llama-3-70B the separation exceeds six standard deviations. This provides a mechanistic account of the paper's title phenomenon: attacks suppress ACHs specifically through their template tokens, thereby bypassing refusal, while SAHs respond positively to the same templates and sustain internal safety signals.

Detection application

As practical validation, the authors build a training-free detector that selects components with low benign-vs.-harmful OVL, aggregates standardized activations via a top-$m$ rule, and classifies by thresholding—no training, gradients, or intervention required. Across ten safety-eval datasets, it substantially improves over vanilla refusal behavior on every architecture tested: Gemma-2B rises from 0.470 to 0.853 weighted Macro-F1 (+81.5% relative), Qwen-7B from 0.506 to 0.860. Adversarial results directly instantiate Robust Harmful Features: on SALAD-Bench, Gemma-2B improves from 0.270 to 0.976, exceeding WildGuard's 0.970 despite using no training. Llama-3-8B attains 0.888 weighted F1, above WildGuard (0.880) and Qwen3Guard (0.877). Against training-free baselines on Qwen-7B, it outperforms GradientCuff (0.856 vs. 0.741) and CAST (0.611), and it also achieves the best over-refusal profile on OR-Bench-Hard-1K (F1 0.606, hard specificity 0.418). Token-attribution visualizations show intent-bearing tokens ("script", "movie") contributing consistently positive scores across Llama, Gemma, and Qwen architectures, supporting cross-family generality.

Limitations and open questions

The paper is explicit about several constraints. The mechanistic analysis covers attention heads only; MLP blocks—which constitute 60–70% of parameters—and LayerNorm/RMSNorm effects are excluded by design, justified by linear decomposability and indirect evidence but not directly tested. The refusal direction is a single first-order summary rather than a full refusal subspace or cone. Primary causal experiments use two 7–8B models, with the 70B evidence limited by sample size. The paired-sample filtering criterion caps data availability precisely for the best-aligned models. Finally, the detector requires white-box access to activations, restricting deployment to open-weight settings, and performance varies across models (e.g., Llama-2-7B reaches only 0.752).

Conclusion

This paper reframes jailbreak success as selective component suppression rather than wholesale erasure of safety representations. Its strongest quantitative results—95% ASR from ablating eight heads, three-to-six-standard-deviation token-attribution separations, and adversarial detection F1 near 0.98 without training—jointly support the claim that aligned models retain exploitable internal safety signals even when their explicit refusal behavior is bypassed. The open questions left by the work concern MLP contributions to refusal, generalization beyond the studied architectures and alignment strengths, and whether defenses can be built that protect or monitor ACHs and SAHs directly.