---
title: Refusal Heads in Transformer Models
url: https://www.emergentmind.com/topics/refusal-heads
type: topic
---

# Refusal Heads in Transformer Models

Refusal heads are a mechanistically identifiable, sparsely distributed subset of transformer attention heads whose outputs play a critical causal role in enforcing a language model’s aligned refusal behavior—suppression of harmful, unsafe, or policy-violating continuations. This concept, rigorously established in recent interpretability-driven research, underlies both the vulnerability of current models to jailbreak attacks and emerging strategies for robust safety alignment. Refusal heads are not a diffuse property: ablation or amplification of their activations directly controls the likelihood of a model refusing or complying with unsafe requests. Recent literature further generalizes refusal as an emergent geometry—one or more directions or cones in residual activation space—often mediated or implemented by these specialized heads.

## 1. Formal Definition and Mechanistic Characterization

Refusal heads, termed “safety heads” in some works, are empirically defined via two intervention operations [2603.08234, 2508.19697, 2506.00085]:

- **Necessity**: Setting the activation of an individual attention head $h^*$ to zero at inference time (head ablation) raises the model’s Attack Success Rate (ASR) on harmful prompts by tens of percent, indicating that this head suppresses noncompliant outputs.
- **Sufficiency**: Amplifying the same head’s activations (scaling by $w>1$) reduces ASR, suppresses harmful continuations, and boosts refusal response rates.

At the circuit level, each layer’s output to the residual stream is a sum over head outputs and feedforward (MLP) contributions:
$$
R^{l+1} = R^l + \sum_{h} s_{l,h} + \ldots
$$
A refusal head $h^*$ adds a vector $s$ to $R$, shifting representations toward the refusal direction—typically associated with tokens “I’m sorry, but I can’t help with that.” Removal or downweighting of $h^*$ unbalances this effect, allowing the intrinsic generative “continuation” heads to drive compliance with harmful requests [2603.08234].

## 2. Methodologies for Identification

Multiple rigorous pipelines have been introduced for the localization of refusal heads:

### a. Path Patching and Causal Tracing
A three-phase protocol—clean prompt ($P_{cl}$), corrupted continuation-inducing prompt ($P_{cor}$), and patched run (clean context, but with particular head’s activation copied from $P_{cor}$)—is used to compute per-head KL divergence effects (“patching effect” $\Delta_{\text{patch}}$). Heads with large positive $\Delta_{\text{patch}}$ are deemed causally critical for safety [2603.08234].

### b. Refusal Direction Projections
Refusal heads are frequently identified as those whose output vectors project maximally onto a learned “refusal direction” $r$ in the residual stream. This direction is obtained via difference-in-means (harmful vs. harmless prompt activations), gradient-based “Refusal Direction Optimization” (RDO), or the automated COSMIC framework [2506.00085, 2508.19697, 2504.04215, 2502.17420]. The per-head alignment is given by:
$$
S_h = \frac{1}{|D_h|} \sum_{x \in D_h} |\langle O_h(x), r \rangle| / \|r\|
$$
where $O_h(x)$ is the output of head $h$ on prompt $x$ [2508.19697].

### c. Activation Scaling
Head-level impact is validated by sweeping a scalar weight $w$:
$$
h' = w \cdot h
$$
and capturing the monotonic tradeoff between refusal rate and ASR [2603.08234].

### d. Head-type Attribution in Chains-of-Thought
For models with reasoning traces, linear probes are trained to score refusal intent at each token. Per-head contributions are then quantified by inserting or ablating head outputs at the critical “cliff” position, directly measuring their effect on the refusal score and ultimately on ASR [2510.06036].

## 3. Structural and Functional Diversity

Refusal heads are not an architectural constant: their function and breadth differ across model scale, training regime, and architecture.

- In Llama-2, refusal heads encode harmfulness recognition, driving up harmfulness detection rates (HDR) on unsafe prompts when amplified, but are prone to over-rejection of safe instructions at high scaling [2603.08234].
- In Qwen2.5, refusal heads specialize in refusal execution, showing low HDR increases on safe data and a sharp HDR dropoff on harder attacks, indicating a division of labor between harm recognition and refusal actuation [2603.08234].
- COSMIC and geometric analyses reveal that refusal-relevant mechanisms frequently form not a single direction, but a multi-dimensional “concept cone” of up to 5 dimensions in large models—the axes of which may be implemented by distinct sets of heads or MLPS. Ablation or suppression along multiple axes is required to neutralize aligned refusals [2502.17420].

## 4. Concentration, Sparsity, and Robustness

Refusal heads are typically sparse:
- In canonical chat models, ablating 50–100 heads (out of 1024 total, i.e., 5–10%) can drive refusal failure rates (harmfulness) from near 0% to 40–80% under red-teaming attacks [2508.19697].
- In chain-of-thought models, as little as 3% of heads account for the majority of the “refusal cliff,” and ablating these restores robust safety even under complex jailbreaks [2510.06036].
- Head-frequency heatmaps confirm a disproportionate functional load on a small subset of heads, often concentrated in mid-to-upper layers.

However, heavy concentration creates a “single-point-of-failure” risk: adaptive jailbreaks can target these heads for suppression or inversion, bypass

Source: https://www.emergentmind.com/topics/refusal-heads