Head-Masked Nullspace Steering (HMNS)

• The paper introduces a novel mechanism-level jailbreak method that identifies and masks safety-critical attention heads using KL divergence to disrupt default refusal behaviors.
• It employs geometry-aware nullspace steering by injecting perturbations in the orthogonal complement of the muted head subspace to control model output.
• The method demonstrates robust performance and efficiency across benchmarks and models, outperforming traditional prompt-only attacks with fewer queries.

Head-Masked Nullspace Steering (HMNS) is a mechanism-level jailbreak method for decoder-only transformers that intervenes inside a model’s circuit at inference time rather than relying on prompt-only manipulation. It identifies the attention heads most causally responsible for the model’s default, safety-aligned refusal behavior for a given prompt, masks those heads’ write paths by zeroing their out-projection columns, injects a steering perturbation in the orthogonal complement of the muted head subspace, and repeats this procedure in a closed loop as decoding unfolds. The method is presented as a geometry-aware, interpretability-informed approach to controlled model subversion, with the central claim that refusal behavior often concentrates in a small, localizable subset of attention heads (Pramanik et al., 11 Apr 2026).

1. Conceptual framing and motivation

HMNS is proposed in response to several limitations attributed to earlier jailbreak paradigms. Prompt-only attacks are described as often brittle under defenses; optimization-based attacks can require many queries and are hard to interpret; rewriting or template attacks manipulate surface form without directly controlling internal computation; and existing methods often do not explain why they work mechanistically. Against this background, HMNS is positioned as interpretable, geometry-aware, defense-resilient, and compute-conscious (Pramanik et al., 11 Apr 2026).

The method’s defining premise is circuit-level intervention. Rather than attempting to “trick” the model from the outside, HMNS operates on the internal attention mechanism during inference. The paper’s one-sentence intuition is that the method works by finding the model’s safety-critical attention heads, silencing their residual writes, and then pushing the residual stream in a direction those heads cannot “explain away” because it lies in their orthogonal complement.

A common misconception would be to treat HMNS as a form of permanent model editing. The paper explicitly distinguishes it from that interpretation: masking is not permanent weight editing, but a forward-pass-only suppression of the out-projection columns for selected heads. Another misconception would be to interpret the method as purely geometric or purely interpretability-driven. The reported ablations argue instead for a compound mechanism in which causal head identification, hard masking, nullspace-constrained injection, residual-norm scaling, and iterative re-identification all contribute materially to effectiveness.

2. Circuit-level mechanism in decoder-only transformers

The formal description assumes a decoder-only transformer with layers $l \in \{1,\dots,L\}$, residual dimension $d$, and $H_l$ attention heads per layer. At the final token position $T$, the concatenated head outputs are

$h_{e,T} \in \mathbb{R}^{H_l d_h},$

and the out-projection matrix is

$W^O \in \mathbb{R}^{d \times (H_l d_h)}.$

The attention block writes into the residual stream as

$h^{out} = W^O h_{e,T}. \tag{2}$

Each head $h$ occupies a column block of $W^O$, so the contribution of an individual head is the corresponding slice,

$W^O[:,\, h d_h : (h+1)d_h]\, h_{e,T}^{(h)}.$

HMNS manipulates these write paths directly (Pramanik et al., 11 Apr 2026).

Causal head identification is carried out by counterfactual ablation combined with KL divergence. For a head $d$0, a diagonal selector matrix $d$1 zeros the slice corresponding to that head, giving the masked out-projection used for probing,

$d$2

Let $d$3 denote the baseline next-token distribution and $d$4 the distribution under ablation. The causal importance score is

$d$5

HMNS ranks heads by this score and selects the top-$d$6 globally; the experiments use a small global $d$7, with $d$8, so that the selected head subspace does not fill the entire residual space.

After selection, HMNS suppresses the chosen heads by zeroing their corresponding column blocks in $d$9. For a layer $H_l$0, with selected head set $H_l$1, the concatenated write matrix is

$H_l$2

This spans the muted subspace $H_l$3. The operational intervention applies masking only for the current forward pass:

$H_l$4

where $H_l$5 is the aggregated selector for all selected heads at that layer.

3. Nullspace steering and the closed-loop procedure

Once the muted subspace has been defined, HMNS constructs a steering direction in its orthogonal complement. For each intervened layer $H_l$6, a thin QR factorization is computed,

$H_l$7

The orthogonal projector onto the complement of the muted subspace is

$H_l$8

HMNS then samples a Gaussian probe,

$H_l$9

and projects it into the nullspace:

$T$0

The resulting $T$1 lies in the orthogonal complement of the muted subspace. The forward-pass perturbation is

$T$2

Here $T$3 is the steering coefficient, $T$4 is the residual activation at the intervention site, and $T$5 scales the perturbation to the local activation magnitude. The paper’s interpretation is that this makes the intervention small relative to the model’s current activation scale and therefore less likely to destabilize fluency.

The complete procedure is explicitly closed-loop rather than one-shot. The described cycle is: run a clean baseline forward pass; score heads via KL-based ablation; construct the masked subspace and nullspace direction; apply masking and steering; generate one continuation attempt; and, if unsuccessful, recompute attribution on the updated context and repeat. The paper reports that the head rankings have a stable core but also meaningful churn across iterations, and that freezing the initial top-$T$6 heads hurts performance (Pramanik et al., 11 Apr 2026).

Algorithmically, the method is summarized as baseline logits $T$7, followed at each iteration by per-head ablations, KL-based causal scoring, global top-$T$8 selection, layerwise muted-subspace construction, nullspace-direction computation, masking plus steering at the final token position after attention, and then generation. If success is achieved, the loop stops; otherwise the baseline distribution is updated from the new context and the process repeats.

4. Geometric and theoretical properties

The paper emphasizes orthogonality, irreproducibility, basis invariance, and nullspace energy. Theorem 2 states that the steering vector lies in the orthogonal complement of the muted write subspace, so masked heads cannot undo it. In symbols, for all $T$9,

$h_{e,T} \in \mathbb{R}^{H_l d_h},$0

Therefore the injected vector $h_{e,T} \in \mathbb{R}^{H_l d_h},$1 cannot be reconstructed by any linear combination of masked-head outputs (Pramanik et al., 11 Apr 2026).

The text characterizes this as local irreproducibility with respect to the muted heads. This does not amount to a global guarantee about full decoding behavior. The paper explicitly notes that its guarantees are local, not global, because decoding is nonlinear and stochastic. That caveat is central to interpreting the formal claims: the geometry constrains what the muted heads can linearly reconstruct at the intervention site, but it does not eliminate broader nonlinear compensatory dynamics elsewhere in the network.

Theorem 3 asserts basis invariance under reparameterizations of the masked head basis. If the masked write matrix undergoes a block-orthogonal change of basis, the span is preserved, and thus the orthogonal projector and resulting nullspace direction remain unchanged up to sign. Proposition 4 gives the expected squared energy of a projected Gaussian:

$h_{e,T} \in \mathbb{R}^{H_l d_h},$2

where $h_{e,T} \in \mathbb{R}^{H_l d_h},$3. This is used to reason about how much energy remains after projection.

A plausible implication is that HMNS depends not merely on removing safety-relevant circuitry but on exploiting residual degrees of freedom that remain after that removal. The paper makes this dependence explicit by noting that HMNS assumes a non-degenerate nullspace.

5. Experimental setting and empirical performance

HMNS is evaluated on four jailbreak and safety benchmarks: AdvBench, HarmBench, JBB-Behaviors, and StrongReject. The prompt pool contains 925 unique prompts, divided into a 150-item analysis subset, a 579-item development subset, and a 196-item held-out test subset. The main models are LLaMA-2-7B-Chat, Phi-3-Medium-4K-Instruct (14B), and LLaMA-3.1-70B; appendix experiments additionally include Mistral-7B-Instruct-v0.2, Qwen2.5-14B-Instruct, and Yi-1.5-72B-Chat. The six evaluated defenses are SmoothLLM, DPP, RPO, Paraphrase, PAT, and SafeDecoding (Pramanik et al., 11 Apr 2026).

The implementation settings are zero-shot decoding with temperature $h_{e,T} \in \mathbb{R}^{H_l d_h},$4, top-$h_{e,T} \in \mathbb{R}^{H_l d_h},$5, $h_{e,T} \in \mathbb{R}^{H_l d_h},$6, batch size $h_{e,T} \in \mathbb{R}^{H_l d_h},$7, KV cache disabled during attribution and steered decoding, QR and orthogonality checks in float32, forward passes in bf16, top-$h_{e,T} \in \mathbb{R}^{H_l d_h},$8, maximum closed-loop attempts $h_{e,T} \in \mathbb{R}^{H_l d_h},$9, and steering schedule $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$0. Grading uses two independent graders, GPT-4o and GPT-5, with Cohen’s $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$1 around $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$2–$W^O \in \mathbb{R}^{d \times (H_l d_h)}.$3 depending on benchmark.

The main quantitative claim is that HMNS is best on all four benchmarks and all three main models. Representative results on LLaMA-2-7B-Chat are 98.0 / 93.0 ASR on AdvBench with ACQ 2.0, 96.0 / 92.0 on HarmBench with ACQ 2.1, 99.0 / 94.0 on JBB-Behaviors with ACQ 1.9, and 94.0 / 89.0 on StrongReject with ACQ 2.2. On Phi-3-Medium-14B, the corresponding results are 92.0 / 86.0 with ACQ 2.0, 90.0 / 84.0 with ACQ 2.1, 94.0 / 88.0 with ACQ 1.9, and 86.0 / 80.0 with ACQ 2.2. On LLaMA-3.1-70B, the reported values are 99.0 / 95.0 with ACQ 1.8, 97.0 / 94.0 with ACQ 2.0, 99.0 / 96.0 with ACQ 1.8, and 96.0 / 92.0 with ACQ 2.1.

Relative to the next-best method, usually ArrAttack, the paper reports ASR gains of about $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$4 percentage points under GPT-4o grading and $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$5 percentage points under GPT-5 grading, while using only about 2 queries on average, compared with roughly 7–13 for many baselines. Under the six tested defenses, HMNS remains the strongest or tied strongest method; example average scores under defenses are 55.8 / 39.8 on LLaMA-2-7B-Chat, 57.3 / 41.7 on Phi-3-Medium-14B, and 55.6 / 36.8 on LLaMA-3.1-70B.

The paper also introduces compute-normalized evaluation. The reported metrics are ACQ, IPC, FPS, and LPS, with a forward-equivalent pass defined as the compute of a full forward pass over the generated sequence with standard KV caching. On LLaMA-3.1-70B / AdvBench under matched FLOP budgets against prompt-only baselines, HMNS records FPS $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$6 trillion FLOPs per success and LPS $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$7 s, compared with ArrAttack at FPS $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$8 and LPS $W^O \in \mathbb{R}^{d \times (H_l d_h)}.$9 s. The stated conclusion is that internal intervention does not preclude compute competitiveness under fair budgeting.

6. Ablations, robustness, and limitations

The ablations are presented as central to the paper’s argument that the components of HMNS are jointly necessary (Pramanik et al., 11 Apr 2026). On Phi-3-Medium-14B / AdvBench, full HMNS reaches 96.8 / 92.1 ASR with ACQ 2.1. Removing masking and retaining projection-only reduces ASR to 89.5 / 84.0. Removing projection and using mask-only reduces ASR to 87.9 / 82.2. Direct non-nullspace injection yields 88.7 / 83.1. Without re-identification, ASR falls to 90.2 / 85.0 and ACQ rises to 2.7. Random head selection gives 81.4 / 76.0. Single-layer intervention gives 86.1 / 80.8. Multi-position injection reaches 95.0 / 90.5 but at higher cost.

These ablations support several explicit conclusions: masking and nullspace injection are synergistic; the nullspace constraint matters; re-identification matters because causal heads shift over time; KL-based attribution matters because random heads perform much worse; multi-layer intervention is better than single-layer intervention; and final-position-only injection provides the best efficiency/ASR tradeoff.

Additional design ablations compare full KL divergence with target-logit drop, confidence drop, entropy change, no proxy preselection, serial versus batched masking, and per-layer versus global top-$h^{out} = W^O h_{e,T}. \tag{2}$0. The main conclusion is that KL divergence performs best because it captures full-distribution changes rather than a single logit or entropy summary. Global top-$h^{out} = W^O h_{e,T}. \tag{2}$1 is reported to outperform per-layer top-$h^{out} = W^O h_{e,T}. \tag{2}$2. In the nullspace and injection studies, the paper reports that strict orthogonality is critical, fp32 QR is preferred, RMS scaling is a strong default, injection after attention is best, and hard-zero masking outperforms partial masking.

In robustness studies, HMNS remains effective under standard defenses, stacked defenses, alternative open-weight models, and multi-turn and long-context settings. On the Multi-Turn Human Jailbreaks dataset in the appendix, the paper reports about 91.6% ASR on Phi-3-Medium-14B and 95.2% ASR on LLaMA-3.1-70B, with around 2 queries. The method also maintains a fluency score around 0.66, reported as best among compared methods; toxicity is higher than for prompt-only baselines, which the paper notes is expected because the method is explicitly designed to produce policy-violating outputs.

The broader implication drawn by the paper is that alignment fine-tuning may concentrate refusal behavior into a sparse set of heads, thereby creating an interpretability-informed attack surface. This suggests that safety mechanisms may be vulnerable to localized circuit interventions and that defenses may need to distribute safety behavior more broadly. At the same time, the limitations are substantial: runtime overhead from internal attribution and QR projection, especially on large models; reliance on white-box access to internal activations and head projections; non-global theoretical guarantees; evaluation mainly on standard jailbreak benchmarks rather than every deployment regime; dependence on a non-degenerate nullspace; and the need for careful numerical stability procedures, including fp32 QR, orthogonality checks, and resampling.