Refusal Vector in Neural Network Safety

Updated 20 March 2026

Refusal vector is a specific direction in a model's residual activation space that governs refusal behavior by distinguishing harmful from harmless prompts.
It is extracted using activation differences between harmful and benign datasets and can be ablated or steered to modulate model responses.
Empirical studies show that manipulating the refusal vector significantly reduces harmful responses and serves as a robust behavioral fingerprint for model provenance.

A refusal vector is a direction in the internal activation space of a neural network—typically the residual stream of a transformer—such that linear manipulation of model activations or weights along this direction sharply controls refusal behavior. Originally developed to explain and attack safety mechanisms in LLMs, the refusal vector concept has also been extended to other generative models. Empirical analysis demonstrates that, in safety-aligned LLMs, nearly all refusal behaviors can be mediated by such a vector: erasing it disables the model's reluctance to output harmful content; injecting it artificially induces refusals even on safe requests. This property enables efficient control, auditing, unlearning, and adversarial bypass of refusal policies, but also exposes a structural vulnerability in current alignment methods (Lermen et al., 2024, Arditi et al., 2024, Xu et al., 10 Feb 2026, Facchiano et al., 9 Jun 2025). The following overviews the mathematical definitions, extraction and ablation protocols, empirical results, geometric structure, and implications for safety and model provenance.

1. Mathematical Definition and Extraction Protocols

Formally, let $x_i^{(\ell)}(t)\in\mathbb{R}^d$ denote the residual-stream activation at layer $\ell$ and token position $i$ for input prompt $t$ . Given two datasets—harmful prompts $D_\mathrm{harmful}$ and harmless prompts $D_\mathrm{harmless}$ —the canonical refusal vector at $(\ell,i)$ is defined as: $\mu_i^{(\ell)} = \frac{1}{|D_\mathrm{harmful}|}\sum_{t\in D_\mathrm{harmful}} x_i^{(\ell)}(t),\quad \nu_i^{(\ell)} = \frac{1}{|D_\mathrm{harmless}|}\sum_{t\in D_\mathrm{harmless}} x_i^{(\ell)}(t)$

$r_i^{(\ell)} = \mu_i^{(\ell)} - \nu_i^{(\ell)}, \qquad \hat{r}_i^{(\ell)} = r_i^{(\ell)}/\|r_i^{(\ell)}\|_2$

The most effective $(\ell^*,i^*)$ is selected empirically by the reduction of refusal behavior (ablation) on held-out harmful prompts, subject to benign-task preservation constraints. The selected $\hat{r}$ is then used for ablation or steering (Lermen et al., 2024, Arditi et al., 2024, 2606.09434, Xu et al., 10 Feb 2026).

A closely related fingerprinting formalism aggregates normalized difference vectors layer-wise: $r_\ell = (\mu_\ell^{(h)} - \mu_\ell^{(s)}) / \|\mu_\ell^{(h)} - \mu_\ell^{(s)}\|_2, \qquad \hat{f} = (1/|L|)\sum_{\ell\in L} r_\ell /\|f\|_2$ The resulting vector $\hat{f}$ serves as a behavioral signature for both alignment status and model provenance (Xu et al., 10 Feb 2026).

2. Refusal-Vector Ablation and Steering

Refusal ablation procedurally orthogonalizes model weights to the refusal vector. Given a weight matrix $W$ projecting into the residual stream, update: $W' = W - \hat{r}(\hat{r}^\top W)$ This rank-one edit eliminates the subspace that mediates refusal response; the model ceases to generate refusals, even to "harmful" prompts. The canonical inference-time variant subtracts the $\hat{r}$ component from each residual activation vector: $x'_l = x_l - (\hat{r}^\top x_l) \hat{r}$ Conversely, steering towards refusal follows $x'_l = x_l + \alpha\hat{r}$ with tuning parameter $\alpha$ (Lermen et al., 2024, Arditi et al., 2024, Marshall et al., 2024, 2606.09434).

In diffusion and multi-modal models, the same principle applies. Per-layer mean activation differences for unsafe vs safe prompt pairs define a refusal direction, which is then subtracted from activations or projected from weights. In video diffusion unlearning, low-rank SVD of the covariance-difference matrix isolates a robust direction minimizing collateral loss (Facchiano et al., 9 Jun 2025).

3. Empirical and Benchmark Results

Refusal-vector ablation on Llama 3.1 70B (Lermen et al., 2024) demonstrated:

Agent mode (Safe Agent Benchmark): harmful tasks—success rate increased from 18/28 to 26/28; explicit refusals reduced from 7 to 0 after ablation. Benign-task performance declined only slightly (21/24 to 19/24 correct).
Similar effects generalize across model scales.
Statistical test: refusal rate on harmful prompts dropped from ~25% (agent scaffold) to 0% ( $p\ll0.01$ ) by McNemar's test.

Fingerprinting via refusal vectors achieved 100% base-model identification accuracy among 76 offspring models; adversarial jailbreaks reduced but did not destroy the signal (cosine similarity to base fingerprint: ~0.5 versus inter-family ~0) (Xu et al., 10 Feb 2026).

In diffusion models, embedding the low-rank refusal vector permanently reduced the presence of unwanted video concepts: e.g., the "censorship rate" for nudity dropped from 44.7% to 13.4% with near-baseline visual fidelity (Facchiano et al., 9 Jun 2025).

4. Geometric and Mechanistic Insights

Empirical and theoretical analyses show:

One-dimensional subspace: In many safety-aligned LLMs, nearly all refusal behavior is mediated by a single activation-space direction, verified across 13 open-source models up to 72B parameters (Arditi et al., 2024).
Sparse vs. dense structure: Ridge-regularized variants (e.g., Surgical Refusal Ablation, Weighted Ridge-Mean Difference) reveal that naïvely defined refusal vectors are polysemantic—entangling safety, logical reasoning, and stylistic confounds ("Ghost Noise"). Spectral cleaning against hand-constructed "Concept Atoms" yields a cleaner, more causal direction, minimizing distribution drift and task degradation (Cristofano, 13 Jan 2026, García-Ferrero et al., 18 Dec 2025).
Universality: The same refusal direction transfers across languages ((Wang et al., 22 May 2025): 14 languages, pairwise vector cosine similarity remains high), across refusal framing categories, and across model architectures (zero-shot transfer) (Alagharu et al., 9 Mar 2026).
High-dimensional extensions: Gradient-based optimization discovers multi-dimensional cones of refusal directions, supporting several functionally independent refusal mechanisms; each independently controllable (Wollschläger et al., 24 Feb 2025).

Refusal Vector Property	Empirical Finding/Significance
Subspace dimensionality	1–5, model/setting dependent
Universality across languages	Verified in >14, directions nearly aligned
Robustness to model modification	Stable under SFT, LoRA, quant, not merges
Effect on refusal rate	~25%→0% on harmful prompts post-ablation
Distribution drift	Ridge/SRA: KL ≪ 0.05; naïve: KL ≫ 2.0

5. Safety, Limitations, and Defense Implications

Brittleness: A single rank-1 ablation can wholesale remove model safety guardrails, even in models with extensive safety fine-tuning (Lermen et al., 2024, Arditi et al., 2024).
Generalization gap: Safety alignment tested on chat completions does not assure robust agentic behavior in tool-using LLMs. Unmodified models already perform many harmful tasks under agent scaffolds (Lermen et al., 2024).
Polysemanticity: Naïve difference-in-means vectors may collateralize unrelated skills, degrading general capabilities. Surgical variants mitigate this (Cristofano, 13 Jan 2026, García-Ferrero et al., 18 Dec 2025).
Provenance tracing: The refusal vector serves as a highly robust behavioral fingerprint for model family provenance and tampering detection. Cosine similarity-based identification is stable under quantization and fine-tuning, less so after merges or severe jailbreaks (Xu et al., 10 Feb 2026).
Mitigations: Defenses include multi-vector (cone-based) interventions, residualization to protected concepts, robustness to dynamic ablation attacks via stochastic refusal-ablating retraining (Xie et al., 18 Sep 2025), and "self-destructing" or watermarking mechanisms (Lermen et al., 2024).

6. Practical Applications

White-box jailbreaks: Attacker can extract and ablate the refusal direction to defeat LLM safety guardrails (Lermen et al., 2024, Arditi et al., 2024).
Behavioral calibration: Fine-grained or conditional steering (e.g., by topic, by refusal type, or by agent role) is enabled by extracting multiple semantically targeted refusal directions (Alagharu et al., 9 Mar 2026, Lee et al., 2024, García-Ferrero et al., 18 Dec 2025).
Selective unlearning: Region-specific unlearning in generative diffusion/video models can be realized by low-rank refusal directions, enabling removal of unwanted capabilities (Facchiano et al., 9 Jun 2025).
Model auditing and red teaming: Extracted refusal directions expose hidden safety properties; inclusion in pre-release evaluation detects latent vulnerabilities (Lermen et al., 2024, Siu et al., 30 May 2025).
Safe fine-tuning: Refusal features enable filter-and-distill pipelines for user data, maintaining safety in downstream customized models (Ham et al., 9 Jun 2025).
Provenance and IP: Behaviorally anchored fingerprints survive most model modifications, supporting blackbox auditing for deployment and compliance (Xu et al., 10 Feb 2026).

7. Open Problems and Future Directions

Multilingual and domain generalization: While the universal axis hypothesis holds in major languages, robustness in low-resource or non-standard dialects remains an open field (Wang et al., 22 May 2025).
Dimensionality and intervention targeting: Determining minimal sufficient bases for robust safe behavior, as well as the best regularization strategies, is ongoing (Wollschläger et al., 24 Feb 2025, Cristofano, 13 Jan 2026).
Defense against direct ablation: Current models lack resilience to internal weight edits or activation-space attacks; integrating dispersed/multimodal refusal representations, dynamic basis adaptation, and "self-repair" mechanisms are top priorities (Xie et al., 18 Sep 2025, Abbas et al., 26 Apr 2025).
Practical calibration: Fine-tuning and architecture-specific heuristics are still needed for optimal layer/position choice and cross-domain transfer.
Standardized evaluation: Benchmarks extending chat/task scenarios to agents, multimodal systems, and sophisticated adversarial pipelines are needed to track real-world safety and compliance impact (Lermen et al., 2024, García-Ferrero et al., 18 Dec 2025).