---
title: Safety-Related Knowledge Neurons
url: https://www.emergentmind.com/topics/safety-related-knowledge-neurons
type: topic
---

# Safety-Related Knowledge Neurons

Searching arXiv for the cited papers to ground the article in current literature.
Safety-related knowledge neurons are model-internal units whose activations, gradients, or causal interventions are disproportionately tied to safety behavior, including refusal, harmful-content suppression, and knowledge-preserving safety editing. The recent literature does not use a single fixed term: it variously refers to safety neurons, safety-related neurons, safety-critical neurons, knowledge-related neurons, refusal neurons, concept neurons, and cross-lingual shared safety neurons. Across these formulations, the common object is a sparse and operationally identifiable substructure inside a model—usually in transformer MLP or FFN layers, though some work studies attention units—that mediates whether harmful knowledge is expressed, refused, suppressed, or re-routed [2406.14144; 2504.21053; 2602.12158; 2605.08513].

## 1. Conceptual scope and historical development

The earliest safety-oriented neural-network work in this lineage did not yet study large language models, but it introduced a precursor idea: “neuron-to-feature understandability,” namely associating individual neurons with conditions under which they activate, as part of a certification-oriented methodology for safety-critical applications [1709.00911]. In parallel, Trusted Neural Networks embedded first-order logic safety constraints into learning objectives through either a multi-headed model with a logic head or a constrained optimization with proximal updates, thereby treating parts of the network as carriers of rule-based safety knowledge, even though specific neurons were not isolated [1805.07075].

In contemporary LLM work, the term becomes more concrete. “Finding Safety Neurons” defines safety neurons operationally as MLP intermediate neurons whose activation patterns change the most between an SFT model and a DPO-aligned model during generation, and whose activations causally restore most of the aligned model’s safety when patched into a less aligned model [2406.14144]. NeuRel-Attack instead treats “neurons responsible for safe alignment” as neurons whose activations and gradients differ between harmful and harmless prompts, and whose modification can remove refusal behavior [2504.21053]. SafeNeuron formalizes safety neurons through two statistics—Activation Effect Size and Safety Activation Shift—computed from unsafe versus safe prompt activations in GLU-FFN layers [2602.12158].

A further refinement separates different safety functions. “A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models” argues for a two-system view: refusal neurons gate whether harmful knowledge is expressed, whereas concept neurons encode the harmful knowledge itself [2605.08513]. “Who Transfers Safety?” adds a multilingual distinction between monolingual safety neurons and cross-lingual shared safety neurons, the latter defined as the intersection of English and non-high-resource language safety-neuron sets [2602.01283]. “Precise Shield” extends the safety-neuron idea to VLLMs and defines safety neurons as FFN units that are selectively strong when the model produces a refusal to a harmful request but are not strongly active on benign multimodal tasks [2604.08881].

This literature therefore does not posit a single universal ontology. Rather, it offers several operational definitions of safety-related knowledge neurons, each tailored to a different question: causal restoration of refusal, harmful-knowledge unlearning, multilingual safety transfer, multimodal refusal alignment, or neuron-level attack surfaces.

## 2. Operational definitions and identification methodologies

Most papers identify safety-related neurons by contrasting harmful and benign conditions, but the scoring functions differ substantially. Some use direct activation statistics, others use gradients, pruning-style importance scores, similarity measures, or corpus-level causal shifts. The unit of analysis is also heterogeneous: most work targets MLP or FFN neurons, but cross-lingual transfer work defines “neurons” as rows or columns of attention matrices [2602.01283].

| Paper | Primary scoring signal | Unit selected |
|---|---|---|
| NeuRel-Attack [2504.21053] | Activation gap \(\Delta_{i,j}\), gradient cosine similarity, harmful gradient magnitude | MLP neurons |
| CKU [2505.18588] | SNIP-style importance \(I(W)=\mathbb{E}_{x\sim D}|W\odot \nabla_W \mathcal{L}(x)|\) | MLP knowledge-related neurons |
| Finding Safety Neurons [2406.14144] | Generation-time RMS activation change between SFT and DPO | MLP intermediate neurons |
| SafeNeuron [2602.12158] | Activation Effect Size \(d_j^l\) and Safety Activation Shift \(\Delta a_j^l\) | GLU-FFN neurons |
| NLSR [2412.12497] | SVD-based safety-region projection and cross-model similarity \(S_j\) | Safety-critical neurons in LoRA modules |
| PKE [2410.03772] | Layer gradient magnitude \(\Delta g_l\), average toxicity \(\overline{T}_l\), neuron activation change \(\Delta h_{l,j}\) | Toxic parameter regions / neurons |
| Precise Shield [2604.08881] | Activation–influence score \(\mathcal{I}_i^{(\ell)}(\mathcal{D})\) and benign-task subtraction | FFN safety neurons |
| SS-Neurons [2602.01283] | Representational shift \(A_{\text{LLM}}(x,N)\), top-\(p\%\) contrast | Attention “neurons” |
| SafeTuning / safety knowledge neurons [2509.01631] | Contribution \(C_{li}=a_{li}\|R_{li}\|\), harmful-vs-benign set difference | MLP knowledge neurons |
| Single-neuron refusal attack [2605.08513] | Gradient–activation score on refusal log-odds | Individual MLP neurons |

In NeuRel-Attack, the identification pipeline begins with per-neuron mean activation
\[
\bar{a}_{i,j}(x)=\frac{1}{T}\sum_{t=1}^{T} a_{i,j}(x_t),
\]
dataset means
\[
\mu_{i,j}^h=\mathbb{E}_{x\sim \mathcal{X}_h}[\bar{a}_{i,j}(x)], \qquad
\mu_{i,j}^s=\mathbb{E}_{x\sim \mathcal{X}_s}[\bar{a}_{i,j}(x)],
\]
and the activation gap
\[
\Delta_{i,j}=|\mu_{i,j}^h-\mu_{i,j}^s|.
\]
This is coupled to a gradient criterion, with cosine similarity
\[
\text{sim}_{i,j}=\frac{G_h^{(i,j)}\cdot G_s^{(i,j)}}{\|G_h^{(i,j)}\|\|G_s^{(i,j)}\|},
\]
and the final selection rule
\[
\mathcal{N}=\{(i,j)\mid \text{sim}_{i,j}<\mu,\ \bar{g}_{i,j}^h>\sigma\},
\]
thereby treating low-similarity, high-harmful-gradient neurons as the core safety-alignment subset [2504.21053].

CKU adopts a different premise. It does not first isolate “harmful neurons”; instead it localizes useful knowledge neurons by SNIP-style scoring,
\[
I(W,x)=|W\odot \nabla_W \mathcal{L}(x)|,\qquad
I(W)=\mathbb{E}_{x\sim D}|W\odot \nabla_W \mathcal{L}(x)|,
\]
selects the top \(p\%\) as a protected set \(U\), and zeros their gradients during harmful-knowledge unlearning [2505.18588]. In this formulation, safety-related knowledge neurons are defined indirectly through a separation between “useful knowledge” to preserve and the complementary editable subspace in which harmful knowledge is unlearned.

SafeNeuron uses two explicit within-layer statistics. The Activation Effect Size is
\[
d_j^l=\frac{\bar{a}_{j}^{\text{unsafe}}-\bar{a}_{j}^{\text{safe}}}{s_j^l},
\]
and the Safety Activation Shift is
\[
\Delta a_j^l=\bar{a}_{j}^{\text{unsafe}}-\bar{a}_{j}^{\text{safe}}.
\]
It then forms
\[
\mathrm{SN}_{\text{ES}}=\{a_j\mid d_j^l>\tau_{ES}\},\qquad
\mathrm{SN}_{\text{SAS}}=\{a_j\mid z(\Delta a_j^l)>\tau_{SAS},\ \Delta a_j^l>0\},
\]
with the final safety-neuron set
\[
\mathrm{SN}=\mathrm{SN}_{\text{ES}}\cup \mathrm{SN}_{\text{SAS}}.
\]
This makes the safety-neuron definition statistical and layer-local, rather than dependent on a specific attack or decoder direction [2602.12158].

A more mechanistic route appears in “Finding Safety Neurons,” which scores neurons by the RMS difference between SFT and DPO activations during generation and then validates them by dynamic activation patching [2406.14144]. “Unraveling LLM Jailbreaks Through Safety Knowledge Neurons” instead starts from MLP contribution scores
\[
C_{li}=a_{li}\cdot \|N_{li}\|,
\]
contrasts harmful and benign corpora to define a refined safety-neuron set \(\mathcal{N}_r=\mathcal{N}_s\setminus \mathcal{N}_f\), and then projects aggregate safety-neuron activity into vocabulary space to obtain interpretable rejection and conformity directions [2509.01631]. “A Single Neuron Is Sufficient” goes even further toward minimal causality by defining a refusal log-odds loss and ranking individual MLP neurons with a gradient–activation score specifically designed to find units whose suppression reduces refusal behavior [2605.08513].

Taken together, these methods suggest that “safety-related knowledge neurons” are not a single discovered object but a family of operationally defined sparse internal features. What unifies them is not one scoring rule, but the repeated finding that harmful-versus-benign contrasts expose a small subset of high-leverage units.

## 3. Causal intervention and offensive manipulation

A defining feature of the field is that identification is almost always followed by causal intervention. This makes the literature less about passive interpretability than about mechanistic control.

“Finding Safety Neurons” showed that dynamic activation patching from a DPO model into an SFT or Base model can restore most of the aligned model’s safety using only about \(5\%\) of all MLP neurons, while random-neuron patching produces no safety improvement [2406.14144]. This established the core causal template: if patched activations of a sparse subset reproduce aligned behavior, then that subset is not merely correlated with safety but functionally sufficient for much of it.

NeuRel-Attack converts this template into an offensive attack. After identifying safety-relevant neurons \(\mathcal{N}\), it fine-tunes only those neurons:
\[
\theta'_{i,j}=
\begin{cases}
\theta_{i,j}+\eta\nabla_{\theta}\mathcal{L}, & (i,j)\in \mathcal{N},\\
\theta_{i,j}, & \text{otherwise},
\end{cases}
\]
using variants such as random-label training, gradient ascent on harmful prompts, and ascent–descent training. Its final MemFlex-Based neuron-local method updates only \(0.05\%-0.13\%\) of parameters yet reaches average ASR \(96\%\), with ASR peaking at \(100\%\) for \(\mu=0.85\) on both MaliciousInstruct and AdvBench [2504.21053]. The same paper reports that standard LoRA updates roughly \(0.28\%-0.30\%\) of parameters, whereas its neuron-local update uses less and still attains higher ASR, directly exposing safety alignment as a neuron-level attack surface.

The most extreme version of causal control appears in “A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models.” There, safety bypass is achieved by fixing one MLP neuron to a constant value,
\[
h_i\leftarrow m,
\]
or by an anchor-based intervention that scales suppression according to prompt-conditioned activation. Across seven models, the single-neuron intervention attains average ASR \(91.7\%\) with the LLM judge and \(91.9\%\) with Llama-Guard under constant intervention; the anchor-based variant preserves capability more effectively, with average MMLU drop \(-0.6\) percentage points and average GSM8K drop \(-0.1\) percentage points [2605.08513]. The same paper distinguishes refusal neurons from concept neurons, and shows that adding a constant offset
\[
h_i\leftarrow h_i+m
\]
to a single suicide concept neuron can induce suicide-themed content from otherwise benign prompts, thereby separating safety gating from harmful content representation.

“Unraveling LLM Jailbreaks Through Safety Knowledge Neurons” provides a different offensive lens. It defines harmful and benign aggregate safety vectors
\[
sv_H,\ sv_B
\]
from refined safety-neuron activations, derives a conformity direction \(d_c=sv_B-sv_H\) and rejection direction \(d_r=sv_H-sv_B\), and modifies the MLP output as
\[
E'_{l+1}=F(X_lW_{l1})W_{l2}+\alpha d.
\]
Using \(\alpha=3\) and a calibrated token depth of \(5\), this steering achieves mean ASR higher than \(97\%\), with Vicuna reaching \(100\%\) ASR and Llama-2-7b-chat reaching \(99\%\) ASR on harmful prompts [2509.01631]. In effect, safety-neuron directions become a jailbreak interface.

These offensive studies converge on a stark conclusion. Safety behavior in current LLMs is often causally vulnerable to sparse internal manipulations that are far smaller than standard fine-tuning updates. This suggests that safety-related knowledge neurons are not merely explanatory artifacts; they are high-leverage control points.

## 4. Defensive alignment, unlearning, and realignment

Defensive work uses the same mechanistic premise in the opposite direction: locate the relevant neurons, then either protect them, redistribute them, or selectively edit surrounding subspaces.

CKU identifies high-importance MLP neurons \(U\) on a benign identification dataset and freezes them during harmful-knowledge unlearning by masking gradients:
\[
\nabla_W^{\text{eff}}\mathcal{L}_f=M\odot \nabla_W\mathcal{L}_f,
\qquad
W\leftarrow W+\eta(M\odot \nabla_W\mathcal{L}_f).
\]
With NLR \(\approx 0.8\), CKU reports the lowest GCG ASR at about \(80\%\) of neurons locked in each target MLP layer, and finds that unlearning in middle MLP layers 8–12 yields \(\sim 4\times\) safety improvement with only \(\sim 0.15\%\) drop in average accuracy for Llama2-7B-Chat [2505.18588]. This implies that a large protected knowledge backbone can coexist with a smaller editable safety subspace.

SafeNeuron pursues a different objective: not just preserving existing safety neurons, but reducing reliance on sparse safety pathways. After identifying safety neurons with SAS and ES, it freezes them during DPO,
\[
\min_{\theta_r}\mathcal{L}_{DPO}(\theta_s,\theta_r)
\quad \text{s.t.}\quad \theta_s=\text{const.},
\]
forcing the model to construct redundant safety representations elsewhere [2602.12158]. Iterative experiments on Qwen2.5-7B show ASR decreasing from \(279/313\) to \(183/313\), then \(169/313\), then \(158/313\), while the identified safety-neuron count increases from \(3717\) to \(5218\). This suggests that safety can be redistributed across the network rather than simply concentrated in a fragile set.

NLSR is a training-free realignment method for harmful fine-tuning. It constructs a super-aligned reference model, identifies safety-critical neurons by SVD-based low-rank approximation, compares reference and fine-tuned safety regions by
\[
S_j=\frac{\langle W'_{e,j},W'_{t,j}\rangle_F}{\|W'_{e,j}\|_F\|W'_{t,j}\|_F},
\]
and selectively transplants corrupted safety regions back into the fine-tuned model [2412.12497]. On SST-2 under harmful fine-tuning, it reduces Harmfulness Score from \(61.1\%\) to \(22.8\%\) on average while maintaining Fine-tuning Accuracy around \(95.1\%\); across SFT, DPO, ORPO, KTO, and SimPO, it reduces average HS from \(56.6\%\) to \(27.1\%\). This treats safety-related knowledge neurons as patchable reference-state components.

Precision Knowledge Editing uses multi-instance toxicity attribution, selects a key layer
\[
l^*=\arg\max_l \Delta g_l,
\]
chooses neurons with large \(\Delta h_{l^*,j}\), and optimizes
\[
L=\alpha T(\hat{y})+\beta C(\hat{y})
\]
to reduce toxicity while preserving correctness [2410.03772]. It reduces Llama-3-8b-instruct baseline ASR from \(11.37\%\) to \(4.5\%\) and adaptive ASR from \(30.5\%\) to \(3\%\), while keeping Winrate essentially unchanged at \(22.7\%\). PKE therefore reframes safety-related neurons as toxic-parameter regions to be precisely edited rather than merely detected.

Several methods combine identification with parameter-efficient safety tuning. Precise Shield first identifies safety-specific FFN neurons by contrasting harmful-refusal inputs against benign MM-Bench inputs, then updates only those rows through masked LoRA,
\[
\Delta W=(M\odot B)A.
\]
The total trainable footprint is less than \(0.03\%\) of parameters, yet it reduces Llama Image-Dominant ASR from \(23.61\) to \(4.21\), Qwen Image-Dominant ASR from \(30.71\) to \(6.19\), and often outperforms standard LoRA, which updates about \(0.11\%\) of parameters [2604.08881]. “Who Transfers Safety?” similarly shows that updating only English monolingual safety neurons with a binary gradient mask,
\[
\theta_{t+1}=\theta_t-\eta\big(M_{\text{mask}}\odot \nabla_\theta \mathcal{L}(D_{\text{parallel}})\big),
\]
can outperform or match full fine-tuning while updating about \(0.51\%-0.57\%\) of parameters, substantially improving non-high-resource language safety [2602.01283].

SafeTuning turns safety-knowledge-neuron analysis into a conventional defense. It first derives refusal-oriented safety directions from harmful and benign corpora, uses the rejection direction to generate refusal-style safety data, and then fine-tunes only the corresponding safety-neuron weights with
\[
\mathcal{L}=-\log P(Y_{\text{refuse}}\mid X_{\text{harm}}).
\]
On Vicuna-7b-v1.5, SafeTuning lowers GCG ASR from \(33\%\) to \(0\%\), Pair ASR from \(66\%\) to \(5\%\), Prompt+RS ASR from \(95\%\) to \(13\%\), and AIM ASR from \(68\%\) to \(0\%\), while preserving a \(54.1\%\) AlpacaEval win rate [2509.01631].

This defensive literature suggests that safety-related knowledge neurons can be treated in three nonexclusive ways: as units to protect, as units to redistribute away from brittle concentration, or as privileged edit sites for low-footprint safety improvement.

## 5. Representation structure: sparsity, localization, transfer, and entanglement

A recurring empirical result is extreme sparsity. “Finding Safety Neurons” reports that about \(5\%\) of all MLP neurons suffice to restore \(90\%\) safety performance [2406.14144]. NeuRel-Attack attains average ASR \(96\%\) after editing only \(0.05\%-0.13\%\) of parameters [2504.21053]. Precise Shield updates fewer than \(0.03\%\) of parameters and still yields the best ASR in most model-and-risk settings [2604.08881]. SS-Neurons are described as a tiny fraction of all neurons, less than \(0.3\%\), yet their masking produces large multilingual safety drops [2602.01283]. The single-neuron bypass result shows that in some settings the relevant scale collapses from thousands of units to one [2605.08513].

The literature is less uniform on where safety-related knowledge neurons are located. CKU finds that modifying only MLP layers works better than modifying all layers or only non-MLP layers, and that middle MLP layers 8–12 are especially effective targets [2505.18588]. Precise Shield reports that safety neurons mainly appear in middle-to-late layers, increasing with depth, peaking around layers 20–30, and declining after about layer 32 [2604.08881]. SafeNeuron also finds that safety neurons are sparse in early layers and concentrated in middle and late layers, with core and shared safety neurons forming the majority in deeper layers [2602.12158]. By contrast, the single-neuron refusal attack often identifies early-to-mid MLP layers rather than only the deepest ones [2605.08513]. This suggests that localization depends on both the operational definition and the intervention objective.

Transfer results imply that at least part of the safety subspace is reusable. Safety neurons found on HH-Harmless transfer across Beavertails, RedTeam, HarmBench, and JailbreakLLMs [2406.14144]. Precise Shield reports moderate cross-lingual overlap, with Romance and Germanic languages sharing about \(55\%-60\%\) of safety neurons, Chinese sharing about \(30\%-40\%\) with Indo-European languages, and image-risk versus text-risk safety neurons overlapping by about \(70\%-80\%\) within each language [2604.08881]. “Who Transfers Safety?” defines SS-Neurons as
\[
SS_\ell=MS_\ell\cap MS_{\text{English}},
\]
and shows that masking them causes concurrent safety drops across non-high-resource languages, while their number is inversely correlated with ASR [2602.01283]. These results suggest a shared safety manifold across languages and modalities, but not a universally identical one.

Entanglement remains central. CKU explicitly states that useful knowledge is sufficiently localized for locking, but also that there is no single “kill switch neuron” and that representations remain distributed [2505.18588]. SafeNeuron finds substantial overlap between safety and helpfulness-relevant neurons, interpreting safety as a stable and shared internal representation rather than a purely isolated module [2602.12158]. “Finding Safety Neurons” sharpens this by showing that safety and helpfulness neurons significantly overlap, yet require different activation patterns of the shared neurons, providing a mechanistic account of “alignment tax” [2406.14144]. AdaCultureSafe extends the entanglement question to cultural safety and argues that cultural knowledge and cultural safety are currently implemented in largely distinct circuits: cultural knowledge is more specialized, while cultural safety is more shared and post-alignment-driven, yielding near-zero or weakly negative Spearman correlation between cultural knowledge accuracy and cultural respect [2603.08275]. This suggests that “safety-related knowledge neurons” may exist for some tasks only as partial couplings between knowledge and policy, not as a unified safety-knowledge substrate.

A plausible implication is that current safety mechanisms are simultaneously sparse enough to be editable and distributed enough to resist trivial semantic labeling. This tension explains why some papers find highly local attack surfaces while others emphasize partial disentanglement rather than complete modularity.

## 6. Limitations, controversies, and future directions

The field is marked by a productive disagreement over how localized safety really is. CKU argues that there is no single “kill switch neuron” for safety and that harmful behavior can be manipulated through a relatively small editable subspace [2505.18588]. By contrast, the single-neuron bypass paper reports that suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests [2605.08513]. This is not a trivial contradiction: the papers define different objects, use different interventions, and ask different causal questions. One studies knowledge retention and constrained unlearning under MLP-layer editing; the other studies the existence of highly sufficient gating neurons for bypass. The literature therefore supports neither pure modularity nor pure diffuseness, but a spectrum of localized safety mechanisms.

A second limitation concerns coverage. Many methods rely on harmful and harmless prompt sets such as AdvBench, MaliciousInstruct, HH-RLHF-Harmless, Beavertails, or translated multilingual benchmarks. Several papers explicitly note incomplete out-of-distribution characterization, dependence on curated datasets, or limited attack diversity [2504.21053; 2410.03772; 2604.08881]. AdaCultureSafe shows that even when cultural knowledge is strong, cultural safety can remain weak, which indicates that generic safety-neuron discovery may miss domain-specific or culturally grounded safety behavior [2603.08275].

A third issue is architectural scope. Most work studies MLP or FFN neurons, often motivated by “FFN-as-key–value memory,” while attention heads, cross-modal connectors, and residual-stream features are less systematically covered [2505.18588; 2604.08881]. The cross-lingual SS-Neuron paper is a notable exception because it studies attention-neuron rows and columns, and the single-neuron bypass paper explicitly shows that single residual-stream features are much weaker control levers than MLP neurons [2602.01283; 2605.08513]. This suggests that the unit of analysis matters, and that current conclusions may be specific to the chosen basis.

A fourth issue is the dual-use character of the research. NeuRel-Attack, the single-neuron bypass work, and SafeTuning all show that the same neuron-level understanding can enable either defense or attack [2504.21053; 2605.08513; 2509.01631]. Several papers therefore propose auxiliary defenses such as watermarking, fingerprinting, hardcoded external safety layers, periodic auditing of safety neurons, or more distributed alignment objectives [2504.21053]. Earlier certification-oriented work offers a complementary trajectory: formal verification of safety properties and explicit logic constraints remain relevant precisely because internal neuron-level safety can be sparse and fragile [1709.00911; 1805.07075].

Future research directions are comparatively consistent. The literature repeatedly calls for better causal discovery of harmful and safety circuits, stronger integration of mechanistic interpretability with alignment training, greater multilingual and multimodal coverage, and methods that make safety harder to localize and surgically disable [2602.12158; 2604.08881; 2602.01283]. A plausible implication is that robust safety may require both internal and external measures: internal redistribution or auditing of safety-related knowledge neurons, and external safeguards that do not depend solely on a sparse internal refusal mechanism.

In present usage, then, safety-related knowledge neurons denote neither a settled taxonomy nor a single mechanistic hypothesis. They designate a growing class of empirically discovered internal units that mediate safety behavior strongly enough to be ranked, patched, frozen, amplified, masked, transplanted, or retrained. The cumulative evidence indicates that modern aligned models do contain sparse, high-leverage safety subspaces, but that the exact grain of that sparsity—single neuron, small cluster, editable subspace, or shared multilingual manifold—depends on the task, architecture, identification criterion, and intervention goal.

Source: https://www.emergentmind.com/topics/safety-related-knowledge-neurons