Safety-Related Knowledge Neurons
- Safety-related knowledge neurons are sparse internal units in transformer models that mediate safety responses such as refusal and harmful content suppression.
- They are identified by contrasting harmful and benign prompt activations using metrics like activation gaps, gradient similarities, and causal intervention methods in MLP/FFN layers.
- Interventions on these neurons demonstrate both attack and defensive potential, highlighting their role as high-leverage control points in model safety alignment.
Searching arXiv for the cited papers to ground the article in current literature. Safety-related knowledge neurons are model-internal units whose activations, gradients, or causal interventions are disproportionately tied to safety behavior, including refusal, harmful-content suppression, and knowledge-preserving safety editing. The recent literature does not use a single fixed term: it variously refers to safety neurons, safety-related neurons, safety-critical neurons, knowledge-related neurons, refusal neurons, concept neurons, and cross-lingual shared safety neurons. Across these formulations, the common object is a sparse and operationally identifiable substructure inside a model—usually in transformer MLP or FFN layers, though some work studies attention units—that mediates whether harmful knowledge is expressed, refused, suppressed, or re-routed (Chen et al., 2024, Zhou et al., 29 Apr 2025, Wang et al., 12 Feb 2026, Kazemi et al., 8 May 2026).
1. Conceptual scope and historical development
The earliest safety-oriented neural-network work in this lineage did not yet study LLMs, but it introduced a precursor idea: “neuron-to-feature understandability,” namely associating individual neurons with conditions under which they activate, as part of a certification-oriented methodology for safety-critical applications (Cheng et al., 2017). In parallel, Trusted Neural Networks embedded first-order logic safety constraints into learning objectives through either a multi-headed model with a logic head or a constrained optimization with proximal updates, thereby treating parts of the network as carriers of rule-based safety knowledge, even though specific neurons were not isolated (Ghosh et al., 2018).
In contemporary LLM work, the term becomes more concrete. “Finding Safety Neurons” defines safety neurons operationally as MLP intermediate neurons whose activation patterns change the most between an SFT model and a DPO-aligned model during generation, and whose activations causally restore most of the aligned model’s safety when patched into a less aligned model (Chen et al., 2024). NeuRel-Attack instead treats “neurons responsible for safe alignment” as neurons whose activations and gradients differ between harmful and harmless prompts, and whose modification can remove refusal behavior (Zhou et al., 29 Apr 2025). SafeNeuron formalizes safety neurons through two statistics—Activation Effect Size and Safety Activation Shift—computed from unsafe versus safe prompt activations in GLU-FFN layers (Wang et al., 12 Feb 2026).
A further refinement separates different safety functions. “A Single Neuron Is Sufficient to Bypass Safety Alignment in LLMs” argues for a two-system view: refusal neurons gate whether harmful knowledge is expressed, whereas concept neurons encode the harmful knowledge itself (Kazemi et al., 8 May 2026). “Who Transfers Safety?” adds a multilingual distinction between monolingual safety neurons and cross-lingual shared safety neurons, the latter defined as the intersection of English and non-high-resource language safety-neuron sets (Zhang et al., 1 Feb 2026). “Precise Shield” extends the safety-neuron idea to VLLMs and defines safety neurons as FFN units that are selectively strong when the model produces a refusal to a harmful request but are not strongly active on benign multimodal tasks (Shi et al., 10 Apr 2026).
This literature therefore does not posit a single universal ontology. Rather, it offers several operational definitions of safety-related knowledge neurons, each tailored to a different question: causal restoration of refusal, harmful-knowledge unlearning, multilingual safety transfer, multimodal refusal alignment, or neuron-level attack surfaces.
2. Operational definitions and identification methodologies
Most papers identify safety-related neurons by contrasting harmful and benign conditions, but the scoring functions differ substantially. Some use direct activation statistics, others use gradients, pruning-style importance scores, similarity measures, or corpus-level causal shifts. The unit of analysis is also heterogeneous: most work targets MLP or FFN neurons, but cross-lingual transfer work defines “neurons” as rows or columns of attention matrices (Zhang et al., 1 Feb 2026).
| Paper | Primary scoring signal | Unit selected |
|---|---|---|
| NeuRel-Attack (Zhou et al., 29 Apr 2025) | Activation gap , gradient cosine similarity, harmful gradient magnitude | MLP neurons |
| CKU (Shi et al., 24 May 2025) | SNIP-style importance | MLP knowledge-related neurons |
| Finding Safety Neurons (Chen et al., 2024) | Generation-time RMS activation change between SFT and DPO | MLP intermediate neurons |
| SafeNeuron (Wang et al., 12 Feb 2026) | Activation Effect Size and Safety Activation Shift | GLU-FFN neurons |
| NLSR (Yi et al., 2024) | SVD-based safety-region projection and cross-model similarity | Safety-critical neurons in LoRA modules |
| PKE (Li et al., 2024) | Layer gradient magnitude , average toxicity , neuron activation change | Toxic parameter regions / neurons |
| Precise Shield (Shi et al., 10 Apr 2026) | Activation–influence score and benign-task subtraction | FFN safety neurons |
| SS-Neurons (Zhang et al., 1 Feb 2026) | Representational shift , top-0 contrast | Attention “neurons” |
| SafeTuning / safety knowledge neurons (Zhao et al., 1 Sep 2025) | Contribution 1, harmful-vs-benign set difference | MLP knowledge neurons |
| Single-neuron refusal attack (Kazemi et al., 8 May 2026) | Gradient–activation score on refusal log-odds | Individual MLP neurons |
In NeuRel-Attack, the identification pipeline begins with per-neuron mean activation
2
dataset means
3
and the activation gap
4
This is coupled to a gradient criterion, with cosine similarity
5
and the final selection rule
6
thereby treating low-similarity, high-harmful-gradient neurons as the core safety-alignment subset (Zhou et al., 29 Apr 2025).
CKU adopts a different premise. It does not first isolate “harmful neurons”; instead it localizes useful knowledge neurons by SNIP-style scoring,
7
selects the top 8 as a protected set 9, and zeros their gradients during harmful-knowledge unlearning (Shi et al., 24 May 2025). In this formulation, safety-related knowledge neurons are defined indirectly through a separation between “useful knowledge” to preserve and the complementary editable subspace in which harmful knowledge is unlearned.
SafeNeuron uses two explicit within-layer statistics. The Activation Effect Size is
0
and the Safety Activation Shift is
1
It then forms
2
with the final safety-neuron set
3
This makes the safety-neuron definition statistical and layer-local, rather than dependent on a specific attack or decoder direction (Wang et al., 12 Feb 2026).
A more mechanistic route appears in “Finding Safety Neurons,” which scores neurons by the RMS difference between SFT and DPO activations during generation and then validates them by dynamic activation patching (Chen et al., 2024). “Unraveling LLM Jailbreaks Through Safety Knowledge Neurons” instead starts from MLP contribution scores
4
contrasts harmful and benign corpora to define a refined safety-neuron set 5, and then projects aggregate safety-neuron activity into vocabulary space to obtain interpretable rejection and conformity directions (Zhao et al., 1 Sep 2025). “A Single Neuron Is Sufficient” goes even further toward minimal causality by defining a refusal log-odds loss and ranking individual MLP neurons with a gradient–activation score specifically designed to find units whose suppression reduces refusal behavior (Kazemi et al., 8 May 2026).
Taken together, these methods suggest that “safety-related knowledge neurons” are not a single discovered object but a family of operationally defined sparse internal features. What unifies them is not one scoring rule, but the repeated finding that harmful-versus-benign contrasts expose a small subset of high-leverage units.
3. Causal intervention and offensive manipulation
A defining feature of the field is that identification is almost always followed by causal intervention. This makes the literature less about passive interpretability than about mechanistic control.
“Finding Safety Neurons” showed that dynamic activation patching from a DPO model into an SFT or Base model can restore most of the aligned model’s safety using only about 6 of all MLP neurons, while random-neuron patching produces no safety improvement (Chen et al., 2024). This established the core causal template: if patched activations of a sparse subset reproduce aligned behavior, then that subset is not merely correlated with safety but functionally sufficient for much of it.
NeuRel-Attack converts this template into an offensive attack. After identifying safety-relevant neurons 7, it fine-tunes only those neurons: 8 using variants such as random-label training, gradient ascent on harmful prompts, and ascent–descent training. Its final MemFlex-Based neuron-local method updates only 9 of parameters yet reaches average ASR 0, with ASR peaking at 1 for 2 on both MaliciousInstruct and AdvBench (Zhou et al., 29 Apr 2025). The same paper reports that standard LoRA updates roughly 3 of parameters, whereas its neuron-local update uses less and still attains higher ASR, directly exposing safety alignment as a neuron-level attack surface.
The most extreme version of causal control appears in “A Single Neuron Is Sufficient to Bypass Safety Alignment in LLMs.” There, safety bypass is achieved by fixing one MLP neuron to a constant value,
4
or by an anchor-based intervention that scales suppression according to prompt-conditioned activation. Across seven models, the single-neuron intervention attains average ASR 5 with the LLM judge and 6 with Llama-Guard under constant intervention; the anchor-based variant preserves capability more effectively, with average MMLU drop 7 percentage points and average GSM8K drop 8 percentage points (Kazemi et al., 8 May 2026). The same paper distinguishes refusal neurons from concept neurons, and shows that adding a constant offset
9
to a single suicide concept neuron can induce suicide-themed content from otherwise benign prompts, thereby separating safety gating from harmful content representation.
“Unraveling LLM Jailbreaks Through Safety Knowledge Neurons” provides a different offensive lens. It defines harmful and benign aggregate safety vectors
0
from refined safety-neuron activations, derives a conformity direction 1 and rejection direction 2, and modifies the MLP output as
3
Using 4 and a calibrated token depth of 5, this steering achieves mean ASR higher than 6, with Vicuna reaching 7 ASR and Llama-2-7b-chat reaching 8 ASR on harmful prompts (Zhao et al., 1 Sep 2025). In effect, safety-neuron directions become a jailbreak interface.
These offensive studies converge on a stark conclusion. Safety behavior in current LLMs is often causally vulnerable to sparse internal manipulations that are far smaller than standard fine-tuning updates. This suggests that safety-related knowledge neurons are not merely explanatory artifacts; they are high-leverage control points.
4. Defensive alignment, unlearning, and realignment
Defensive work uses the same mechanistic premise in the opposite direction: locate the relevant neurons, then either protect them, redistribute them, or selectively edit surrounding subspaces.
CKU identifies high-importance MLP neurons 9 on a benign identification dataset and freezes them during harmful-knowledge unlearning by masking gradients: 0 With NLR 1, CKU reports the lowest GCG ASR at about 2 of neurons locked in each target MLP layer, and finds that unlearning in middle MLP layers 8–12 yields 3 safety improvement with only 4 drop in average accuracy for Llama2-7B-Chat (Shi et al., 24 May 2025). This implies that a large protected knowledge backbone can coexist with a smaller editable safety subspace.
SafeNeuron pursues a different objective: not just preserving existing safety neurons, but reducing reliance on sparse safety pathways. After identifying safety neurons with SAS and ES, it freezes them during DPO,
5
forcing the model to construct redundant safety representations elsewhere (Wang et al., 12 Feb 2026). Iterative experiments on Qwen2.5-7B show ASR decreasing from 6 to 7, then 8, then 9, while the identified safety-neuron count increases from 0 to 1. This suggests that safety can be redistributed across the network rather than simply concentrated in a fragile set.
NLSR is a training-free realignment method for harmful fine-tuning. It constructs a super-aligned reference model, identifies safety-critical neurons by SVD-based low-rank approximation, compares reference and fine-tuned safety regions by
2
and selectively transplants corrupted safety regions back into the fine-tuned model (Yi et al., 2024). On SST-2 under harmful fine-tuning, it reduces Harmfulness Score from 3 to 4 on average while maintaining Fine-tuning Accuracy around 5; across SFT, DPO, ORPO, KTO, and SimPO, it reduces average HS from 6 to 7. This treats safety-related knowledge neurons as patchable reference-state components.
Precision Knowledge Editing uses multi-instance toxicity attribution, selects a key layer
8
chooses neurons with large 9, and optimizes
0
to reduce toxicity while preserving correctness (Li et al., 2024). It reduces Llama-3-8b-instruct baseline ASR from 1 to 2 and adaptive ASR from 3 to 4, while keeping Winrate essentially unchanged at 5. PKE therefore reframes safety-related neurons as toxic-parameter regions to be precisely edited rather than merely detected.
Several methods combine identification with parameter-efficient safety tuning. Precise Shield first identifies safety-specific FFN neurons by contrasting harmful-refusal inputs against benign MM-Bench inputs, then updates only those rows through masked LoRA,
6
The total trainable footprint is less than 7 of parameters, yet it reduces Llama Image-Dominant ASR from 8 to 9, Qwen Image-Dominant ASR from 0 to 1, and often outperforms standard LoRA, which updates about 2 of parameters (Shi et al., 10 Apr 2026). “Who Transfers Safety?” similarly shows that updating only English monolingual safety neurons with a binary gradient mask,
3
can outperform or match full fine-tuning while updating about 4 of parameters, substantially improving non-high-resource language safety (Zhang et al., 1 Feb 2026).
SafeTuning turns safety-knowledge-neuron analysis into a conventional defense. It first derives refusal-oriented safety directions from harmful and benign corpora, uses the rejection direction to generate refusal-style safety data, and then fine-tunes only the corresponding safety-neuron weights with
5
On Vicuna-7b-v1.5, SafeTuning lowers GCG ASR from 6 to 7, Pair ASR from 8 to 9, Prompt+RS ASR from 0 to 1, and AIM ASR from 2 to 3, while preserving a 4 AlpacaEval win rate (Zhao et al., 1 Sep 2025).
This defensive literature suggests that safety-related knowledge neurons can be treated in three nonexclusive ways: as units to protect, as units to redistribute away from brittle concentration, or as privileged edit sites for low-footprint safety improvement.
5. Representation structure: sparsity, localization, transfer, and entanglement
A recurring empirical result is extreme sparsity. “Finding Safety Neurons” reports that about 5 of all MLP neurons suffice to restore 6 safety performance (Chen et al., 2024). NeuRel-Attack attains average ASR 7 after editing only 8 of parameters (Zhou et al., 29 Apr 2025). Precise Shield updates fewer than 9 of parameters and still yields the best ASR in most model-and-risk settings (Shi et al., 10 Apr 2026). SS-Neurons are described as a tiny fraction of all neurons, less than 00, yet their masking produces large multilingual safety drops (Zhang et al., 1 Feb 2026). The single-neuron bypass result shows that in some settings the relevant scale collapses from thousands of units to one (Kazemi et al., 8 May 2026).
The literature is less uniform on where safety-related knowledge neurons are located. CKU finds that modifying only MLP layers works better than modifying all layers or only non-MLP layers, and that middle MLP layers 8–12 are especially effective targets (Shi et al., 24 May 2025). Precise Shield reports that safety neurons mainly appear in middle-to-late layers, increasing with depth, peaking around layers 20–30, and declining after about layer 32 (Shi et al., 10 Apr 2026). SafeNeuron also finds that safety neurons are sparse in early layers and concentrated in middle and late layers, with core and shared safety neurons forming the majority in deeper layers (Wang et al., 12 Feb 2026). By contrast, the single-neuron refusal attack often identifies early-to-mid MLP layers rather than only the deepest ones (Kazemi et al., 8 May 2026). This suggests that localization depends on both the operational definition and the intervention objective.
Transfer results imply that at least part of the safety subspace is reusable. Safety neurons found on HH-Harmless transfer across Beavertails, RedTeam, HarmBench, and JailbreakLLMs (Chen et al., 2024). Precise Shield reports moderate cross-lingual overlap, with Romance and Germanic languages sharing about 01 of safety neurons, Chinese sharing about 02 with Indo-European languages, and image-risk versus text-risk safety neurons overlapping by about 03 within each language (Shi et al., 10 Apr 2026). “Who Transfers Safety?” defines SS-Neurons as
04
and shows that masking them causes concurrent safety drops across non-high-resource languages, while their number is inversely correlated with ASR (Zhang et al., 1 Feb 2026). These results suggest a shared safety manifold across languages and modalities, but not a universally identical one.
Entanglement remains central. CKU explicitly states that useful knowledge is sufficiently localized for locking, but also that there is no single “kill switch neuron” and that representations remain distributed (Shi et al., 24 May 2025). SafeNeuron finds substantial overlap between safety and helpfulness-relevant neurons, interpreting safety as a stable and shared internal representation rather than a purely isolated module (Wang et al., 12 Feb 2026). “Finding Safety Neurons” sharpens this by showing that safety and helpfulness neurons significantly overlap, yet require different activation patterns of the shared neurons, providing a mechanistic account of “alignment tax” (Chen et al., 2024). AdaCultureSafe extends the entanglement question to cultural safety and argues that cultural knowledge and cultural safety are currently implemented in largely distinct circuits: cultural knowledge is more specialized, while cultural safety is more shared and post-alignment-driven, yielding near-zero or weakly negative Spearman correlation between cultural knowledge accuracy and cultural respect (Kang et al., 9 Mar 2026). This suggests that “safety-related knowledge neurons” may exist for some tasks only as partial couplings between knowledge and policy, not as a unified safety-knowledge substrate.
A plausible implication is that current safety mechanisms are simultaneously sparse enough to be editable and distributed enough to resist trivial semantic labeling. This tension explains why some papers find highly local attack surfaces while others emphasize partial disentanglement rather than complete modularity.
6. Limitations, controversies, and future directions
The field is marked by a productive disagreement over how localized safety really is. CKU argues that there is no single “kill switch neuron” for safety and that harmful behavior can be manipulated through a relatively small editable subspace (Shi et al., 24 May 2025). By contrast, the single-neuron bypass paper reports that suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests (Kazemi et al., 8 May 2026). This is not a trivial contradiction: the papers define different objects, use different interventions, and ask different causal questions. One studies knowledge retention and constrained unlearning under MLP-layer editing; the other studies the existence of highly sufficient gating neurons for bypass. The literature therefore supports neither pure modularity nor pure diffuseness, but a spectrum of localized safety mechanisms.
A second limitation concerns coverage. Many methods rely on harmful and harmless prompt sets such as AdvBench, MaliciousInstruct, HH-RLHF-Harmless, Beavertails, or translated multilingual benchmarks. Several papers explicitly note incomplete out-of-distribution characterization, dependence on curated datasets, or limited attack diversity (Zhou et al., 29 Apr 2025, Li et al., 2024, Shi et al., 10 Apr 2026). AdaCultureSafe shows that even when cultural knowledge is strong, cultural safety can remain weak, which indicates that generic safety-neuron discovery may miss domain-specific or culturally grounded safety behavior (Kang et al., 9 Mar 2026).
A third issue is architectural scope. Most work studies MLP or FFN neurons, often motivated by “FFN-as-key–value memory,” while attention heads, cross-modal connectors, and residual-stream features are less systematically covered (Shi et al., 24 May 2025, Shi et al., 10 Apr 2026). The cross-lingual SS-Neuron paper is a notable exception because it studies attention-neuron rows and columns, and the single-neuron bypass paper explicitly shows that single residual-stream features are much weaker control levers than MLP neurons (Zhang et al., 1 Feb 2026, Kazemi et al., 8 May 2026). This suggests that the unit of analysis matters, and that current conclusions may be specific to the chosen basis.
A fourth issue is the dual-use character of the research. NeuRel-Attack, the single-neuron bypass work, and SafeTuning all show that the same neuron-level understanding can enable either defense or attack (Zhou et al., 29 Apr 2025, Kazemi et al., 8 May 2026, Zhao et al., 1 Sep 2025). Several papers therefore propose auxiliary defenses such as watermarking, fingerprinting, hardcoded external safety layers, periodic auditing of safety neurons, or more distributed alignment objectives (Zhou et al., 29 Apr 2025). Earlier certification-oriented work offers a complementary trajectory: formal verification of safety properties and explicit logic constraints remain relevant precisely because internal neuron-level safety can be sparse and fragile (Cheng et al., 2017, Ghosh et al., 2018).
Future research directions are comparatively consistent. The literature repeatedly calls for better causal discovery of harmful and safety circuits, stronger integration of mechanistic interpretability with alignment training, greater multilingual and multimodal coverage, and methods that make safety harder to localize and surgically disable (Wang et al., 12 Feb 2026, Shi et al., 10 Apr 2026, Zhang et al., 1 Feb 2026). A plausible implication is that robust safety may require both internal and external measures: internal redistribution or auditing of safety-related knowledge neurons, and external safeguards that do not depend solely on a sparse internal refusal mechanism.
In present usage, then, safety-related knowledge neurons denote neither a settled taxonomy nor a single mechanistic hypothesis. They designate a growing class of empirically discovered internal units that mediate safety behavior strongly enough to be ranked, patched, frozen, amplified, masked, transplanted, or retrained. The cumulative evidence indicates that modern aligned models do contain sparse, high-leverage safety subspaces, but that the exact grain of that sparsity—single neuron, small cluster, editable subspace, or shared multilingual manifold—depends on the task, architecture, identification criterion, and intervention goal.