NeuroBreak: LLM Jailbreak Analysis

• NeuroBreak is a visual analytics system that defines internal LLM vulnerabilities by probing neuron- and layer-level dynamics underlying jailbreak attacks.
• It employs layer-wise representation probing and gradient-based attribution to reveal toxic semantic shifts and identify dedicated safety neurons with high precision.
• By enabling targeted fine-tuning of less than 0.2% of parameters, NeuroBreak improves LLM safety without compromising overall performance.

NeuroBreak is a visual analytics system developed for the mechanistic analysis of jailbreak attacks in LLMs, focusing on the neuron- and layer-level dynamics that underlie vulnerabilities and safety enforcement. Its central innovation lies in moving beyond output-level analyses, providing an integrated platform to probe, attribute, and intervene on internal neural mechanisms that mediate both harmful and protective behaviors under adversarial prompting.

1. System Overview and Objectives

NeuroBreak is designed to address escalating challenges in AI safety induced by jailbreak techniques—prompt engineering methods that bypass LLM safety alignment protocols and elicit illegal or unethical outputs. The approach is explicitly top-down: starting from global model behaviors under attack, then progressing to detailed layer-wise and neuron-level analyses to reveal how the semantic substrate of harmful content is constructed internally. System requirements were defined in consultation with AI security experts to ensure relevance to emerging threat landscapes.

The system's principal objectives are to identify neuron-level failure modes, reveal the semantic evolution during generation, and facilitate targeted hardening of defense mechanisms without degrading model performance. Key outputs include identification of "dedicated safety neurons", quantification of attack success rates, and mechanistic visualizations of semantic shifts across layers.

2. Methodological Framework

NeuroBreak's methodology integrates three synergistic components:

• Overall Security and Utility Assessment:

Establishes baseline and attack-specific metrics such as Attack Success Rate (ASR) and general task accuracy, forming the quantitative basis for comparative studies across attacks and mitigation strategies.

• Layer-wise Representation Probing:

Applies linear probes at each layer to extract the expression of harmful semantics within intermediate representations. The probes are supervised via toxicity vectors, operationalized by the classifier:

$$P(\text{Toxic} \mid h) = \text{softmax}(w_{\text{toxic}} \cdot h + b)$$

where $h$ denotes the hidden activation at a layer. Systematic analysis reveals that discriminative power for toxic semantics increases sharply in mid-to-deep layers (layers 15–28 in transformer architectures), with probe accuracy exceeding 90% in these regimes.

• Neuron-level Attribution and Interaction Analysis:

Critical neurons are scored by perturbation-based attribution:

$$I_i(x) = \left|w_i \cdot \Delta \mathcal{L}(x)\right|$$

where $I_i(x)$ measures the consequence of neuron $i$'s perturbation on the loss $\mathcal{L}$ for input $x$, with $w_i$ the output weight. Neurons are classified by their roles using safety and utility importance percentiles:

$$\begin{align*} S(q) &= \{i \mid I_i^s \text{ in top } q\% \text{(safety)}\} \ U(p) &= \{i \mid I_i^u \text{ in top } p\% \text{(utility)}\} \ D(p, q) &= S(q) \setminus U(p) \end{align*}$$

Further, gradient-based attribution maps neuron alignment ($S^k_i$) relative to the toxicity vector and activation contribution ($A^k_i$):

$$S^k_i = \frac{w_{\text{down}, i}^k \cdot w_{\text{toxic}}^k}{\|w_{\text{down}, i}^k\| \|w_{\text{toxic}}^k\|}, \qquad A^k_i = a_{\text{down}, i}^k \cdot \frac{w_{\text{toxic}}^k}{\|w_{\text{toxic}}^k\|}$$

Gradient propagation measures collaboration effects between neurons:

$$G_{ij} = \frac{\partial a_{\text{down}, i}^k}{\partial w_{\text{upstream}, j}^k}$$

This dual semantic-functional analysis supports detailed causal studies and fine-grained interventions.

3. Analysis of Jailbreak Attacks

NeuroBreak supports the analysis of a broad suite of jailbreak techniques, including human-designed adversarial prompts, TAP (prompt rewriting), AutoDan, GPTFuzzer, and GCG. Data for analysis are sourced from SALAD-Bench.

Key findings demonstrate that attack susceptibility is highly layer-dependent. While early layers manifest limited semantic divergence under attack, mid- and late-stage layers can amplify adversarial signals along the learned toxicity direction, increasing the likelihood of harmful output. Specific attack methods result in distinct activation fingerprints; for example, template-based attacks like GPTFuzzer steer latent representations directly, whereas TAP attacks induce multimodal semantic transitions through complex manipulation.

Comparisons across methods reveal that the model’s internal representation of harmfulness is progressively concentrated in the deeper layers and can be manipulated by attacking certain critical neurons or steering the semantic direction vector.

4. Quantitative Evaluation and Case Studies

NeuroBreak’s efficacy is validated through both aggregate metrics and targeted case studies:

• Case Study I—Semantic Evolution:

Tracing sample trajectories from benign to toxic activation regions through successive layer views; experts identified “decision points” (e.g., layer 11) where divergence becomes pronounced and validated the defensive potency of specific neurons via temporary ablation.

• Case Study II—Security Hardening:

Direct comparison of activation profiles for successful/unsuccessful attacks allowed targeted correction (fine-tuning select safety neurons), resulting in enhanced defense against multiple attack types without degrading general LLM performance.

• Fine-Tuning Experiments:

Systematic trials on models such as Llama3-Instruct compared full fine-tuning, LoRA, Targeted Safety Fine-Tuning (TSFT), and NeuroBreak’s assisted TSFT. Notably, updating less than 0.2% of total parameters via NeuroBreak achieved ASRs equivalent or superior to full model retraining while maintaining utility. Further, NeuroBreak converged faster, evident in the comparative loss curves.

5. Mechanistic Insights Into Safety and Vulnerabilities

NeuroBreak provides mechanistic clarity on how harmful semantics are propagated and modulated in LLM inference:

• Semantic Construction:

Latent harmfulness arises gradually, with “decision gatekeeping” concentrated in specific layers that either suppress or facilitate toxic direction progression. This aligns with findings that certain layers act as bottlenecks for safety-related semantic features.

• Functional Specialization of Neurons:

Neurons are stratified by their alignment and activation with respect to safety vectors. S⁺A⁺ neurons tend to amplify toxicity, while S⁻A⁺ suppress benign features. Upstream neurons with high $G_{ij}$ scores exert disproportionate influence, suggesting focal points for both vulnerabilities and targeted defense.

• Defense Strategies:

By isolating and fine-tuning dedicated safety neurons ($D(p,q)$), NeuroBreak enables targeted refusal-guided correction procedures. The interactive interface supports causal “what-if” experimentation, where ablation or modification of neuron subsets immediately displays downstream effects in activation profiles.

6. Implications for LLM Security and Future Defense

NeuroBreak’s platform and methodology introduce a paradigm shift in the mitigation of jailbreak vulnerabilities. The combination of multi-granular representation probing, semantic/functional neuron attribution, and real-time intervention forms a robust foundation for next-generation LLM defense strategies—precise, parameter-efficient, and responsive to adversarial innovation.

Empirical evidence suggests that such targeted approaches can achieve state-of-the-art security without compromising task utility. The mechanistic insights into decision gating, neuron specialization, and collaborative activation dynamics open avenues for both safer model design and continual adaptation against evolving jailbreak threats.

This comprehensive treatment situates NeuroBreak as a critical tool for the systematic paper and reinforcement of internal LLM safety architectures (Zhang et al., 4 Sep 2025).