---
title: Separable Sink Divergence in LLM Safety
url: https://www.emergentmind.com/topics/separable-sink-divergence-hypothesis
type: topic
---

# Separable Sink Divergence in LLM Safety

The separable sink divergence hypothesis is an empirical and theoretical framework for understanding and mitigating harmful fine-tuning in large language models (LLMs) via the behavior of internal attention mechanisms. It posits that attention heads implicated in learning harmful patterns can be linearly separated from those supporting safe refusal by the sign of a statistic called sink divergence, thus enabling targeted defenses such as Surgery that regularize this property during further fine-tuning [2602.05228].

## 1. Definition and Computation of Sink Divergence

Sink divergence is defined at the level of individual attention heads within a transformer architecture. For a given model input sequence, an attention sink token is the position $k$ that receives the highest aggregate attention across all heads and source positions. For each attention head $h$ and input $\mathbf X$, the mean attention weight it assigns to the sink token is:
\[
\alpha_{h}(\mathbf X) = \frac{1}{|\mathcal N_{k}|} \sum_{i\in\mathcal N_{k}} A_{h,k,i}(\mathbf X)
\]
where $k$ is the sink token index and $\mathcal N_{k}$ the set of non-sink positions.

Given batches of harmful (malicious) samples $\mathbf X_{m}$ and safe/refusal samples $\mathbf X_{r}$, the sink divergence for head $h$ is:
\[
D_{\mathrm{sink}(h)} = d_{h} = \alpha_{h}(\mathbf X_{m}) - \alpha_{h}(\mathbf X_{r})
\]
This scalar statistic quantifies the change in how each head allocates attention to the sink token between harmful and benign data.

## 2. Empirical Analysis: Bimodality and Correlation with Model Harmfulness

Upon measuring $d_{h}$ for each attention head over large batches from $\mathbf X_{m}$ and $\mathbf X_{r}$, the distribution displays a marked bimodality: roughly half of the heads have $d_{h}>0$ (positive sink divergence), and half have $d_{h}<0$. Empirical investigation reveals several key phenomena:

- As the proportion $p$ of harmful data in fine-tuning increases, both the model’s harmfulness (quantified by the Harmful Score, HS) and the number of heads with $d_{h}>0$ rise correspondingly (e.g., for the Lisa baseline, heads with $d_{h}>0$ increase from 553 to 580 as $p$ is raised from 0 to 0.5).
- Disabling attention heads with $d_{h}>0$ after fine-tuning causes HS to decrease, while disabling heads with $d_{h}<0$ increases HS.

These findings establish that positive-sink heads support the acquisition and expression of unsafe behaviors, while negative-sink heads reinforce safe refusal [2602.05228].

## 3. Formal Statement of the Separable Sink Divergence Hypothesis

The hypothesis asserts a divisibility in the functional roles of attention heads by the sign of their sink divergence. Denoting
\[
H^{+} = \{h\mid D_{\mathrm{sink}(h)} > 0\}, \qquad H^{-} = \{h\mid D_{\mathrm{sink}(h)} < 0\}
\]
the hypothesis posits:
\[
\forall\,h\in H^{+}:\;\text{“harmful-pattern head”}, \quad \forall\,h\in H^{-}:\;\text{“benign/refusal head”}
\]
This partition offers a mechanistically interpretable lens on harmful pattern acquisition: heads responsible for harmful learning are separable by a simple sign criterion on $d_{h}$.

## 4. Sink Divergence Regularization: The Surgery Defense

Leveraging the separable sink divergence property, the Surgery defense is formulated as a regularization-based method for safe fine-tuning. The training objective minimizes the sum of standard cross-entropy loss and a regularizer that penalizes positive sink divergence:
\[
\min_{\mathbf w} \;\ell_{\mathrm{CE}}(\mathbf w) + \lambda \frac{1}{|\mathcal H|}\sum_{h\in\mathcal H} \mathrm{ReLU}\left(D_{\mathrm{sink}(h)}\right)
\]
where $\ell_{\mathrm{CE}}$ is the cross-entropy loss, $\mathcal H$ is the set of all heads, $\mathrm{ReLU}(x)=\max(0,x)$, and $\lambda$ is a weighting hyperparameter.

This penalty explicitly steers all heads toward the negative-sink regime, thereby suppressing the emergence of harmful pattern heads. Backpropagation through the attention computations enables head-specific modulation of attention allocation on harmful vs. safe data during fine-tuning.

## 5. Experimental Validation and Mechanistic Evidence

Empirical evaluation on several safety benchmarks demonstrates the efficacy of Surgery:

| Benchmark     | Harmful Score (HS), Surgery | Baseline (Lisa) | Improvement |
|---------------|----------------------------|-----------------|-------------|
| BeaverTails   | 8.90%                      | 14.80%          | +5.90 pp    |
| HarmBench     | 9.50%                      | 20.75%          | +11.25 pp   |
| SorryBench    | 12.95%                     | 22.50%          | +9.55 pp    |

- When the fine-tuning mix contains $p=0.1$ of harmful samples ($n=1000$ samples, GSM8K task), Surgery reduces the harmfulness below all tested baselines while matching their finetune accuracy (approximately 68.5%).
- Post-Surgery, more than 96% of heads originally in $H^{+}$ transition to $H^{-}$, and sink values on harmful examples decrease layer-wise, indicating a systematic reallocation of attention away from harmful patterns.
- Disabling heads in $H^{+}$ post hoc reliably reduces HS, while disabling $H^{-}$ increases HS.

## 6. Model-Wide Robustness, Generalization, and Limitations

Surgery exploits the intrinsic phenomenon of attention sinks, sidestepping external data or architectural modifications. Because attention sinks are a ubiquitous property of transformer LLMs, Surgery generalizes across models of varying architecture and size, including Llama3-8B, Gemma2-9B, and Qwen2-14B.

However, some early-layer heads are relatively resistant to regularization, suggesting that further improvements might be attainable with layer-specific penalties. Experimental support is presently restricted to open-source models in the 8–14B parameter regime; applicability to larger (50–100B) or proprietary models represents an open direction.

## 7. Implications for LLM Safety and Mechanistic Interpretability

The separable sink divergence hypothesis provides both a discriminative metric for identifying heads implicated in harmful pattern learning and a mechanism for direct intervention at the architectural level. Surgery achieves high efficacy without recourse to costly data selection or projection-based defenses. A plausible implication is that attention sink statistics offer a generic axis for diagnosing and controlling problem behaviors in large transformer models. Further extensions may address the small subset of resistant heads or evaluate transferability to other safety-critical domains [2602.05228].

Source: https://www.emergentmind.com/topics/separable-sink-divergence-hypothesis