Reference Sink Dynamics in Transformers

Updated 4 July 2026

Reference Sink is a phenomenon where a token or set of tokens receives disproportionately high attention, acting as a computational anchor in transformers.
It manifests stably as a BOS-like anchor in autoregressive models but appears dynamically, shifting with denoising steps in diffusion language models.
Empirical studies show that masking these sinks severely degrades performance in autoregressive models while having minimal impact on diffusion language models.

“Reference sink” is not a universally standardized term in transformer research, but it is a useful umbrella for a recurring phenomenon: a token, or small set of tokens, that receives disproportionately large incoming attention and thereby behaves as an anchor for subsequent computation. In the strongest autoregressive sense, a reference sink is a stable, globally relied-upon attractor, often near the beginning of the sequence. Recent work on diffusion LLMs (DLMs), however, shows that this interpretation transfers only partially. In DLMs, attention sinks are real, but they are typically dynamic concentration points—often moving across denoising steps, sometimes disappearing, sometimes splitting by token type, and usually not indispensable for successful generation (Rulli et al., 17 Oct 2025).

1. Terminology and formalization

The literature usually speaks of attention sinks rather than “reference sinks.” In one geometric line of work, the preferred vocabulary is attention sink and reference frame, with the claim that sink tokens can function as anchors or coordinate references in representation space rather than mere artifacts (Ruscio et al., 4 Aug 2025). In another line, the phrase “reference sink” is treated as an overloaded surface description: a visually identical vertical stripe in an attention map may correspond either to a null destination used to suppress updates or to a shared workspace token used to collect and rebroadcast information (Fesser et al., 6 Jun 2026). This terminological ambiguity matters because it separates the appearance of a sink from its computation.

In “Attention Sinks in Diffusion LLMs,” attention at head $h$ and layer $l$ is written as

$A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$

with $q_i$ and $k_j$ the query and key projections. Because DLM attention is bidirectional, the analysis is sequence-wide at each denoising step. The paper’s central statistic is the cumulative incoming attention of token $j$ at denoising step $t$ ,

$\bar{A}^{(t,l,h)}_{j} = \frac{1}{S} \sum_{i=1}^{S} A_{ij}^{(t,l,h)},$

where $S$ is sequence length. A token is then identified as a sink when its cumulative attention is an outlier above the rest by margin $\epsilon$ , with $l$ 0 in all experiments. The paper states that this threshold filters out at least $l$ 1 of sequence tokens and is empirically robust; it also notes a likely typographical inconsistency in the printed formula, while making the intended rule clear: a sink is a token whose incoming attention is substantially above the average incoming attention of the others (Rulli et al., 17 Oct 2025).

The strictest notion of a reference sink comes from autoregressive models (ARMs), where sink positions are typically stable and often prefix-like. One geometric study identifies a centralized reference frame in decoder-only models with standard RoPE, describing a star-like attention topology in which many tokens route to one dominant anchor, usually $l$ 2, <s>, or <bos>. In that account, LLaMA variants show 100% BOS specialization, and one summary statistic for LLaMA-3.2-3B gives average sink concentration $l$ 3 with maximum $l$ 4 (Ruscio et al., 4 Aug 2025). This is close to the classical “reference sink” picture: one privileged token acts as a latent-space origin.

A separate mechanistic account reframes the familiar first-token sink as an implicit gate. For head $l$ 5 in layer $l$ 6, the effective gate is

$l$ 7

so the larger the sink weight, the less that head contributes. In this view, Vanilla Attention, Sink Attention, and Gated Attention all instantiate a native MoE-like routing pattern inside attention layers; the sink is the null branch that absorbs redundant probability mass, and “head collapse” is interpreted as expert collapse (Fu et al., 1 Feb 2026). Under that interpretation, a reference sink is not merely a token that attracts attention; it is the position through which a head can effectively choose to do little.

Vision-transformer analysis complicates the picture further by showing that the same sink pattern can encode two different algorithms. One is adaptive $l$ 8, where the sink has negligible value norm and suppresses the residual update. The other is broadcast, where the sink carries meaningful information and induces low-rank outputs across many positions. The paper’s practical conclusion is that sink patterns alone are not mechanistic explanations; a “reference sink” may be either a null destination or a shared reference/workspace token, and the distinction must be diagnosed from values and outputs rather than from attention maps alone (Fesser et al., 6 Jun 2026).

3. Diffusion LLMs as a distinct setting

The DLM study examines three masked discrete DLMs—Dream-7B, LLaDA-8B, and MMaDA-8B—and compares them with the autoregressive baseline Llama-3.1-8B in sink-masking experiments (Rulli et al., 17 Oct 2025). The three DLMs differ in training objective and inference procedure, and those differences are central to how their sinks behave.

LLaDA-8B and MMaDA-8B are trained from scratch with a masked language modeling objective of the form

$l$ 9

At inference, they use semi-autoregressive block diffusion: the sequence is divided into blocks, and tokens are gradually unmasked within each block from left to right. Dream-7B instead is initialized from an autoregressive model and uses a shifted objective,

$A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 0

so it predicts the next token in a way more reminiscent of autoregression; it does not use the same blockwise decoding. These architectural and training differences are later reflected in whether sinks are semantic, positional, stable, or mobile.

The analysis does not introduce entropy-based or more elaborate concentration metrics. Visualization relies on attention heatmaps, incoming-attention distributions, cross-layer/head sink-intensity maps, and sink-displacement plots. The histogram figures for LLaDA-8B, Dream-7B, and MMaDA-8B all show a heavy concentration pattern: most tokens receive near-zero incoming attention, while a few receive very high values. In LLaDA-8B, later layers typically have fewer sinks and therefore stronger concentration on the remaining ones; the layerwise textual trend is that sink counts decrease in deeper layers, converging to one or two sinks per layer (Rulli et al., 17 Oct 2025).

4. Dynamic sinks in diffusion LLMs

The defining result of the DLM paper is that sinks in DLMs are often moving sinks, explicitly described as a new kind of sink “not observed in ARMs” in this form. In LLaDA-8B, sinks may persist for several denoising steps and then vanish, or jump from one position to another between adjacent steps. One example shows a sink moving from position 62 at step 38 to position 88 at step 39. Another shows an erratic case where a sink appears at step 96 and disappears at step 97. This already departs from the idea of a stable universal reference token (Rulli et al., 17 Oct 2025).

The paper also reports deeper-layer patterns in which masked and unmasked tokens have separate sinks, and the dominant sink switches as denoising proceeds. At the end of a block, masked tokens can attend heavily to one sink while unmasked tokens concentrate on another, creating a sharp visual separation between the two token sets. Such complementary, state-dependent assignments are difficult to reconcile with a single global reference sink. This suggests that sink roles in DLMs are conditional on the current denoising state rather than fixed for the entire generation.

Sink location is likewise broader than in causal ARMs. The paper states that DLM sinks “do not necessarily appear in the beginning of the sentence,” but can also occur in the middle or toward the end because attention is bidirectional. In LLaDA-8B, sink selection has a strong semantic or structural flavor: punctuation marks such as periods and commas, whitespace, and end-of-sequence tokens repeatedly become sinks. The authors interpret this as evidence that LLaDA has learned semantically aware attention mechanisms that identify structurally important tokens as reference points, while still noting that these positions move or disappear across denoising steps.

By contrast, Dream-7B exhibits primarily positional behavior. Its sinks often originate at the rightmost masked token and then shift leftward as tokens are progressively unmasked, largely independent of content. This migration is strongest in early layers and tracks the unmasking frontier. The paper hypothesizes that Dream’s initialization from a pretrained autoregressive model leaves it less adapted to bidirectional attention, so it falls back more on positional cues. These are therefore not BOS-like anchors, but dynamic positions tied to the current masked boundary.

Among the three DLMs, MMaDA-8B comes closest to the traditional reference-sink picture. Sinks are less frequent overall; many layers show no strong sinks; but when a sink exists, it often remains fixed at its initial position throughout generation. One figure shows a sink at the beginning of the sequence remaining there at denoising steps 0 and 127. Even there, however, the paper emphasizes that bidirectional attention permits patterns “not possible in causal models,” and another example shows tokens shifting attention toward an already unmasked token from one step to the next. In the paper’s overview of sink displacement, MMaDA-8B is mostly static, LLaDA-8B tends to shift right as more blocks are denoised, and Dream-7B tends to shift from right to left (Rulli et al., 17 Oct 2025).

5. Sink masking and the question of necessity

The decisive test of whether DLM sinks are true reference sinks in the autoregressive sense is intervention. The paper masks attention scores directed toward the top- $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 1 sinks identified by its metric and evaluates on GSM8K and HumanEval, using Hugging Face checkpoints, PyTorch, and the official lm-evaluation-harness scripts. The sink-masking settings $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 2 correspond to masking the top 1, 5, and 10 sinks, and are unrelated to the sink-detection margin $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 3 (Rulli et al., 17 Oct 2025).

The reported numbers show that masking one sink barely affects DLMs, whereas it catastrophically breaks the ARM baseline.

Model	GSM8K: unmasked $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 4 top 1 $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 5 top 10	HumanEval: unmasked $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 6 top 1 $A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 7 top 10
Dream-7B	$A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 8	$A_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right),$ 9
LLaDA-8B	$q_i$ 0	$q_i$ 1
MMaDA-8B	$q_i$ 2	$q_i$ 3
Llama-3.1-8B	$q_i$ 4	$q_i$ 5

The intermediate top-5 setting follows the same pattern: on GSM8K, Dream-7B drops to $q_i$ 6, LLaDA-8B to $q_i$ 7, MMaDA-8B stays at $q_i$ 8, while Llama-3.1-8B remains at $q_i$ 9; on HumanEval, Dream-7B is $k_j$ 0, LLaDA-8B $k_j$ 1, MMaDA-8B $k_j$ 2, and Llama-3.1-8B $k_j$ 3. The authors summarize the key effect by stating that “for all the tested DLMs, masking one sink leads to a degradation in performance smaller than 1%.” Significant degradation appears only when many sinks are masked, especially in MMaDA-8B. By contrast, the ARM baseline collapses after masking a single sink. This is the strongest empirical reason that DLM sinks should not generally be called stable reference sinks (Rulli et al., 17 Oct 2025).

The paper’s interpretation has two parts. First, bidirectional attention means every token can access the full context at each denoising step, reducing dependence on a single routing node. Second, in iterative denoising, tokens with reduced confidence can simply be delayed: if a sink is masked, the most affected tokens may become lower-confidence and therefore may “not consider them for unmasking,” rather than forcing the whole generation to fail. This suggests both why DLMs remain robust when sinks are removed and why sink locations can reconfigure as the partially denoised sequence changes.

6. Broader interpretations, limitations, and open questions

The broader sink literature implies that “reference sink” should not be treated as a single mechanism. One geometric account argues that attention sinks are the observable footprint of reference-frame construction in transformer latent spaces, with centralized, distributed, and bidirectional reference frames depending strongly on positional encoding and architecture (Ruscio et al., 4 Aug 2025). Another reinterprets sinks as a native MoE gate, where the sink absorbs redundant probability mass and thereby modulates head activation (Fu et al., 1 Feb 2026). A third insists that the same sink-like pattern may mean either adaptive $k_j$ 4 or broadcast, depending on value norms and induced output rank (Fesser et al., 6 Jun 2026). These perspectives are mutually compatible in one narrow sense: visual sink patterns alone do not fix the underlying computation.

Against that backdrop, the DLM evidence points toward a narrower conclusion. In diffusion LLMs, sinks are real and often useful, but they are usually not stable universal attractors. They are better described as temporally contingent anchors whose role depends on denoising state, masking structure, and model-specific training dynamics. The paper explicitly argues that their emergence cannot be explained solely by positional encoding or absolute position: the DLMs use RoPE, yet still show semantic sinks on punctuation and whitespace, split sinks between masked and unmasked token subsets, and model-specific motion patterns. The authors instead suggest a combination of training dynamics, token frequency, and the structural role of certain tokens (Rulli et al., 17 Oct 2025).

The same paper is also careful about what remains unresolved. It studies only instruct models, not base models. It does not alter training procedures to test causal hypotheses about sink formation. It does not perform a mechanistic analysis of what information future-position sinks store, and it names tools such as Logit Lens as natural future directions. Several of the more exotic observed behaviors, especially complementary masked/unmasked sinks, are established empirically rather than fully explained. This suggests that the current evidence is strongest on phenomenology—movement, robustness, and model specificity—rather than on a complete mechanistic account.

In current usage, then, reference sink is best reserved for the strong autoregressive case of a stable, indispensable attention anchor. For DLMs, the closest accurate description is weaker: they contain sink-like tokens that can act as structural or semantic references, but these are typically flexible, moving, and often non-essential anchors rather than a single stable coordinate origin for the whole generation process (Rulli et al., 17 Oct 2025).