Attention Sinks in Diffusion Language Models

• The paper establishes that attention sinks in diffusion language models dynamically shift positions during denoising, contrasting with the static sinks in autoregressive models.
• The study quantifies sinks using cumulative attention scores and reveals that only a small fraction (~4%) of tokens consistently attract disproportionate attention, indicating flexible context allocation.
• The research highlights practical benefits such as minimal performance loss upon sink masking, improved model pruning, and potential efficiency gains in deployment.

Attention sinks in diffusion LLMs (DLMs) refer to tokens or positions in the input sequence that receive disproportionately high attention weights during the model’s denoising process. Although the attention sink phenomenon has been extensively studied in autoregressive models (ARMs), recent analyses reveal that DLMs not only exhibit attention sinks, but do so with distinct, dynamic behaviors that impact model robustness, interpretability, and computational efficiency (Rulli et al., 17 Oct 2025). The following sections provide a detailed exposition of attention sinks in DLMs, covering their mathematical definition, empirical characteristics, mechanistic differences from ARMs, implications for training and inference, and consequences for model design and practical deployment.

1. Mathematical Definition and Detection

In DLMs, attention sinks are identified by computing cumulative attention scores for each token across all queries within a head and layer at a specific diffusion step. The cumulative score for token $j$ is:

$$\bar{A}^{(t,l,h)}_j = \frac{1}{S}\sum_{i=1}^{S} A^{(t,l,h)}_{ij}$$

where $$A^{(t,l,h)}_{ij} = \text{softmax}_j\left(\frac{q_i^\top k_j}{\sqrt{d}}\right)$$

A token is classified as an attention sink if:

$$\bar{A}^{(l,h)}_{j} > \frac{1}{S-1}\sum_{k\neq j}\bar{A}^{(l,h)}_{k} + \epsilon$$

with $\epsilon$ empirically set to ensure only a modest fraction (∼4%) of tokens are detected as sinks (Rulli et al., 17 Oct 2025). This cumulative scoring criterion distinguishes sinks without reliance on semantic or positional heuristics.

2. Dynamic and Structural Properties

Unlike ARMs, where attention sinks are typically static—anchored near the sequence boundaries (often the first token)—DLMs exhibit dynamic sinks whose positions can shift throughout the denoising process. In models such as LLaDA–8B, a sink may persist at position 62 for several steps before jumping to position 88 or disappearing altogether. Dream–7B, initialized from an ARM, shows sink migration from the rightmost masked token leftwards as unmasking occurs. Models initialized from scratch (e.g., MMaDA–8B) often display sinks centered on punctuation, whitespace, or end-of-sequence markers, indicating a semantic or structural basis for sink formation rather than purely positional anchoring (Rulli et al., 17 Oct 2025).

Across denoising steps and increasing layer depth, the number of sinks per head typically decreases, converging to one or two in the deepest layers. In these layers, distinct sinks may be observed for masked and unmasked tokens, with gradual switching over iterations. The dynamic nature of sinks in DLMs suggests a mechanism for flexible context allocation and dynamic anchoring during token prediction.

3. Impact on Model Robustness and Performance

Empirical studies demonstrate that DLMs are robust to the masking or removal of attention sinks. On mathematical reasoning and code generation benchmarks (e.g., GSM8K, HumanEval), masking the top-ranked sink results in less than a 1% drop in accuracy (Rulli et al., 17 Oct 2025). In contrast, ARMs such as LLama‑3.1‑8B suffer catastrophic performance degradation if even a single sink (usually the first token) is removed, highlighting a crucial difference: while ARMs rely heavily on static, sink-based anchoring for context propagation, DLMs, through bidirectional attention and iterative denoising, can re-route attention dynamically, maintaining performance when dominant sinks are disrupted.

This robustness is largely attributed to the parallel context availability in DLMs; all tokens can access information bidirectionally at each step, which allows alternative attention pathways and mitigates the collapse seen in left-to-right causal models when a key anchor is disturbed.

4. Mechanistic Contrast with Autoregressive Models

The static, positionally anchored sinks in ARMs arise from the causal structure and sequential nature of autoregressive decoding (Gu et al., 14 Oct 2024, Barbero et al., 3 Apr 2025). In such models, boundary tokens (often the first token or special markers) accumulate steady attention due to their persistent visibility and the necessity for softmax normalization over all previous tokens.

DLMs, in contrast, leverage transformer encoders with bidirectional attention. Sinks are not tied to a token’s position in the sequence but may arise on structurally significant or contextually relevant tokens, depending on the masking pattern, prompt, and step. This difference is reflected in both the dynamic movement of sink positions and the model’s insensitivity to sink removal. DLMs actively redistribute attention as the signal-to-noise ratio changes across denoising steps, contrasting sharply with the fixed anchoring in ARMs (Rulli et al., 17 Oct 2025).

5. Implications for Attention Map Alignment and Training Objectives

Recent work demonstrates that aligning attention maps with linguistic or structural attributes can mitigate misbindings and semantic leakage in text-to-image diffusion settings (Rassin et al., 2023, Zhang et al., 11 Mar 2024). Techniques such as loss terms over cross-attention maps and test-time optimization steer attention distributions to reflect syntax-derived entity–modifier bindings, correcting “attention sink” effects where irrelevant or misplaced tokens attract undue attention.

Blockwise SFT for DLMs exploits this insight, restructuring supervision granularity by partitioning responses into fixed-size blocks and carefully controlling attention over clean prefixes, stochastically masked blocks, and fully hidden suffixes. This alignment reduces “attention sinks” induced by noisy prefixes or leaky suffixes and empirically yields significant improvements over classical SFT (Sun et al., 27 Aug 2025).

The dynamic sink phenomenon forces reconsideration of which tokens merit explicit supervision or precision preservation during training and quantization. For example, quantization schemes and inference optimizations that only preserve the first-N tokens (as in ARMs) may miss dynamically emerging sinks in DLMs, prompting methods such as KVSink that adaptively detect and preserve sink tokens regardless of position (Su et al., 6 Aug 2025).

6. Efficiency, Pruning, and Sparsification in DLMs

The shifting and distributed nature of sinks in DLMs opens avenues for computational efficiency. Dormant head detection via metrics like HONOR, which considers both attention weights and output value norms, suggests that a non-trivial fraction (4–14%) of attention heads in DLMs can be zeroed without degrading performance, unlike ARMs where sink heads are indispensable (Sandoval-Segura et al., 4 Apr 2025). For video diffusion transformers (VDiTs), sink heads can be pruned or retrained in late layers to enable greater sparsification without loss of quality, thus improving the efficiency–quality Pareto frontier (Wen et al., 14 Apr 2025).

Sparse attention strategies such as SparseD exploit the head-specific, temporally stable attention patterns in DLMs, enabling pre-computation and reuse of sparse patterns after initial full-attention burn-in steps. This organization avoids the computational “sink” associated with quadratically growing attention costs, and maintains high generation quality during later denoising steps (Wang et al., 28 Sep 2025).

7. Implications for Model Deployment, Security, and Future Research

The flexibility and robustness of attention sinks in DLMs yield practical improvements for long-context modeling, quantization, and streaming, as well as resilience to attention-based attacks or perturbations (Barbero et al., 3 Apr 2025, Xiao et al., 2023, Shang et al., 19 Oct 2025). Dynamic attention sinks serve as anchor points for inference and optimization (such as cache retention or selective quantization), and may represent gateways for backdoor attacks in unlearning scenarios (Shang et al., 19 Oct 2025). Their detection and characterization are essential for safe and effective DLM deployment.

Open questions remain concerning the optimal identification and utilization of dynamic sinks for further compression, sparsification, and efficient inference in DLMs, as well as the mitigation of potential vulnerabilities (e.g., in backdoored unlearning). Research is ongoing into adaptive blockwise supervision, advanced sparse selection methods, and robust alignment of attention with linguistic or semantic constraints.

---

Attention sinks in diffusion LLMs embody a dynamic, context-dependent phenomenon in transformer attention: unlike the static, critical anchors of autoregressive models, DLM sinks move and adapt across denoising steps, arise from semantic or structural features, and confer robustness to perturbations. Their empirical properties and mechanistic distinctiveness offer both improved efficiency and new challenges in model deployment, supervision alignment, and security, motivating further exploration into their foundational role in large-scale generative language systems.