Information stored in future-position attention sinks in diffusion language models

Determine what type of information is encoded by attention sink tokens that correspond to future positions during the iterative denoising process in masked discrete diffusion language models with bidirectional attention, such as those studied in this work, to clarify the role and content of these future-position sinks.

Background

The paper presents the first empirical analysis of attention sinks in masked diffusion LLMs (DLMs), showing that sinks emerge consistently but behave differently from those in autoregressive models. In DLMs, sinks are dynamic and can shift across denoising steps, often aligning with semantic or structural tokens.

A notable phenomenon observed is the presence of sinks at future positions relative to the current denoising step. The authors highlight that, despite extensive empirical characterization of sink dynamics, the specific type of information stored in these future-position sinks remains unresolved, suggesting the need for mechanistic interpretability analyses (e.g., using the Logit Lens) to understand their function.