Anchored Diffusion Language Models

• ADLM is a framework that defines important anchor tokens to structure and stabilize the reconstruction process in diffusion language models.
• It uses a two-stage process with an anchor network predicting crucial tokens followed by a denoiser reconstructing remaining tokens.
• The method reduces sample complexity and improves performance metrics such as perplexity and output quality across various language tasks.

Anchored Diffusion LLM (ADLM) denotes a family of anchored generative methods for diffusion LLMs. In the original formulation, ADLM is a two-stage diffusion LLM that predicts distributions over important tokens with an anchor network and then reconstructs the remaining tokens conditioned on those predictions (Rout et al., 24 May 2025). Subsequent work uses anchored diffusion language modeling more broadly to denote inference-time insertion of structural anchors into the masked sequence so that diffusion LLMs can satisfy reasoning templates, suffix constraints, or parseable JSON formats; “Dynamic Infilling Anchors” is presented as a concrete instantiation and extension of that broader paradigm (Han et al., 3 Jun 2026). Across these formulations, the common principle is to treat tokens that organize the rest of the sequence—either semantically important lexical items or explicit structural delimiters—as first-class objects in the denoising process.

1. Conceptual scope and motivation

The original ADLM paper starts from a specific diagnosis of why diffusion LLMs lag strong autoregressive transformers. Autoregressive models factorize the joint distribution as $q(x) = q(x^1)\prod_{l=2}^L q(x^l \mid x^{1:l-1})$ and are trained as likelihood models over growing prefixes. Diffusion LLMs instead corrupt a sequence by masking tokens according to a schedule and then iteratively reconstruct them. This affords parallel generation and bidirectional conditioning, but the paper argues that the performance gap arises when important tokens—described as “key words or low-frequency words that anchor a sentence”—are masked early in the forward process, limiting contextual information for accurate reconstruction (Rout et al., 24 May 2025).

This motivation yields two related meanings of ADLM. In the narrow sense, ADLM is the 2025 two-stage model with an explicit anchor network. In the broader sense, later work treats anchored diffusion language modeling as a paradigm in which structural anchors are injected into the mask grid and preserved through denoising. The broader usage is not merely terminological drift: it reflects the same underlying thesis that some tokens should be privileged because they stabilize or organize the reconstruction of the rest of the sequence.

Paper	Scope	Contribution
“Anchored Diffusion LLM” (Rout et al., 24 May 2025)	Training-time two-stage DLM	Anchor network, denoiser, ANELBO, sample-complexity analysis
“Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion LLMs” (Han et al., 3 Jun 2026)	Inference-time anchored dLLM generation	Dynamic end-anchor planning and iterative infilling
“When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion LLMs” (Park et al., 27 May 2026)	Inference-time fully non-AR decoding	Suffix anchoring with progress-dependent confidence modulation

2. Two-stage anchored diffusion architecture

In the original ADLM, each reverse step is split into two learned substeps. The anchor network $\varphi$ predicts a mixture over important tokens from the current noisy sequence $Z_t$, and the denoising network $\psi$ predicts missing tokens conditioned on these anchored predictions. The model therefore parameterizes the reverse process as

$$p_\theta(Z_s \mid Z_t) \equiv q\big(Z_s \mid Z_t, Y_\psi(Y_\varphi(Z_t))\big),$$

where $Y_\varphi(Z_t)$ is the anchor network output and $Y_\psi(Y_\varphi(Z_t))$ is the denoiser’s refined mixture used in the inference posterior (Rout et al., 24 May 2025).

A central design question is what counts as an anchor token. The paper defines importance heuristically by low within-sequence frequency:

$$\mu(x^l) = \frac{1}{L}\sum_{j=1}^L \mathbb{1}_{\{x^j = x^l\}},$$

and tokens with $\mu(x^l) \le \tau$ are treated as important anchors during training. The operator $\mathcal{A}(\cdot)$ maps a sequence $\varphi$0 to an important-token mixture $\varphi$1, with explicit selection used for loss masking. This operationalizes the intuition that low-frequency content words, named entities, or structurally central lexical items provide disproportionate information about the rest of the sequence (Rout et al., 24 May 2025).

Architecturally, the anchor network uses a transformer similar to SEDD, specifically a Diffusion Transformer with rotary embeddings, and outputs a full token distribution at each position. The denoiser uses a similar base architecture with half the number of layers and consumes both the noisy sequence and a projection of anchor logits. Because anchored logits are passed through a linear projection into the denoiser’s embedding space, there is no discrete sampling step between $\varphi$2 and $\varphi$3, so gradients flow end-to-end. Sampling begins from a fully masked sequence $\varphi$4 and iterates anchor prediction, denoiser prediction, and reverse sampling under either a locked-in sampler with $\varphi$5 or a remasking sampler with $\varphi$6 (Rout et al., 24 May 2025).

3. Objective functions and theoretical analysis

The training criterion augments standard diffusion likelihood training with an explicit anchor supervision term. The anchor loss is defined as a KL divergence between an ideal anchor transition and the learned anchor transition, applied only on important tokens. The main theoretical result is the Anchored Negative Evidence Lower Bound (ANELBO), stated as

$\varphi$7

The resulting bound combines a reconstruction term conditioned on anchor predictions with denoising log-likelihood and anchor-prediction log-likelihood terms; when $\varphi$8 and $\varphi$9, ANELBO reduces to the MDLM NELBO (Rout et al., 24 May 2025).

The theoretical argument is not limited to a new loss. The paper also gives a graphical-model sample-complexity analysis under tabular CPT parameterization. If each token depends on an anchor set $Z_t$0 of bounded size $Z_t$1, then standard autoregressive modeling scales as $Z_t$2, standard diffusion as $Z_t$3, while anchored autoregressive modeling (A2R) and ADLM scale as $Z_t$4. The intended conclusion is that, when the underlying structure admits small anchor sets, anchoring yields an exponential reduction in sample complexity. The appendix further interprets anchored training as an EM-like procedure in which anchors behave as latent variables estimated before updating the denoiser, and the ANELBO is shown to improve monotonically under a specific anchored EM update rule (Rout et al., 24 May 2025).

This analysis also clarifies a common misconception: ADLM is not merely a larger diffusion model with extra capacity. The paper reports that even the two-stage architecture without anchor loss, denoted ADLM* with $Z_t$5, improves over MDLM on OpenWebText, but adding the anchored loss improves further. The claim is therefore structural rather than purely parametric: the benefit comes from separating anchor prediction from residual denoising and supervising that separation explicitly.

4. Empirical performance and extensions beyond diffusion

On language modeling benchmarks, the original ADLM reports improved perplexity over prior diffusion baselines. On LM1B, ADLM achieves test perplexity 26.40 at 33B training tokens and 24.46 at 65B, compared with MDLM 27.04 and 25.49, respectively. On OpenWebText, ADLM achieves 21.66 at 110B tokens, 20.62 at 262B, and 20.14 at 524B; the paper notes that the two-stage architecture alone improves MDLM from 23.17 to 21.79 at 262B, while full anchoring improves further to 20.62. On generated text quality under the remasking sampler, ADLM reaches GPT-2 Large generated-sample perplexity 26.8 at 524B training tokens, versus 44.2 for MDLM. For MAUVE, ADLM attains 0.699 at $Z_t$6, 0.788 at $Z_t$7, and 0.791 at $Z_t$8; the paper describes this as the first time a diffusion LLM generates more human-like text, by MAUVE, than a comparable autoregressive model in this setting (Rout et al., 24 May 2025).

The zero-shot generalization results are similarly emphasized. For models trained on OpenWebText at 524B tokens, ADLM achieves the best diffusion perplexity on 6 of 7 reported benchmarks and outperforms the autoregressive baseline on Lambada (44.32 vs. 51.28), PubMed (37.56 vs. 49.01), and ArXiv (33.69 vs. 41.73). These are treated as distribution-shift settings, and the paper argues that the anchor network learns a notion of importance that generalizes out of distribution (Rout et al., 24 May 2025).

The anchoring idea is also extended beyond diffusion. In anchored autoregressive modeling (A2R), the two-stage factorization improves OpenWebText perplexity from 17.94 to 17.29 at 110B tokens, from 17.53 to 16.23 at 262B, and from 17.26 to 15.86 at 524B. In Anchored Chain-of-Thought (ACoT), discrete [ANT] tokens are inserted between question and reasoning-answer tokens and supervised to predict important reasoning elements. Reported accuracies are 45.2 on GSM8K, 100 on ProntoQA, and 97.3 on ProsQA, with 8.2 tokens on ProsQA; these results exceed or match several discrete and latent reasoning baselines in the paper’s comparison set (Rout et al., 24 May 2025).

5. Format-constrained generation and dynamic infilling anchors

A later line of work adapts the anchored principle from training-time semantic anchoring to inference-time structural anchoring in diffusion LLMs. In this setting, a diffusion LLM generates from an initial state of the form

$Z_t$9

and denoises all masked positions in parallel. Because the model starts from a fully masked sequence, mandatory structure can be written directly into the initial mask grid rather than imposed solely through prompting or post-processing. This yields anchored diffusion language modeling in the structural sense: inject delimiters such as >, </think>, <answer>, </answer>, or JSON scaffolds into the mask grid so they remain fixed during denoising (Han et al., 3 Jun 2026).

The same paper identifies the main limitation of naive fixed anchors. In the fixed-position infilling baseline, anchors are placed at pre-defined indices, so the spans between anchors are rigid. If a reasoning span is too short, content is truncated; if it is too long, the model tends to fill the extra space with repeated anchors, junk content, or drifting reasoning. The paper reports that in JSON tasks the fixed infilling baseline collapses to approximately 0% valid JSON, and in reasoning tasks it induces severe accuracy collapse relative to prompt-only decoding (Han et al., 3 Jun 2026).

Dynamic Infilling Anchors (DIA) addresses this by making end-anchor placement instance-adaptive while remaining training-free. The method has two stages. Stage 1 performs generation-length adjustment: it inserts begin anchors at the starts of semantic blocks, runs a single diffusion inference step, searches for exact or partial matches to end anchors such as `or</answer>`, and uses a confidence threshold $\psi$0 to decide whether the current block is long enough. If no sufficiently confident end anchor is found, the block is expanded by $\psi$1 masked tokens, up to a maximum block length $\psi$2. When multiple positions satisfy the threshold, the leftmost is chosen to avoid repeated end anchors. Stage 2 then fixes the begin and end anchors and performs standard iterative denoising only within those boundaries. The paper gives example hyperparameters $\psi$3 on GSM8K, $\psi$4 on MATH, $\psi$5, and $\psi$6 (Han et al., 3 Jun 2026).

Evaluation is reported with a format score

$\psi$7

and answer accuracy

$\psi$8

On GSM8K, Dream-7B-Base and Dream-7B-Instruct with prompt-only formatting achieve 0.00 format score; the fixed infilling baseline reaches 58.83 format score and 14.86 accuracy, while DIA reaches 72.63 format score and 46.78 accuracy. On MATH, prompt-only again yields 0.00 format score; fixed infilling reaches 29.10 format score and 21.52 accuracy, while DIA reaches 76.82 format score and 20.08 accuracy. On WikiBio JSON generation, DIA achieves 79.84 format score under both regex and raw evaluation, with hallucination score 0.15, whereas the fixed infilling baseline records 0.01 format score and 0.00 hallucination. The latency analysis reports that DIA is slower on GSM8K, approximately 26.5 versus 10.7 seconds, but slightly faster on MATH, approximately 30.6 versus approximately 31 to 31.7 seconds, because dynamic length planning avoids over-generation on longer tasks (Han et al., 3 Jun 2026).

6. Confidence-modulated suffix anchoring, misconceptions, and limitations

A further refinement concerns how anchored diffusion decoding should decide which positions to unmask. The central misconception challenged by “When Confidence Misleads” is that high-confidence positions are necessarily ready to decode. In fully non-autoregressive diffusion decoding, end-of-text tokens can become overconfident and cause incomplete generation; inserting a suffix anchor mitigates that problem, but then positions near the anchor become locally overconfident and are decoded too early. The proposed remedy is Suffix-Anchored Confidence Modulation: keep the suffix anchor, define an anchor-proximity weight

$\psi$9

define decoding progress $$p_\theta(Z_s \mid Z_t) \equiv q\big(Z_s \mid Z_t, Y_\psi(Y_\varphi(Z_t))\big),$$0, and modulate the base confidence by

$$p_\theta(Z_s \mid Z_t) \equiv q\big(Z_s \mid Z_t, Y_\psi(Y_\varphi(Z_t))\big),$$1

This down-weights anchor-adjacent positions early and restores their original confidence later in decoding (Park et al., 27 May 2026).

The reported empirical effect is consistent across reasoning, vision-language reasoning, and code generation. For LLaDA with top-probability decoding on text-only reasoning, the average score rises from 21.11 without anchors to 44.03 with suffix anchoring and to 53.88 with full confidence modulation; GSM8K specifically rises from 14.94 to 49.89 to 76.88. Across six tasks comparing top-probability and top-margin decoding, the proposed method outperforms explicit end-of-text suppression, and on GSM8K it exceeds semi-autoregressive block decoding at all reported step budgets while retaining fully non-autoregressive parallelism. The paper also reports that the EOT ratio drops sharply with suffix anchoring, from approximately 0.66 to approximately 0.05–0.2 on GSM8K when anchors are placed late in the response region (Park et al., 27 May 2026).

Across the ADLM lineage, the main limitations are explicit. In the original ADLM, the definition of importance is heuristic and task-specific, anchor-related hyperparameters such as $$p_\theta(Z_s \mid Z_t) \equiv q\big(Z_s \mid Z_t, Y_\psi(Y_\varphi(Z_t))\big),$$2 and $$p_\theta(Z_s \mid Z_t) \equiv q\big(Z_s \mid Z_t, Y_\psi(Y_\varphi(Z_t))\big),$$3 require tuning, and the model introduces architectural and training complexity through two networks and an extra projection path (Rout et al., 24 May 2025). In inference-time structural anchoring, begin and end anchors must be manually specified, inference can slow down because of expand-and-probe loops, and the method relies on the diffusion LLM having learned good priors over end positions for the chosen anchors (Han et al., 3 Jun 2026). These limitations imply that anchoring is not a universal substitute for grammar-constrained decoding or task-specific fine-tuning; rather, it is a family of mechanisms for exposing latent structure to the denoising process.

The main research direction suggested by these works is a move from heuristic or static anchoring toward adaptive and hierarchical anchoring. Proposed directions include LLM-guided or adaptive anchor selection based on attention salience or gradient attributions, combination with more advanced samplers and larger-scale diffusion LMs, and use of anchors for explicit planning or hierarchical generation in autoregressive models (Rout et al., 24 May 2025). The structural-anchoring work further suggests richer, possibly hierarchical anchor schemes and more sophisticated length planners for diffusion LLMs, so that structure-aware generation becomes a native capability rather than a post hoc correction layer (Han et al., 3 Jun 2026).