Conv in Diffusion Language Models

• Convolutional Decoding (Conv) is a normalization-based inference method that reweights token probabilities using position-dependent gating to address the long decoding-window issue in diffusion language models.
• It computes local anchor densities through convolution over mask patterns and applies a global normalization to keep the overall token mass calibrated during decoding.
• Conv narrows the decoding window without hard segmentation, preserving bidirectionality while improving speed and quality compared to semi-autoregressive methods.

Convolutional Decoding, abbreviated “Conv,” is a normalization-based inference method for diffusion LLMs that narrows the decoding window without hard segmentation. In the formulation introduced for Masked Diffusion LLMs (MDLMs), Conv reweights per-position token probabilities using a position-dependent gate computed from the current mask configuration, then applies a global normalization so that the total token mass remains calibrated. Its stated purpose is to address the long decoding-window problem, in which tokens generated far from the available context become irrelevant or repetitive during parallel diffusion decoding (Seo et al., 18 Sep 2025).

1. Definition and scope

In the relevant diffusion-language-model literature, Conv is not a convolutional neural architecture in the usual CNN sense. It is an inference-time procedure applied to the reverse process of a masked diffusion LLM. The method “narrows the decoding window without hard segmentation,” thereby preserving the parallel and bidirectional character of diffusion decoding while changing which masked positions are most likely to unmask at a given step (Seo et al., 18 Sep 2025).

The problem setting is a fixed decoding window of length $L$. At inference step $t$, the model maintains a binary mask $m \in \{0,1\}^L$, where $m_i = 1$ indicates that position $i$ is still masked. The masked positions are

$W_t = \{ i \mid m_i = 1 \},$

and the unmasked positions, which act as anchors, are

$A_t = \{ i \mid m_i = 0 \}.$

For each masked position $i$, the distance to the nearest anchor is defined as

$d_t(i) = \min_{j \in A_t} |i-j|.$

This distance is central to the long decoding-window failure mode: positions with large $d_t(i)$ are weakly conditioned and are more likely to produce high-prior or repetitive tokens rather than semantically aligned continuations (Seo et al., 18 Sep 2025).

2. The long decoding-window problem in MDLM decoding

Masked diffusion LLMs decode multiple positions in parallel over $t$0 denoising steps. This gives them a structural advantage over autoregressive models, which decode one token per step, but it also introduces a failure mode that becomes acute when the decoding window is long. The paper characterizes this as the long decoding-window problem: tokens generated far from the prompt or previously unmasked content often become irrelevant or repetitive (Seo et al., 18 Sep 2025).

The formulation used in the paper is MDLM. The reverse model predicts a categorical distribution over vocabulary tokens at each masked position,

$t$1

where $t$2 denotes per-position logits. In the reverse process, a masked position may remain masked or may be replaced by a sampled token, with the expected number of newly unmasked tokens per step roughly $t$3. Because many positions are updated simultaneously, positions far from anchor tokens can become dominated by the token prior rather than by local semantic conditioning (Seo et al., 18 Sep 2025).

The paper describes two characteristic failure zones. One is a repetition zone, with outputs such as “:”, “Question”, and “Answer”. The other is a high-prior zone, with function words and punctuation such as “the”, “is”, and punctuation marks. These zones overwhelm the “meaning” zone when deterministic decoding is applied over a long unstructured window. Conv is designed as an inference-time correction for precisely this issue (Seo et al., 18 Sep 2025).

3. Mathematical formulation of Conv

Conv computes a local anchor-density statistic by a one-dimensional convolution over the current mask pattern. If $t$4 denotes whether position $t$5 is masked, then the local “anchor count” at position $t$6 is

$t$7

where $t$8 is a normalized kernel of width $t$9 with $m \in \{0,1\}^L$0 and support $m \in \{0,1\}^L$1. Larger $m \in \{0,1\}^L$2 indicates that position $m \in \{0,1\}^L$3 lies closer to already unmasked tokens (Seo et al., 18 Sep 2025).

This statistic is converted into a gate through a monotone squashing function. The paper reports that $m \in \{0,1\}^L$4 works best: $m \in \{0,1\}^L$5 with temperature $m \in \{0,1\}^L$6 and centering parameter $m \in \{0,1\}^L$7. In practice, $m \in \{0,1\}^L$8 is clipped to $m \in \{0,1\}^L$9, with the paper giving $m_i = 1$0 as an example (Seo et al., 18 Sep 2025).

Let $m_i = 1$1 be the model’s predicted categorical distribution over $m_i = 1$2 at position $m_i = 1$3. Conv then applies position-wise reweighting: $m_i = 1$4 Because this changes the aggregate probability mass, the method introduces a global normalization factor

$m_i = 1$5

where $m_i = 1$6 is the candidate set at position $m_i = 1$7, such as a top-$m_i = 1$8 subset. The normalized probabilities are then

$m_i = 1$9

This global normalization is explicitly intended to preserve the total token mass used by the unmasking schedule, so that Conv narrows the effective decoding region without accidentally changing the effective sample rate (Seo et al., 18 Sep 2025).

4. Decoding procedure and operational behavior

Conv is inserted into the MDLM inference loop after the model has produced per-position logits and, optionally, after candidate restriction such as top-$i$0. The paper’s stepwise procedure is: compute model predictions, restrict to top-$i$1 if desired, compute $i$2, form $i$3, apply global normalization, compute local unmask masses, sample or select positions to unmask, and optionally apply EOS-fill (Seo et al., 18 Sep 2025).

The unmasking mass for a masked position is described as

$i$4

with $i$5 given by the diffusion schedule. Positions can then be chosen either by Bernoulli sampling or by fixed-budget selection to match the expected unmask count. Conv therefore changes where the decoder concentrates its effort rather than changing the underlying denoising objective (Seo et al., 18 Sep 2025).

A key feature is that the effective decoding frontier moves continuously. Regions near anchors have larger $i$6, hence larger $i$7, and are more likely to unmask early. Once newly generated tokens become anchors, they increase $i$8 in neighboring positions, so the narrowed window propagates outward. Because this is achieved by smooth reweighting rather than block partitioning, Conv preserves bidirectionality. The paper emphasizes that this differs from semi-autoregressive decoding, which splits the sequence into blocks and thereby sacrifices bidirectionality while also inflating the effective time interval per block (Seo et al., 18 Sep 2025).

The method adds only an $i$9 overhead for the kernel operation, while preserving the parallel structure of diffusion decoding. The same paper also describes an optional EOS-fill step: once EOS is unmasked at position $W_t = \{ i \mid m_i = 1 \},$0, all positions to the right can be filled with EOS and marked unmasked. Reported speedups from EOS-fill are $W_t = \{ i \mid m_i = 1 \},$1 to $W_t = \{ i \mid m_i = 1 \},$2, especially in smaller models (Seo et al., 18 Sep 2025).

5. Interaction with Rejecting Rule-based Fine-Tuning

Conv is presented together with Rejecting Rule-based Fine-Tuning (R2FT), a post-hoc training scheme that targets repetitive and high-prior failures. R2FT is not part of Conv itself, but the paper treats the two methods as complementary: Conv narrows the effective window during inference, whereas R2FT changes the model’s ranking of desirable and undesirable continuations (Seo et al., 18 Sep 2025).

R2FT uses a preference-style objective contrasting wanted outputs $W_t = \{ i \mid m_i = 1 \},$3 and undesired outputs $W_t = \{ i \mid m_i = 1 \},$4: $W_t = \{ i \mid m_i = 1 \},$5 The negative samples are not arbitrary model outputs; they are synthesized by rule-based corruption that repeats a recent span into the future, thereby targeting the repetition behavior diagnosed in long-window diffusion decoding (Seo et al., 18 Sep 2025).

This coupling matters operationally. After R2FT, the paper reports that deterministic decoding with top-$W_t = \{ i \mid m_i = 1 \},$6 and global normalization becomes more effective, because the “meaning” zone rises relative to repetition and high-prior zones. Conv and R2FT are therefore presented as orthogonal interventions: one on the geometry of inference, the other on the learned distribution (Seo et al., 18 Sep 2025).

6. Empirical behavior, comparisons, and naming ambiguity

The reported empirical results show that Conv improves diffusion-language-model decoding quality at substantially smaller step counts than semi-autoregressive baselines. In the small MDLM setting ($W_t = \{ i \mid m_i = 1 \},$7M parameters, $W_t = \{ i \mid m_i = 1 \},$8, $W_t = \{ i \mid m_i = 1 \},$9), “Top-k+Global+Conv after R2FT” reaches AlpacaEval scores of LC $A_t = \{ i \mid m_i = 0 \}.$0 and win $A_t = \{ i \mid m_i = 0 \}.$1, while “Top-k+Global+Conv without R2FT” reaches LC $A_t = \{ i \mid m_i = 0 \}.$2 and win $A_t = \{ i \mid m_i = 0 \}.$3. In the same setting, the categorical baseline is LC $A_t = \{ i \mid m_i = 0 \}.$4, win $A_t = \{ i \mid m_i = 0 \}.$5, and LLADA with stride $A_t = \{ i \mid m_i = 0 \}.$6 is reported as LC $A_t = \{ i \mid m_i = 0 \}.$7, win $A_t = \{ i \mid m_i = 0 \}.$8 (Seo et al., 18 Sep 2025).

In the larger LLADA-8B-based setting, “Top-k+Global+Conv after R2FT” is reported at AlpacaEval LC $A_t = \{ i \mid m_i = 0 \}.$9 and win $i$0, compared with “LLADA-8B-Instruct” at LC $i$1 and win $i$2. On GSM8K with $i$3, $i$4, deterministic decoding, LLADA is reported at $i$5 and Conv at $i$6. The paper also states that Conv+R2FT at $i$7 achieves quality similar to semi-autoregressive decoding at $i$8, which it presents as a favorable speed–quality trade-off (Seo et al., 18 Sep 2025).

The comparison with semi-autoregressive decoding is central. Semi-AR improves local conditioning by block segmentation, but the paper argues that it causes “time-interval expansion,” because the effective step budget per block becomes $i$9. This violates the continuous-time assumption underlying MDLM training and degrades quality when $d_t(i) = \min_{j \in A_t} |i-j|.$0 is small. Conv is introduced specifically to avoid that trade-off by narrowing the effective window without block-wise decomposition (Seo et al., 18 Sep 2025).

The name also warrants disambiguation. The phrase “convolutional decoding” appears elsewhere in the literature with unrelated meanings, including CNN-based PAC-code decoders (Dai et al., 2024), U-net decoding for classical convolutional codes (Katz, 2020), and CNN decoders for surface codes (Bordoni et al., 2023). This suggests that the capitalized abbreviation “Conv” in diffusion-language-model work is method-specific rather than a field-wide standard term.

In that specific sense, Conv denotes a normalization-based, position-aware inference mechanism for diffusion LLMs: it uses a kernel over the current mask pattern to prioritize positions near anchors, globally renormalizes per-position token mass, preserves bidirectionality, and is intended to mitigate the long decoding-window pathology without resorting to hard segmentation (Seo et al., 18 Sep 2025).