VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

Abstract: MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by (+17.84) points over the original model and (+6.95) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.

• The paper introduces VoidPadding to decouple the [EOS] token’s dual role, eliminating EOS overflow in masked diffusion language models.
• Empirical results show significant performance gains, including up to +17.84 points improvement and robust accuracy even with large block sizes.
• The method leverages the [EOS]Termination rule and VoidExpansion to reduce forward passes by 84.6% while maintaining high output quality.

VoidPadding: Decoupling Semantic Termination from Padding in Masked Diffusion LLMs

Introduction

This paper introduces VoidPadding, a targeted training and inference co-design for masked diffusion LLMs (MDLMs). The objective is to resolve the "EOS overflow" failure mode observed during blockwise decoding in MDLMs by disentangling the roles of the EOS indicator as both semantic terminator and padding token. The proposed approach replaces the traditional re-use of [EOS] for padding (inherited from autoregressive LM (ARLM) conventions) with a new [VOID] token dedicated to padding, allowing [EOS] to serve solely as a semantic termination signal. The paper further establishes both an early stopping criterion ([EOS]Termination) and an adaptive canvas allocation strategy (VoidExpansion) based on the [VOID]-induced length signal.

Figure 1: Overview of VoidPadding. VoidPadding decouples termination from padding by using [EOS] only for stopping and [VOID] only for padding. During inference, [VOID] token generation is banned, decoding stops at the leftmost visible [EOS], and the canvas may be expanded based on tail [VOID] signals.

Analysis of EOS Overflow in MDLMs

In standard MDLM instruction tuning, response sequences are batch-padded for efficiency. Following ARLM traditions, padding uses [EOS], which in ARLMs is safe due to causal masking. In MDLMs, however, bidirectional attention exposes all padded positions—leading to a dual role where [EOS] acts as both terminator and padding target during training. Empirically, this produces EOS overflow: during inference, especially with large block sizes, the model generates spurious [EOS] tokens early and terminates responses prematurely, degrading output quality.

The analysis proceeds as follows:

• [EOS] as a Length Signal: Training with [EOS] padding teaches the model to infer response length from [EOS] position; high [EOS] probabilities at the canvas tail reflect the underlying response-length distribution.

  Figure 2: [EOS] as a learned length signal. Initial [EOS] confidence aligns with the [EOS] label ratio of padded data and the raw response-length CDF.

• Block Decoding Cascade: In blockwise decoding with $B=L$, initial decoding on a fully masked canvas causes the model to commit [EOS] tokens at high-confidence tail positions, initiating a leftward chain reaction (Stage 1 and Stage 2 overflow) where further [EOS] tokens propagate to subsequent positions via self-conditioning under bidirectional attention.
• Attention Masking Ablation: Masking out attention to already committed [EOS] positions prevents the leftward cascade and restores correct decoding trajectories.

Figure 3: Attention intervention on the $B=L=512$ [EOS] overflow trajectory. Both cases use the same prompt. Masking attention to committed [EOS] tokens avoids [EOS] overflow and recovers the correct answer.

• Root Cause: The fundamental issue is the mismatch between training (where [EOS] is a dual-purpose signal) and inference (where it is expected to denote termination only). ARLM causal masking shields padding from earlier tokens, avoiding this interaction.

VoidPadding Methodology

VoidPadding resolves the above mismatch by introducing a dedicated [VOID] token for padding and relieves [EOS] to its intended semantic role.

• Training Construction: Instruction-tuning output sequences are formatted as $[prompt]\ [response]\ [EOS]\ [VOID]^*$, ensuring a single [EOS], with all padding being [VOID]. The [EOS] token is always masked during training, enforcing prediction invariance.
• Inference Protocol: [VOID] tokens are banned from generation (logit set to $-\infty$). Decoding terminates at the leftmost visible [EOS], discarding all subsequent positions—this is the [EOS]Termination rule.
• VoidExpansion: Adaptive response-canvas allocation uses the learned [VOID] signal. If the tail region of the masked canvas displays low [VOID] preference relative to a non-[VOID] threshold, the canvas is expanded in block-aligned increments.

Empirical Results

Fixed-Length Block-Size Robustness

VoidPadding demonstrates superior robustness compared to both the original and RainbowPadding variants under various block sizes. For example, on Dream-7B-Instruct, VoidPadding yields a block-size-averaged four-task mean improvement of +17.84 points over the original model and +6.95 points over RainbowPadding, while on LLaDA-8B-Instruct the gains are +7.06 and +3.59, respectively. Notably, with $B=512$ on GSM8K, the original model accuracy collapses from 76.80% to 28.43%, while VoidPadding remains stable near 78%.

Decoding Efficiency

The [EOS]Termination rule decreases the number of forward-passes needed for decoding (NFE) by up to 84.6% (MBPP) on Dream, while increasing or maintaining response quality, representing a substantial practical efficiency gain.

VoidExpansion for Adaptive Canvas Allocation

Starting from short initial canvases ($L_0=64$), VoidExpansion maintains or surpasses the output quality of previous approaches like $\rho$-[EOS] and Daedal, while reducing mean NFE and wall-clock time. On Dream, VoidExpansion achieves a four-task mean of 60.37 compared to 39.20 (VOID fixed-length, $L_0=64$).

Large Block-Size Stress Testing

Variable-length allocation via inference-only [EOS]-based methods remains susceptible to overflow failure in large-block regimes ($B=256$), whereas VoidExpansion built on VoidPadding is stable and effective.

Ablations and Analysis

• [EOS]Termination rule is only robustly beneficial when paired with VoidPadding; applying it to [EOS] padding or RainbowPadding can reduce quality.
• VoidExpansion is robust to hyperparameters (tail-window size, thresholds, initial canvas length, block size).
• Benefits hold on both instruction-tuned and base MDLM checkpoints.

Theoretical and Practical Implications

VoidPadding not only addresses the specific EOS overflow pathology in MDLMs but also underscores the perils of naive ARLM-to-MDLM convention transfer (e.g., padding conventions, special-token semantics, batching strategies) in bidirectional denoising settings. It demonstrates that lightweight, MDLM-tailored changes to training and inference procedures can yield substantial improvements in both quality and efficiency.

Theoretical implications include the necessity for precise alignment of special-token semantics between training and inference, and the importance of context/exposure differences between unidirectional and bidirectional generation for the utility of special tokens. The framework supports future research on more advanced token-level allocation, sampler design, and further disambiguation of role signals in denoising generative paradigms.

Conclusion

VoidPadding provides an effective, implementation-light solution to the EOS overflow pathology in MDLMs by introducing a dedicated [VOID] padding token. This design enables early and reliable response termination, block-size and canvas-length robustness, and unlocks efficient, adaptive variable-length decoding. The methodology serves as a foundation for further architectural and protocol innovations in non-AR sequence modeling frameworks, with immediate implications for the deployment, efficiency, and reliability of next-generation LLMs.