VoidPadding: Decoupling Padding in MDLMs

• VoidPadding is a method that reassigns padding from repeated [EOS] tokens to a dedicated [VOID] token, clearly separating semantic termination from non-semantic padding.
• The approach diagnoses [EOS] overflow as a bidirectional decoding failure, where repeated [EOS] tokens reinforce each other and truncate generated responses.
• Empirical results show that VoidPadding improves decoding efficiency and robustness, with block-size-averaged gains and significant reductions in normalized function evaluations.

Searching arXiv for the cited VoidPadding paper and closely related padding papers to ground the article. arXiv search query: all:("VoidPadding" OR "padding in LLMs" OR "Padding Module" OR "Masked diffusion LLMs") VoidPadding is a training–inference co-design for masked diffusion LLMs (MDLMs) that assigns padding to a dedicated \texttt{[VOID]} token and reserves \texttt{[EOS]} exclusively for semantic termination. In the formulation introduced for instruction-tuned MDLMs, the method targets a specific failure mode of bidirectional denoising over a preallocated masked response canvas: when repeated \texttt{[EOS]} tokens are used both as batch padding and as end-of-response markers, large-block decoding can induce \texttt{[EOS]} overflow, producing truncated answers followed by long \texttt{[EOS]} suffixes. VoidPadding decouples those roles, then exploits the learned \texttt{[EOS]} signal for early stopping and the learned \texttt{[VOID]} signal for adaptive canvas expansion (Liu et al., 16 Jun 2026).

1. Model setting and problem formulation

MDLMs generate text by denoising a preallocated masked response canvas in parallel with bidirectional attention. In instruction tuning, responses are commonly padded to the batch maximum length. Many MDLMs inherit the autoregressive convention of padding with repeated \texttt{[EOS]} tokens. That convention is benign in causal autoregressive decoding because earlier tokens cannot attend to right-padding, but in MDLMs the padding tokens are visible context and training targets under bidirectional attention. As a result, \texttt{[EOS]} is trained to encode two incompatible semantics: end of response and “position beyond the response length” (Liu et al., 16 Jun 2026).

The paper formalizes the resulting length prior through the raw response-length cumulative distribution function

$F(m)=\Pr(\ell \le m).$

With $e$ denoting \texttt{[EOS]}, the initial inferred \texttt{[EOS]} probability is described as aligning with both the padded-data \texttt{[EOS]} ratio and the model’s own initial prediction,

$F(i-1)\approx p_i(e)\approx q_i(e).$

This establishes the central diagnosis: the padding convention itself induces a learned positional preference for \texttt{[EOS]} in the suffix of the response canvas, and that positional bias becomes operational under large-block denoising.

A common misconception is that padding-token choice is a minor implementation detail. In the MDLM setting considered here, it is not. Padding tokens participate directly in denoising and therefore alter the learned response-length model. VoidPadding is defined precisely at that interface between batching convention and sequence semantics.

2. \texttt{[EOS]} overflow as a bidirectional decoding cascade

The paper identifies \texttt{[EOS]} overflow as the failure mode in which the model emits a short substantive response and then commits a long suffix of \texttt{[EOS]} tokens, especially at large block sizes. The mechanism is two-stage. First, tail positions learn high $\texttt{[EOS]}$ probability because training padded those positions with repeated \texttt{[EOS]}). Second, once a suffix of \texttt{[EOS]}) is committed during decoding, those visible tokens reinforce further \texttt{[EOS]}) predictions to the left under bidirectional attention (Liu et al., 16 Jun 2026).

The start of the high-confidence \texttt{[EOS]} tail is defined as

$r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$

If the first block reaches that region, all selected positions there can be committed as \texttt{[EOS]}. The paper then gives a theorem for the committed-\texttt{[EOS]} suffix event $S_e$: $q_i(e\mid S_e)=\frac{F(i-1)}{F(r-1)}.$ If

$2F(i-1)>F(r-1),$

then position $i$ also commits to \texttt{[EOS]}. This converts a suffix preference into a self-reinforcing leftward cascade.

The authors further note that the failure is not simply a capacity bottleneck. When \texttt{[EOS]} is banned during decoding, GSM8K accuracy at $L=B=512$ rises sharply, indicating that the model retains latent ability to generate longer useful outputs; the degradation arises from the learned \texttt{[EOS]} prior rather than an inability to represent the answer.

3. Training design: separating padding from termination

VoidPadding replaces repeated \texttt{[EOS]} padding with a dedicated \texttt{[VOID]} token. Under the conventional target construction, a response $e$0 padded to batch length $e$1 is represented as

$e$2

VoidPadding instead trains on

$e$3

The single \texttt{[EOS]} remains the semantic terminator, while all residual tail positions become explicitly non-semantic padding (Liu et al., 16 Jun 2026).

This change does not alter the base MDLM denoising objective; it changes the targets to which that objective is applied: $e$4 An important implementation detail is that the single \texttt{[EOS]} token is always masked, so the model is explicitly trained to predict termination, while standard masking is used for all non-\texttt{[EOS]} positions. The reported instruction tuning uses LoRA fine-tuning on the same 0.5M-example dataset used by RainbowPadding.

The semantic effect is direct. \texttt{[EOS]} is no longer required to serve as a length marker, and \texttt{[VOID]} is no longer required to carry termination meaning. This removes the training pressure that previously conflated sequence completion with padded suffix occupancy.

4. Inference procedures: \texttt{[EOS]Termination} and VoidExpansion

VoidPadding modifies inference in two coupled ways. First, \texttt{[VOID]} is banned from visible output by setting its decoding logit to $e$5. Second, decoding uses \texttt{[EOS]Termination}: instead of waiting until every masked position in the canvas is filled, decoding stops once there are no masks remaining before the leftmost visible \texttt{[EOS]}. This allows semantic completion to terminate generation without denoising an already-useless suffix (Liu et al., 16 Jun 2026).

The method also supports adaptive response-canvas expansion, termed VoidExpansion. For prompt tokens $e$6, mask token $e$7, tail-window size $e$8, and current canvas length $e$9, the model is evaluated on

$F(i-1)\approx p_i(e)\approx q_i(e).$0

Over the tail window $F(i-1)\approx p_i(e)\approx q_i(e).$1, the method defines

$F(i-1)\approx p_i(e)\approx q_i(e).$2

$F(i-1)\approx p_i(e)\approx q_i(e).$3

$F(i-1)\approx p_i(e)\approx q_i(e).$4

$F(i-1)\approx p_i(e)\approx q_i(e).$5

Canvas sufficiency is then determined by the joint criterion

$F(i-1)\approx p_i(e)\approx q_i(e).$6

If the condition fails, one more block is appended and the test is repeated up to $F(i-1)\approx p_i(e)\approx q_i(e).$7. In the paper’s interpretation, \texttt{[EOS]} becomes a learned semantic endpoint detector, whereas \texttt{[VOID]} becomes a learned signal of tail insufficiency or surplus. VoidPadding therefore turns two formerly entangled token roles into two separate control signals.

5. Empirical behavior and benchmark results

The reported empirical pattern is robustness across block sizes together with substantial decoding-efficiency gains. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by $F(i-1)\approx p_i(e)\approx q_i(e).$8 points over the original model and $F(i-1)\approx p_i(e)\approx q_i(e).$9 points over RainbowPadding. The reported four-task means are $\texttt{[EOS]}$0 for the original model, $\texttt{[EOS]}$1 for RainbowPadding, and $\texttt{[EOS]}$2 for VoidPadding (Liu et al., 16 Jun 2026).

On LLaDA-8B-Instruct, the block-size-averaged four-task mean improves by $\texttt{[EOS]}$3 points over the original model and $\texttt{[EOS]}$4 points over RainbowPadding. The paper highlights fixed-length robustness more concretely with GSM8K: on the original LLaDA model, accuracy drops from $\texttt{[EOS]}$5 at $\texttt{[EOS]}$6 to $\texttt{[EOS]}$7 at $\texttt{[EOS]}$8, whereas VoidPadding remains near $\texttt{[EOS]}$9.

Efficiency gains are tied to \texttt{[EOS]Termination}. For Dream, averaged over $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$0, reported NFE reductions are $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$1 on GSM8K, $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$2 on MATH500, $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$3 on HumanEval, and $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$4 on MBPP, with the abstract summarizing an average decoding-NFE reduction of $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$5. Absolute NFE values are reported as $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$6 for the original and RainbowPadding baselines, versus $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$7 on GSM8K, $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$8 on MATH500, $r=\min\{i\in\{1,\dots,L\}: F(i-1)>1/2\}.$9 on HumanEval, and $S_e$0 on MBPP for VoidPadding.

Ablations further delimit the method’s operating conditions. On LLaDA, \texttt{[EOS]Termination} yields almost no benefit or slight harm under the original and RainbowPadding models, but under VoidPadding it raises the mean score from $S_e$1 to $S_e$2 while reducing mean NFE by $S_e$3. The paper also reports that removing the \texttt{[VOID]} ban causes performance collapse at larger block sizes, showing that \texttt{[VOID]} is intended as an internal padding-length signal rather than an output token.

6. Relation to broader padding research and terminological scope

Outside MDLMs, the term “VoidPadding” is not standard, and several nearby papers address different padding problems without using the term explicitly. In convolutional vision models, a trainable “Padding Module” learns border extensions from neighboring content through local MSE training and serves as a drop-in replacement for zero, reflection, or replication padding (Alrasheedi et al., 2023). In adversarial robustness studies for CNNs, padding is analyzed as a hidden architectural factor that shapes boundary-localized perturbation anomalies; that work compares zero, reflect, replicate, circular, and no-padding settings rather than introducing a VoidPadding method (Gavrikov et al., 2023).

In LLMs more generally, padding tokens have been shown to be behaviorally non-inert when masking is mishandled: controlled prepended padding can shift hidden activations, degrade generation quality, alter bias non-monotonically, and weaken safety refusals, with $S_e$4 reported at $S_e$5 pads for Llama-3.1-8B in one setup (Himelstein et al., 23 Sep 2025). A separate line of work addresses padding inefficiency rather than semantic interference: sequence packing formalizes batch construction as a bin-packing problem and reports that up to $S_e$6 of all tokens can be padding, with up to $S_e$7 in less common but not extreme cases such as GLUE-CoLA at sequence length $S_e$8 (Krell et al., 2021).

These neighboring results suggest a useful distinction. In CNN and batching work, padding is treated primarily as a boundary condition or throughput inefficiency. In VoidPadding for MDLMs, padding is instead treated as a learned symbol whose semantics must be disentangled from termination. The innovation is therefore not merely “better padding,” but role separation within bidirectional generative denoising.

7. Interpretation, misconceptions, and limitations

The primary misconception addressed by VoidPadding is that reusing \texttt{[EOS]} for padding should transfer harmlessly from autoregressive practice to MDLMs. The paper argues that this fails because autoregressive causal masking and MDLM bidirectional denoising expose padding to computation in fundamentally different ways. Under bidirectional attention, a padding convention becomes part of the model’s learned length prior (Liu et al., 16 Jun 2026).

A second misconception is that the method is equivalent to simply banning \texttt{[EOS]} or increasing response length. The reported results do not support that interpretation. VoidPadding depends on aligned training and inference: \texttt{[EOS]} must be trained as a pure terminator, \texttt{[VOID]} must be trained as pure padding, \texttt{[VOID]} must be banned from outputs, and early stopping must be keyed to the learned \texttt{[EOS]} signal. The ablation behavior indicates that removing any of these pieces substantially weakens the method.

The term itself also admits ambiguity. Several papers in other domains discuss “voids,” “padding-free” computation, or conceptually similar length-aware padding, but they are not the same object. The most precise use of “VoidPadding” presently refers to the MDLM method introduced for Dream-7B-Instruct and LLaDA-8B-Instruct (Liu et al., 16 Jun 2026). A plausible implication is that future work on non-autoregressive or bidirectional generative models will need to treat padding tokens as part of sequence semantics rather than as neutral bookkeeping, especially wherever length modeling, termination, and blockwise parallel decoding are tightly coupled.