Sample-Level Mask Normalization

• Sample-level mask normalization is a method that produces normalized prediction distributions over the entire vocabulary without using special [MASK] tokens.
• It integrates uniform-state diffusion models with joint CTC–USDM decoding by combining log-probabilities and leveraging robust log-space normalization to avoid numerical issues.
• Empirical evaluations show that this approach achieves up to a 0.30% absolute reduction in WER compared to traditional methods, enhancing overall model performance.

Sample-level mask normalization is a normalization strategy that arises in the context of non-autoregressive text denoising and sampling frameworks, particularly those leveraging uniform-state diffusion models (USDM) for speech recognition. In contrast to mask-based denoising models, which treat masked tokens with special symbols, USDM and related approaches define masking and normalization operations entirely without mask tokens by leveraging uniform corruption and categorical normalization over the full vocabulary at every sequence position.

1. Conceptual Framework

Sample-level mask normalization refers to the operation of producing a normalized prediction distribution over vocabulary tokens for every position in a noisy input sequence at each denoising step. In USDM, the forward noising procedure replaces sequence elements with samples uniformly drawn from the vocabulary rather than a dedicated [MASK] token. The denoiser predicts, for each position $i$ and vocabulary element $v_j \in V$, a categorical probability $P_{\mathrm{USDM},i}(v_j | z_{t_\ell})$ where $z_{t_\ell}$ is the noisy sequence at step $\ell$ (Naveriani et al., 15 Apr 2026).

A distinguishing feature is the absence of mask-specific normalization: every token's probability is normalized over the full vocabulary $V$, never conditioned on mask presence. This stands in contrast to masked diffusion LLMs (MDLM), which rely on special tokens and associated normalization.

2. Mathematical Specification

The core normalization operation within each denoising step is as follows. After computing raw log-probabilities for each token $v_j$ at position $i$, the probabilities are exponentiated and normalized across $V$:

$$P_{\mathrm{USDM},i}(v_j | z_{t_\ell}) = \frac{\exp(\log P_{\mathrm{USDM},i}(v_j | z_{t_\ell}))}{\sum_{v' \in V} \exp(\log P_{\mathrm{USDM},i}(v' | z_{t_\ell}))}$$

When combining USDM prediction with acoustic (CTC) predictions at position $v_j \in V$0, the combined log-probability is computed using log-linear interpolation:

$v_j \in V$1

This is normalized across the vocabulary with:

$v_j \in V$2

The normalization ensures the resulting probabilities sum to one at each position and prevents numerical overflow/underflow, typically implemented by subtracting the log-sum-exp over $v_j \in V$3 from all logits.

3. Operational Context: USDM and Mask-Free Noising

USDM's essential property is its "mask-free" sample corruption. Forward corruption at each step of USDM denoising replaces original sequence tokens with samples drawn from the uniform distribution over $v_j \in V$4, rather than marking corruptions with a special symbol. Consequently, the scoring and normalization process is unconstrained by mask token handling and treats every sequence position identically regardless of whether it originated as a "clean" or "corrupted" token.

This masking/noising design stands in contrast to MDLM, which enforces explicit [MASK] tokens—significantly impacting the normalization strategy and complicating token-level integration in joint frameworks.

4. Implementation Details and Practical Considerations

In the context of joint CTC–USDM decoding for speech recognition, sample-level mask normalization interacts closely with probability renormalization and log-space computation. Immediately prior to normalization, blank probabilities from CTC outputs are removed, and logits for non-blank tokens are renormalized over $v_j \in V$5. The normalized combined probabilities $v_j \in V$6 are then used for position-wise ancestral sampling at each denoising step.

The procedure ensures stability and correctness:

• Remove blank token probability from CTC, renormalize over $v_j \in V$7.
• Compute log-probability combination in log-space before exponentiation.
• Normalize each position's probability distribution across $v_j \in V$8 to unity, avoiding numerical issues (Naveriani et al., 15 Apr 2026).

Such normalization is critical to ensure that no position is influenced by residual blank scores or unnormalized probabilities, which would otherwise bias sampling and degrade denoising quality.

5. Empirical Impact and Model Performance

Sample-level mask normalization contributes directly to performance improvements observed in joint decoding frameworks employing USDM. For example, the CTC+USDM joint decoding framework, utilizing strict position-wise normalization at every denoising step, achieves a consistent 0.05–0.10% absolute gain in WER over static USDM rescoring and approximately 0.30% better than greedy CTC decoding, on LibriSpeech "dev-other" (Naveriani et al., 15 Apr 2026). The table below summarizes WERs under different settings:

Method	Training Epochs	K (steps)	WER (%)
Greedy CTC (baseline)	—	—	5.08
USDM rescoring (static)	5	256	4.82
CTC+USDM joint decoding	5	32	4.78
CTC+USDM joint decoding	25	64	4.71

A plausible implication is that stable, position-wise normalization not only facilitates integration between acoustic and LLM distributions but also supports effective denoising convergence within a practical number of steps (K = 32–64).

6. Distinction from Mask-Token Normalization

Sample-level mask normalization, as instantiated in USDM, avoids the use of [MASK] tokens entirely. The model always predicts over the entire vocabulary distribution, with no explicit distinction between corrupted and uncorrupted positions during normalization. In contrast, MDLM and classical masked language modeling approaches require normalization and sampling to be conditioned on the presence of mask tokens, potentially complicating their integration in token-level fusion and denoising procedures. USDM's approach—uniform forward corruption with sample-level normalization—enables greater modularity and facilitates hybrid decoding architectures (Naveriani et al., 15 Apr 2026).

7. Practical Recommendations and Limitations

Effective implementation of sample-level mask normalization requires:

• Tuning interpolation weights (λ{\mathrm{CTC}}, λ{\mathrm{DiffLM}}) on representative development data to balance acoustic and diffusion model contributions.
• Ensuring probability normalization steps are robust to under/overflow, especially for long sequences.
• Adjusting the starting noise level $v_j \in V$9 and denoising steps (K) to optimize performance and efficiency.
• Recognizing that while USDM normalization supports easier integration into joint decoding, MDLM may outperform in rescoring, though it introduces added complexities in per-position normalization logic.

Careful adherence to these principles underpins reproducible and efficient deployment of sample-level mask normalization in diffusion-based LLMs for ASR (Naveriani et al., 15 Apr 2026).