Masked Prediction Distribution (MPD)

• Masked Prediction Distribution (MPD) is a conditional law that predicts missing tokens from partially observed inputs using methodologies like diffusion and masked decoding.
• It is applied in causal audio generation and masked diffusion language models to enable efficient token-wise predictions and asynchronous decoding.
• MPD also underpins theoretical identifiability analyses in latent-variable models, linking optimal predictive behavior to model parameter recovery.

Searching arXiv for the cited papers to ground the article in current sources. Masked Prediction Distribution (MPD) denotes the conditional law used to predict masked or skipped content from partially observed context, but its precise formalization depends on the modeling regime. In causal audio generation with continuous-valued tokens, MPD is the diffusion-based conditional distribution $p_\theta(x_f \mid x_{< i}, w, v, p_t)$ for a future token selected by a masking pattern (Yang et al., 14 Jul 2025). In masked diffusion LLMs, MPD can be defined as the per-position categorical distribution $p_\theta(\cdot \mid X_{\text{in}}(t))^i$ induced by the current continuous decoding state (Wang et al., 27 Jun 2026). In the identifiability analysis of masked prediction tasks, MPD is the true conditional law $P(x_t \mid x_{-t}; \theta)$ under the generative model, and the optimal masked predictor should realize it (Liu et al., 2022). Taken together, these formulations place MPD at the intersection of conditional modeling, self-supervised prediction, and decoding dynamics.

1. Definitions across modeling paradigms

The cited literature does not present MPD as a single universal object; instead, it specifies closely related conditional distributions for different architectures and data types. A concise comparison is useful.

Setting	MPD form	Role
Causal audio LM	$p_\theta(x_f \mid x_{< i}, w, v, p_t)$	Diffusion-based next-token distribution
MDLM continuous decoding	$p_\theta(\cdot \mid X_{\text{in}}(t))^i$	Per-step belief over tokens
Masked-prediction theory	$P(x_t \mid x_{-t}; \theta)$	True conditional law under the model

In "Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction" (Yang et al., 14 Jul 2025), the relevant task is masked next-token prediction with continuous-valued tokens. An audio clip is mapped by a VAE encoder into a sequence of continuous tokens $x = \{x_1,\dots,x_n\}$ with $x_t \in \mathbb{R}^h$, conditioned on a text prompt $w$. A mask variable $v = \{v_1,\dots,v_n\}$ drops tokens to form a shorter sequence $p_\theta(\cdot \mid X_{\text{in}}(t))^i$0, and for a visible position $p_\theta(\cdot \mid X_{\text{in}}(t))^i$1 the prediction target is $p_\theta(\cdot \mid X_{\text{in}}(t))^i$2, a future token that may be multiple steps ahead in original time. The MPD is then the conditional next-token distribution for that target under causal constraints (Yang et al., 14 Jul 2025).

In "Masked Diffusion Decoding as $p_\theta(\cdot \mid X_{\text{in}}(t))^i$3-Prediction Flow" (Wang et al., 27 Jun 2026), MPD is introduced as an inferred term for the per-position predictive categorical law in masked diffusion LLMs. The vocabulary is augmented with a special mask token $p_\theta(\cdot \mid X_{\text{in}}(t))^i$4, and the model produces logits $p_\theta(\cdot \mid X_{\text{in}}(t))^i$5 and probabilities $p_\theta(\cdot \mid X_{\text{in}}(t))^i$6. Under continuous decoding, the same object is evaluated on a continuous input state $p_\theta(\cdot \mid X_{\text{in}}(t))^i$7 rather than only on a discrete masked string, yielding a continuously evolving MPD (Wang et al., 27 Jun 2026).

In "Masked prediction tasks: a parameter identifiability view" (Liu et al., 2022), MPD is the true conditional law of a masked token given the observed context under a parametric latent-variable model. For a masked position $p_\theta(\cdot \mid X_{\text{in}}(t))^i$8, the quantity of interest is $p_\theta(\cdot \mid X_{\text{in}}(t))^i$9. Under proper loss functions, the optimal masked predictor recovers this true conditional; under squared loss it becomes the conditional expectation, and for discrete one-hot targets that conditional expectation equals the categorical MPD vector (Liu et al., 2022).

2. MPD in causal audio language modeling with continuous-valued tokens

The audio formulation is explicitly probabilistic and diffusion-based. For any visible position $P(x_t \mid x_{-t}; \theta)$0 and target $P(x_t \mid x_{-t}; \theta)$1, the masked prediction distribution is

$P(x_t \mid x_{-t}; \theta)$2

The Transformer decoder is causal: it sees the visible past $P(x_t \mid x_{-t}; \theta)$3, the prompt $P(x_t \mid x_{-t}; \theta)$4, a BOS token, and a target positional embedding $P(x_t \mid x_{-t}; \theta)$5 indicating which future index is to be predicted. The diffusion head then defines the MPD through a token-wise denoising diffusion process (Yang et al., 14 Jul 2025).

The forward process uses the standard variance-preserving DDPM parameterization with a cosine schedule. With $P(x_t \mid x_{-t}; \theta)$6, $P(x_t \mid x_{-t}; \theta)$7, and $P(x_t \mid x_{-t}; \theta)$8, the noised target token satisfies

$P(x_t \mid x_{-t}; \theta)$9

equivalently,

$p_\theta(x_f \mid x_{< i}, w, v, p_t)$0

The reverse process conditions on the causal context vector $p_\theta(x_f \mid x_{< i}, w, v, p_t)$1 produced by the Transformer decoder and the diffusion step $p_\theta(x_f \mid x_{< i}, w, v, p_t)$2. With an $p_\theta(x_f \mid x_{< i}, w, v, p_t)$3-prediction parameterization, the diffusion head predicts

$p_\theta(x_f \mid x_{< i}, w, v, p_t)$4

and the reverse transition is Gaussian,

$p_\theta(x_f \mid x_{< i}, w, v, p_t)$5

with mean

$p_\theta(x_f \mid x_{< i}, w, v, p_t)$6

This yields the implicit diffusion-chain definition

$p_\theta(x_f \mid x_{< i}, w, v, p_t)$7

with prior $p_\theta(x_f \mid x_{< i}, w, v, p_t)$8 (Yang et al., 14 Jul 2025).

Training minimizes the standard denoising diffusion MSE loss token-wise, averaged over dataset samples $p_\theta(x_f \mid x_{< i}, w, v, p_t)$9, masking patterns $p_\theta(\cdot \mid X_{\text{in}}(t))^i$0, visible positions $p_\theta(\cdot \mid X_{\text{in}}(t))^i$1, diffusion steps $p_\theta(\cdot \mid X_{\text{in}}(t))^i$2, and Gaussian noise $p_\theta(\cdot \mid X_{\text{in}}(t))^i$3:

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$4

The paper emphasizes that masking is implemented by dropping rather than inserting special mask tokens, reducing sequence length and compute when the masking ratio is high. The masking-ratio distribution is a mixture schedule,

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$5

where the Normal component emphasizes high masking ratios and the truncated-normal component contributes a long tail of low ratios to reduce train-test mismatch for standard next-token decoding (Yang et al., 14 Jul 2025).

This formulation differs from both standard autoregressive next-token prediction on discrete tokens and masked language modeling. In the discrete autoregressive setting, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$6 is modeled by a softmax over a fixed vocabulary and trained with cross-entropy. Here the next-token law is continuous and modeled via token-wise diffusion, with denoising MSE on $p_\theta(\cdot \mid X_{\text{in}}(t))^i$7 rather than cross-entropy. In contrast to bidirectional MLM, the masked next-token task is strictly causal at train and test time: the model predicts a future token using only the visible past, and the target positional embedding disambiguates which future index is being predicted. The paper states that removing this target positional embedding severely hurts performance because different future targets otherwise conflict during training (Yang et al., 14 Jul 2025).

3. MPD in masked diffusion LLMs and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$8-prediction flow

In masked diffusion LLMs, MPD is the token distribution the decoder predicts at each position and each diffusion-progress value. The basic MDLM training objective is masked cross-entropy over masked positions,

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$9

and for an input $P(x_t \mid x_{-t}; \theta)$0 the mask predictor produces logits and per-position probabilities

$P(x_t \mid x_{-t}; \theta)$1

The MPD at position $P(x_t \mid x_{-t}; \theta)$2 and progress $P(x_t \mid x_{-t}; \theta)$3 is then defined as

$P(x_t \mid x_{-t}; \theta)$4

This is the central object preserved across continuous decoding steps (Wang et al., 27 Jun 2026).

The continuous state is anchored at the mask embedding $P(x_t \mid x_{-t}; \theta)$5 and interpolates toward the clean embedding $P(x_t \mid x_{-t}; \theta)$6 by a per-token progress vector $P(x_t \mid x_{-t}; \theta)$7:

$P(x_t \mid x_{-t}; \theta)$8

At $P(x_t \mid x_{-t}; \theta)$9, the state is the mask embedding; at $x = \{x_1,\dots,x_n\}$0, it is the clean embedding. The model forms a clean-state prediction either by an argmax-embedding readout,

$x = \{x_1,\dots,x_n\}$1

or by a soft embedding readout,

$x = \{x_1,\dots,x_n\}$2

A velocity field then moves the current state toward the predicted clean state,

$x = \{x_1,\dots,x_n\}$3

with update

$x = \{x_1,\dots,x_n\}$4

Because $x = \{x_1,\dots,x_n\}$5 is evaluated from $x = \{x_1,\dots,x_n\}$6 at every step, its evolution is induced by this state update; the paper states that the distribution sharpens as $x = \{x_1,\dots,x_n\}$7 approaches the clean embedding (Wang et al., 27 Jun 2026).

A further alignment objective stabilizes clean-state prediction in embedding space:

$x = \{x_1,\dots,x_n\}$8

This alignment is introduced because continuous decoding requires stability on clean inputs as well as masked ones (Wang et al., 27 Jun 2026).

The decoding schedule is asynchronous and confidence-based. Each token has its own progress $x = \{x_1,\dots,x_n\}$9, updated by

$x_t \in \mathbb{R}^h$0

where $x_t \in \mathbb{R}^h$1 is selected by a learned policy. Confidence at position $x_t \in \mathbb{R}^h$2 is

$x_t \in \mathbb{R}^h$3

The paper also adds two discrete adjustments on top of the continuous flow. First, re-editing resets a token if its confidence falls substantially below its current commitment:

$x_t \in \mathbb{R}^h$4

when $x_t \in \mathbb{R}^h$5. Second, hard commitment sets the token with the highest confidence among those with $x_t \in \mathbb{R}^h$6 to $x_t \in \mathbb{R}^h$7 at each step (Wang et al., 27 Jun 2026).

The paper’s interpretation is explicit: standard discrete masked decoding discards the predictive distribution between steps by reducing it to a committed token or a fully masked state, whereas continuous decoding preserves and exploits the full MPD. This enables revisable partial progress, token-wise asynchrony, and more efficient use of limited decoding budget (Wang et al., 27 Jun 2026).

4. MPD as the optimal conditional law in masked-prediction theory

The theoretical treatment in the identifiability paper starts from a generative model rather than a decoder. There, MPD is the true conditional law of a masked token given observed context:

$x_t \in \mathbb{R}^h$8

The principal question is whether this object, or a family of such objects induced by a masked prediction task, identifies the parameters of the underlying latent-variable model. The paper studies Hidden Markov Models with both discrete and conditionally Gaussian observations and defines identifiability as injectivity of the mapping from the model parameters, modulo hidden-state permutations, to the optimal predictors (Liu et al., 2022).

In the discrete HMM case, with transition matrix $x_t \in \mathbb{R}^h$9 and emission matrix $w$0, the optimal masked predictor for $w$1 given $w$2 is

$w$3

where $w$4 is the posterior over the hidden state at time $w$5 given $w$6. Since $w$7 is a one-hot vector, this conditional expectation is exactly the categorical MPD vector $w$8. The paper also gives the more explicit one-sided formula

$w$9

under its notation and assumptions (Liu et al., 2022).

For conditionally Gaussian HMMs with identity covariance and emission means $v = \{v_1,\dots,v_n\}$0, the MPD for $v = \{v_1,\dots,v_n\}$1 given $v = \{v_1,\dots,v_n\}$2 is a mixture of Gaussians,

$v = \{v_1,\dots,v_n\}$3

where $v = \{v_1,\dots,v_n\}$4. Its conditional mean, which is the optimal square-loss predictor, is

$v = \{v_1,\dots,v_n\}$5

with $v = \{v_1,\dots,v_n\}$6 (Liu et al., 2022).

For general masked positions in an HMM with both-side context, the MPD is obtained by smoothing. Let $v = \{v_1,\dots,v_n\}$7 and $v = \{v_1,\dots,v_n\}$8. Then the latent posterior at the masked time is

$v = \{v_1,\dots,v_n\}$9

with standard forward-backward recursions, and the masked-token law is

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$00

The optimal predictor for one-hot $p_\theta(\cdot \mid X_{\text{in}}(t))^i$01 equals this MPD vector (Liu et al., 2022).

The paper’s central conclusion is that the informativeness of MPD depends strongly on the masked task. For discrete HMMs, pairwise single-token predictions such as $p_\theta(\cdot \mid X_{\text{in}}(t))^i$02 are non-identifiable, even when multiple pairwise tasks are combined, because they only constrain non-unique matrix products. By contrast, multi-token masked prediction of adjacent tokens, such as $p_\theta(\cdot \mid X_{\text{in}}(t))^i$03, is identifiable via tensor decomposition under Kruskal-rank conditions. For Gaussian HMMs, even the pairwise task $p_\theta(\cdot \mid X_{\text{in}}(t))^i$04 is identifiable, due to the structure of the posterior $p_\theta(\cdot \mid X_{\text{in}}(t))^i$05 and the identifiability of Gaussian mixtures (Liu et al., 2022).

5. Architectures, objectives, and decoding mechanisms

Although the three treatments of MPD arise in different settings, each couples a conditional distribution with a specific optimization and inference mechanism.

In the audio model, MPD is realized by a decoder-only Transformer $p_\theta(\cdot \mid X_{\text{in}}(t))^i$06 plus a diffusion head $p_\theta(\cdot \mid X_{\text{in}}(t))^i$07. The backbone takes text prompt tokens from CLAP and FLAN-T5 embeddings, a BOS token, visible past audio tokens, content positional embeddings $p_\theta(\cdot \mid X_{\text{in}}(t))^i$08, and a target positional embedding $p_\theta(\cdot \mid X_{\text{in}}(t))^i$09. The diffusion head is a small MLP predicting $p_\theta(\cdot \mid X_{\text{in}}(t))^i$10. The paper uses a cosine schedule, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$11 diffusion steps and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$12 steps by default. It states that the MLP diffusion head is the only change from a standard decoder LLM head, and that all other infrastructure, including KV-cache and streaming, remains applicable (Yang et al., 14 Jul 2025).

Inference in that system is standard left-to-right decoding with no masking and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$13 set to the next position, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$14. Each next token is sampled by a diffusion denoising chain from Gaussian noise to the clean token. The paper also reports optional classifier-free guidance: during training, the prompt $p_\theta(\cdot \mid X_{\text{in}}(t))^i$15 is replaced with a learned “fake/uncond” embedding $p_\theta(\cdot \mid X_{\text{in}}(t))^i$16 with probability $p_\theta(\cdot \mid X_{\text{in}}(t))^i$17, and at test time the conditional and unconditional predictions are blended with an annealed guidance scale. It states that $p_\theta(\cdot \mid X_{\text{in}}(t))^i$18 is a good default and standard temperature $p_\theta(\cdot \mid X_{\text{in}}(t))^i$19 is used (Yang et al., 14 Jul 2025).

In the masked diffusion decoding paper, the backbone MDLM is complemented by a lightweight policy network that chooses per-token step sizes from features extracted from the MPD and decoding state. The policy feature vector is

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$20

where $p_\theta(\cdot \mid X_{\text{in}}(t))^i$21 are the top probabilities, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$22 is the confidence margin, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$23 is normalized entropy over the top-$p_\theta(\cdot \mid X_{\text{in}}(t))^i$24 probabilities with $p_\theta(\cdot \mid X_{\text{in}}(t))^i$25, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$26 is current token progress, and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$27 is normalized global step index. The policy is a two-layer MLP with SiLU activations and predicts Beta mean and concentration,

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$28

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$29

followed by

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$30

and

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$31

The policy is trained with GRPO, and the reward is

$p_\theta(\cdot \mid X_{\text{in}}(t))^i$32

This makes MPD not only a prediction object but also a control signal for scheduling and commitment decisions (Wang et al., 27 Jun 2026).

In the identifiability framework, the “architecture” is deliberately abstracted away. The critical assumption is that the predictor class contains the true conditional law, so that under correct specification, infinite data, and proper loss, the minimizer equals the MPD. This suggests a methodological distinction: in the generative audio and MDLM papers, MPD is operationalized by specific neural parameterizations and sampling rules, whereas in the identifiability paper it is the target statistical object against which predictor classes are evaluated (Liu et al., 2022).

6. Empirical behavior, misconceptions, and open directions

The audio paper reports that continuous tokens plus diffusion next-token prediction already outperform a previous discrete autoregressive baseline. On AudioCaps, using only AudioCaps plus WavCaps for training, AudioNTP Base (193M) yields $p_\theta(\cdot \mid X_{\text{in}}(t))^i$33 and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$34 relative improvements over AudioGen Base (285M) in FAD and KL respectively. Adding masked next-token prediction to obtain AudioMNTP Base yields a further $p_\theta(\cdot \mid X_{\text{in}}(t))^i$35 FAD, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$36 KL, and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$37 CLAP relative gains over AudioNTP Base. AudioMNTP Large (462M) achieves the best FD and FAD among all compared systems and is near the leading diffusion models on KL, IS, and CLAP, while remaining causal and streamable. Relative FAD improvements versus AudioGen are $p_\theta(\cdot \mid X_{\text{in}}(t))^i$38 against AudioGen Base (285M) and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$39 against AudioGen Large (1B). Human evaluation indicates that AudioMNTP approaches Tango 2 and significantly outperforms AudioGen in both text relevance and overall quality, with particular strength on speech OVL (Yang et al., 14 Jul 2025).

The masked diffusion decoding paper evaluates efficiency under limited decoding budget. On HumanEval with LLaDA-8B-Instruct, standard masked decoding at full budget $p_\theta(\cdot \mid X_{\text{in}}(t))^i$40 obtains $p_\theta(\cdot \mid X_{\text{in}}(t))^i$41, while $p_\theta(\cdot \mid X_{\text{in}}(t))^i$42-prediction flow at $p_\theta(\cdot \mid X_{\text{in}}(t))^i$43 budget obtains $p_\theta(\cdot \mid X_{\text{in}}(t))^i$44, which the paper describes as $p_\theta(\cdot \mid X_{\text{in}}(t))^i$45 of the full-budget baseline. On HumanEval with LLaDA2.0-mini, mask prediction at $p_\theta(\cdot \mid X_{\text{in}}(t))^i$46 budget gives $p_\theta(\cdot \mid X_{\text{in}}(t))^i$47 and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$48-prediction flow at the same budget gives $p_\theta(\cdot \mid X_{\text{in}}(t))^i$49, a $p_\theta(\cdot \mid X_{\text{in}}(t))^i$50 absolute improvement. MBPP shows similar gains at $p_\theta(\cdot \mid X_{\text{in}}(t))^i$51 budget: $p_\theta(\cdot \mid X_{\text{in}}(t))^i$52 for LLaDA-8B and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$53 for LLaDA2.0-mini. Ablations at $p_\theta(\cdot \mid X_{\text{in}}(t))^i$54 budget on HumanEval for LLaDA-8B report $p_\theta(\cdot \mid X_{\text{in}}(t))^i$55 for full $p_\theta(\cdot \mid X_{\text{in}}(t))^i$56-prediction flow, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$57 without hard commitment, $p_\theta(\cdot \mid X_{\text{in}}(t))^i$58 without re-editing, and $p_\theta(\cdot \mid X_{\text{in}}(t))^i$59 without $p_\theta(\cdot \mid X_{\text{in}}(t))^i$60-prediction alignment (Wang et al., 27 Jun 2026).

Several common misconceptions are directly addressed by these results and formulations. First, masked prediction need not be bidirectional. The audio MNTP setup is strictly causal at train and test time, even though it incorporates masking by dropping tokens and predicting skip tokens from sparse past context (Yang et al., 14 Jul 2025). Second, MPD is not restricted to discrete vocabularies. In the audio setting it is a continuous conditional distribution over latent vectors modeled by token-wise diffusion rather than by a softmax (Yang et al., 14 Jul 2025). Third, preserving the full predictive distribution can matter independently of the underlying backbone. The MDLM paper argues that standard masked decoding discards runner-up probabilities and forces premature, irrevocable commitments, whereas continuous decoding retains the full MPD and exploits it for re-editing and asynchronous scheduling (Wang et al., 27 Jun 2026). Fourth, stronger masked objectives do not automatically imply stronger parameter recovery. The identifiability results show that for discrete HMMs, even multiple pairwise MPDs can remain non-identifying, while adjacent multi-token prediction can become identifiable through tensor structure (Liu et al., 2022).

Open directions are stated most explicitly in the audio and MDLM work. The audio paper notes that a causal decoder remains less expressive than large, fully bidirectional latent diffusion systems, that the system currently depends on a VAE and a vocoder in the AudioLDM pipeline, and that parallel decoding via target positional manipulation is a promising research direction (Yang et al., 14 Jul 2025). The MDLM paper identifies failure modes tied to the embedding-linearity assumption, premature or delayed commitments, oscillations caused by re-editing, and uneven contextual constraints under asynchronous schedules (Wang et al., 27 Jun 2026). A plausible implication is that MPD is becoming not only a target of prediction but also an internal state variable for generation, control, and theoretical analysis.