Masked Next-Token Prediction

Updated 3 July 2026

Masked Next-Token Prediction is a framework that enables Transformer-based models to predict multiple future tokens in parallel using specialized masking techniques.
It leverages diverse masking protocols and advanced attention schemes to bridge traditional autoregressive and bidirectional prediction strategies for enhanced inference.
The approach incorporates auxiliary modules such as sampler networks and latent consistency matching to boost accuracy across applications in language, vision, and audio.

Masked next-token prediction (MNTP) generalizes the classical sequential next-token prediction paradigm by enabling Transformer-based models to predict multiple future tokens in parallel through the introduction of mask tokens or masking mechanisms. This framework encompasses a spectrum of methodologies, from autoregressive decoding with masked inputs in language modeling to generalized masked prediction schemes for continuous-valued data and causal prediction in other modalities such as vision and audio. It leverages architectural, algorithmic, and training innovations to substantially increase inference throughput, enhance optimization, or facilitate multi-tasking, while maintaining or even improving accuracy. MNTP serves as an intermediate between strictly causal (autoregressive) and bidirectional (fully masked) prediction strategies, unifying or bridging them for diverse generative and predictive objectives.

1. Foundational Formulations and Variants

The core principle in masked next-token prediction is to allow the model to predict one or more tokens beyond the current context, either by explicitly inserting mask tokens or by masking certain positions according to a schema tailored to the data modality.

In the language modeling context, given a sequence $X = [x_1,\dots,x_n]$ , masked next-token prediction seeks to model the joint conditional:

$P(x_{n+1}, x_{n+2}, \dots, x_{n+k} \mid x_{\leq n}),$

typically realized by augmenting the sequence with $k$ special mask tokens $m_1, \dots, m_k$ to yield $X_m = [x_1,\dots x_n, m_1,\dots, m_k]$ , then training the model to fill in all $k$ mask positions jointly in a single forward pass (Samragh et al., 16 Jul 2025). Efficient training inserts blocks of $k$ masks after every position $i\leq n$ in parallel, enabling the model to learn the conditional for all subsequences.

In continuous or non-language domains—such as generative audio modeling—MNTP generalizes to the regime where, for sequence $x = (x_1, ..., x_n)$ of continuous-valued or discretized tokens, the model randomly drops a subset of past tokens and predicts a randomly chosen future token, enforcing causality by restricting context to only preceding tokens (Yang et al., 14 Jul 2025).

Encoder-style MNTP (as used in anomaly detection in HEP), applies masked token operations at random positions and reconstructs the masked tokens from the visible context, formally optimizing:

$L(\theta; X, M) = -\sum_{i\in M} \log p_\theta(x_i | X_{\setminus M}),$

where $P(x_{n+1}, x_{n+2}, \dots, x_{n+k} \mid x_{\leq n}),$ 0 indexes the randomly chosen masked positions (Visive et al., 30 Sep 2025).

2. Architectures and Attention Schemes

The MNTP paradigm imposes specific attention and masking rules to ensure correct information flow, coherence in joint predictions, and causality or independence as needed for the application. Multiple architectural strategies exist:

Tree-shaped Attention: In multi-token predictive LLMs, position-ids and attention biases enforce that next-token positions attend only to their relevant context (earlier tokens), with each group of $P(x_{n+1}, x_{n+2}, \dots, x_{n+k} \mid x_{\leq n}),$ 1 mask tokens forming a separate prediction chain, preserving autoregressive semantics (Samragh et al., 16 Jul 2025).
Block-diagonal and Non-causal Masks: For multi-output prediction—such as object recognition cast as parallel label generation—the Transformer decoder builds a block-structured mask so that each output chain (e.g., object label) is independent of others but fully attends to the common prefix (e.g., image features and prompt) (Yue et al., 2023).
Causal Masking in Continuous Domains: In generative audio, masking is realized by random dropping of contiguous or arbitrary context tokens, with no insertion of explicit mask tokens; positions to be predicted are indicated by special learned position-embeddings, maintaining causal restriction (Yang et al., 14 Jul 2025).
Sequential Masked Decoding: In time series and video tracking, models such as TAPNext treat unknown track positions as masked tokens, applying causal state-space recurrences and spatial attention within frames (Zholus et al., 8 Apr 2025).

A summary comparison is provided below:

Application	Masking Schema	Attention Pattern
Language modeling (Samragh et al., 16 Jul 2025)	Mask appended to prefix	Prefix causal, mask block diagonal
Audio generation (Yang et al., 14 Jul 2025)	Drop random pasts, 1 future target	Causal, target-position embedding
Object recognition (Yue et al., 2023)	Block-diagonal non-causal mask	Shared prefix, independent label chains
Anomaly detection (Visive et al., 30 Sep 2025)	Random position mask	Fully bidirectional (encoder-style)
Video tracking (Zholus et al., 8 Apr 2025)	Tokenize tracks, mask all but query	Causal SSM + spatial attention

3. Auxiliary Modules and Losses

Standard single-token models produce one logit per position, which is insufficient for joint multi-token outputs due to lack of sequential dependency among the predicted tokens. MNTP frameworks address this limitation via:

Sampler Modules: A lightweight sequential sampler (typically a two-layer MLP) conditions each predicted token on previously sampled outputs, enforcing sequential coherence across jointly predicted tokens. Formally, the sampler concatenates the embedding of the previously sampled token with the latent at the current mask, transforming the concatenated vector to logits via a shared unembedding matrix (Samragh et al., 16 Jul 2025).
Latent Consistency Matching (LCM): An auxiliary squared-norm loss aligns hidden states at mask positions (for multi-token prediction) with those at standard next-token positions (frozen NTP pathway). This encourages the parallel predictions to agree with what the underlying autoregressive model would produce (Samragh et al., 16 Jul 2025).
Diffusion-based Losses: In continuous MNTP for audio, a token-wise diffusion loss is applied, training the model to denoise a corrupted latent target from partial context, which both regularizes and extends prediction range (Yang et al., 14 Jul 2025).