Multi-Token Prediction (MTP)

• Multi-token prediction is a technique that predicts several future tokens from a shared representation, reducing serial decoding and leveraging richer future dependencies.
• It employs diverse architectures such as offset-specific heads, sequential rollouts, and grouped token outputs to manage structured outputs across domains like speech and document parsing.
• Adaptive decoding strategies and curriculum training optimize the trade-off between throughput and quality, ensuring effective integration of multi-token forecasting in modern models.

Multi-token prediction (MTP) denotes a family of language-modeling and sequence-modeling methods in which a model predicts more than one future token from a shared prefix or hidden state, rather than restricting supervision and decoding to standard next-token prediction (NTP). Across the recent literature, MTP appears in several forms: explicit joint modeling of a future block, multiple offset-specific heads over a shared representation, grouped prediction of dense modalities such as speech, speculative drafting modules for lossless acceleration, and latent compression schemes that change the autoregressive unit itself (Mehra et al., 13 Feb 2025, Samragh et al., 16 Jul 2025, Grivas et al., 14 Nov 2025). The common motivation is to reduce serial decoding cost and/or to force hidden states to encode information useful beyond the immediate next symbol, but the surveyed work shows that these two goals are not identical and often require different architectures, losses, and decoding policies.

1. Formal scope and definitions

The canonical NTP objective predicts only the immediate continuation, typically written as $p(x_{t+1}\mid x_{\le t})$, or equivalently as an autoregressive factorization over one-step conditionals (Mehra et al., 13 Feb 2025). MTP enlarges the prediction target to a horizon of future tokens. In the most general formulation considered in the literature, the target is a conditional distribution over a block,

$$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$

while practical implementations often approximate that object by offset-specific predictions over the same contextual state (Grivas et al., 14 Nov 2025).

The surveyed papers use this idea in several distinct ways. Some define MTP as predicting the next $k$ adjacent tokens from one prefix and summing the per-offset cross-entropies, as in small-language-model pre-training and several LLM retrofitting methods (Aynetdinov et al., 28 May 2025, Samragh et al., 16 Jul 2025). Others redefine the autoregressive unit itself. In speech-LLMs, one hidden state may emit a fixed group of $g$ future speech tokens, and the backbone consumes one fused representation per group rather than per individual speech token (Fan et al., 14 Jun 2025). In structured 3D scene modeling, one decoder pass proposes several consecutive structured tokens and a decoder-side filter decides how many to accept (Yin et al., 5 Dec 2025). Leap-based MTP extends the horizon non-locally by supervising positions such as $t+1,t+3,t+5,\dots$ rather than only adjacent offsets (Liu et al., 23 May 2025).

This diversity matters because “MTP” is not a single algorithm. In some papers it is primarily a training objective, in others an inference-time drafting mechanism, and in others a representational compression device. A direct consequence is that comparisons across papers are only meaningful when the prediction target, horizon semantics, and acceptance policy are specified explicitly.

2. Architectural patterns

A large fraction of the literature begins from a shared backbone state and adds multiple prediction heads. The simplest case is a bank of independent linear heads, one per future offset, as in the “fully factorized” view of future-token prediction and in lightweight MTP retrofits for LLMs and speech decoders (Aynetdinov et al., 28 May 2025, Grivas et al., 14 Nov 2025). This design is computationally attractive, but it assumes conditional independence among future positions given the current context. Several papers identify that assumption as the main expressiveness bottleneck, especially for byte-level generation and acoustically dense speech tokens (Grivas et al., 14 Nov 2025, Fan et al., 14 Jun 2025).

A second family uses sequential or cascaded latent rollouts. VocalNet adds $N-1$ sequential Transformer-based MTP modules above the speech decoder backbone, with the $k$-th module predicting the $(k+1)$-st future speech token from the transformed hidden state of the preceding module rather than from a shared static state (Wang et al., 5 Apr 2025). FastMTP similarly trains a single shared MTP head to operate recursively, so that later draft steps condition on the head’s own previous hidden state and drafted tokens rather than on isolated offset labels (Cai et al., 16 Sep 2025). P-MTP uses the same principle in document parsing, with a lightweight recurrent MTP module that rolls hidden states forward depth by depth (Xiang et al., 23 Jun 2026). These serial designs are motivated by the observation that later future tokens are not independent “extra labels”; they depend on earlier drafted futures.

A third pattern changes the sequence unit by grouping or compressing tokens. In speech-LLMs with decoupled tokenizers, multiple future speech tokens are packed into a group, the backbone runs on one fused representation per group, and the output side emits several tokens in parallel through position-specific heads (Fan et al., 14 Jun 2025). SuperThoughts compresses pairs of chain-of-thought tokens into one latent representation and then decodes two reasoning tokens per step via a lightweight MTP module, preserving token-level supervision while halving the reasoning-step granularity (Xiong et al., 11 Jun 2026). VocalNet-M2 combines multi-codebook acoustic frames with multi-step future prediction, so that one decoding step can emit multiple future acoustic frames, each itself containing eight codebook tokens (Wang et al., 13 Nov 2025).

More recent work moves beyond independent heads entirely and models the future block as a structured joint distribution. “Fast and Expressive Multi-Token Prediction with Probabilistic Circuits” represents

$q_\theta(x_{t+1:t+n}\mid x_{\le t})$

with smooth, decomposable probabilistic circuits, recovering fully factorized models, conditional mixtures, hidden Markov models, and binary-tree tensor-network-like parameterizations as special cases (Grivas et al., 14 Nov 2025). This line is explicitly concerned with the expressiveness–latency trade-off: richer dependence structure can improve acceptance, but only if the resulting drafting cost remains small enough to increase net throughput.

Training-free probing methods form a fifth pattern. “Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing” appends synthetic mask embeddings drawn from the model’s own embedding space and interprets the logits at those positions as future-token proposals, then verifies them in parallel by the base model (Goel et al., 18 Mar 2026). Rather than adding heads, this approach treats latent future prediction as a property already present in the frozen decoder.

3. Objectives, optimization, and curriculum design

The default MTP training objective is a horizon-summed cross-entropy over future offsets. Small-model curriculum work writes this directly as

$$\mathcal{L}_{\text{MTP}}=-\sum_t\sum_{i=1}^k \log P(x_{t+i}\mid x_{\le t};\theta),$$

with one head aligned to each offset (Aynetdinov et al., 28 May 2025). Speech and spoken-language papers use analogous sums over grouped speech tokens or future acoustic frames (Fan et al., 14 Jun 2025, Wang et al., 13 Nov 2025). Several papers introduce explicit depth weighting. VocalNet scales deeper losses by $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$0 to emphasize shallower futures (Wang et al., 5 Apr 2025), while FastMTP uses exponentially decaying coefficients $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$1 in its recursive drafting loss (Cai et al., 16 Sep 2025).

A recurrent theme is that naïvely adding more future offsets makes optimization unstable. “Pre-Training Curriculum for Multi-Token Prediction in LLMs” addresses this by stage-wise curricula over the active horizon. The forward curriculum starts from NTP and gradually increases the number of active heads; the reverse curriculum does the opposite (Aynetdinov et al., 28 May 2025). Forward curriculum preserves self-speculative benefits better, whereas reverse curriculum improves main-head NTP quality more strongly but largely loses speculative-decoding gains.

Several papers tackle the train–test mismatch between teacher-forced future supervision and recursive deployment. “Self-Distillation for Multi-Token Prediction” introduces MTP-D, in which the main head acts as a detached teacher for the auxiliary MTP heads through a Top-$$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$2 KL loss, added to the standard MTP cross-entropy (Zhao et al., 25 Mar 2026). The paper reports a $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$3 improvement in MTP-head acceptance rates for four-head MTP-D, while largely preserving main-head performance, and then extends head count economically through looped copying and continued pretraining (Zhao et al., 25 Mar 2026). FastMTP pursues the same mismatch from another angle by fine-tuning a shared recursive head on self-distilled data so that the head learns the verifier model’s own distribution under recursive use (Cai et al., 16 Sep 2025).

Other work moves MTP supervision earlier in the network or anchors it to latent states rather than only to discrete targets. MTP-S2UT applies future-token loss not to the final decoder layer but to the intermediate decoder layer that already carries CTC text supervision, arguing that “information enrichment” should begin earlier (Wang et al., 11 Oct 2025). LSE-MTP augments standard MTP with latent consistency and semantic anchoring losses so that future-predictive states are aligned to teacher-forced future hidden states and token embeddings, with the stated goal of reducing “structural hallucinations” in world-model learning (Zhong et al., 7 Apr 2026). P-MTP introduces a Progressive Curriculum Loss in which each depth-$$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$4 loss is weighted by the product of a cumulative path-reliability term and a retrospective target-consistency term,

$$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$5

thereby suppressing noisy distal supervision when nearer predictions are still unreliable (Xiang et al., 23 Jun 2026).

4. Decoding, verification, and adaptive control

Inference-time MTP is usually coupled to speculative verification. The simplest deployment pattern is “draft several, verify in parallel, accept the longest valid prefix.” FastMTP makes this explicit with an EAGLE-style self-speculative pipeline, reporting an average $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$6 speedup over standard next-token prediction with lossless output quality, and showing that ordinary vanilla MTP collapses at deeper recursive draft positions unless training is aligned to recursive use (Cai et al., 16 Sep 2025). CLP diagnoses a separate failure mode: if an auxiliary head is allowed to compete with the backbone LM head on the very first next token, repetitive and incoherent outputs can result. Its “Backbone-as-Architect” principle keeps the first token under the backbone LM head and lets auxiliary heads predict only later positions, with a lightweight span-length classifier deciding how many extra tokens to accept (Xie et al., 9 Jun 2026).

A major recent development is adaptive speculation depth. EntMTP argues that fixed draft-tree topologies are mismatched to the nonstationary entropy of generation. It therefore constructs a task-specific bank of Pareto-optimal MTP trees and switches among them online using a path-confidence proxy derived from base-model and Hydra-head probabilities, yielding a reported $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$7–$$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$8 speedup over Hydra and a peak $$q_\theta(x_{t+1},x_{t+2},\dots,x_{t+n}\mid x_{\le t}),$$9 speedup over Medusa baselines (Chen, 25 Jun 2026). P-MTP makes a related claim in document parsing through Confidence-Gated Dynamic Drafting, which extends the speculative horizon up to $k$0 only when confidence remains high (Xiang et al., 23 Jun 2026). SuperThoughts uses a simpler confidence gate based on the max softmax probability of the MTP token and falls back to standard single-token decoding when the MTP module is uncertain (Xiong et al., 11 Jun 2026). Fast SceneScript adapts self-speculative decoding and introduces confidence-guided decoding to filter unreliable structured-layout tokens, because naïve multi-token acceptance degrades accuracy even in a highly regular output space (Yin et al., 5 Dec 2025).

Not all MTP systems use strict verification. VocalNet directly commits several predicted future speech tokens per step and therefore treats MTP as a direct decoder rather than a draft-and-verify mechanism (Wang et al., 5 Apr 2025). By contrast, FlashTTS explicitly treats MTP branches as speculative predictors whose outputs are validated by the frozen backbone distributions, reporting that MTP-3 is the best operating point, whereas MTP-5 adds only marginal speed and degrades quality (Xie et al., 8 Jun 2026).

5. Domain-specific realizations

The clearest empirical lesson of the recent literature is that MTP behaves very differently across domains. It is strongest where future continuations are dense, low-entropy, repetitive, or highly structured, and weaker when long-range continuation uncertainty is dominant. The table summarizes representative instances.

Domain	MTP role	Representative result
LLM inference	Self-speculative drafting	FastMTP reaches an average $k$1 speedup with lossless output quality (Cai et al., 16 Sep 2025)
Speech-language modeling	Grouped speech-token prediction for alignment and speed	Decoupled-tokenizer MTP gives up to $k$2 faster decoding and reduces WER from 6.07 to 3.01 (Fan et al., 14 Jun 2025)
3D scene layout estimation	Structured-token drafting with reliability filtering	Fast SceneScript generates up to 9 tokens per decoder inference step while adding only $k$3 additional parameters (Yin et al., 5 Dec 2025)
Chain-of-thought reasoning	Pair compression plus conditional second-token decoding	SuperThoughts reports $k$4--$k$5 CoT length reduction with minimal degradation (Xiong et al., 11 Jun 2026)
Streaming TTS	Parallel future speech-token prediction	FlashTTS reduces First-Packet Latency to 325ms, with MTP-3 as the best speed-quality compromise (Xie et al., 8 Jun 2026)
Document parsing	Deep look-ahead speculative drafting	P-MTP reports up to a $k$6 speedup with negligible loss in accuracy (Xiang et al., 23 Jun 2026)

Speech is the most heavily developed application area. In LLM-centric speech-LLMs, MTP is framed not only as acceleration but as information-density matching between sparse text tokens and dense speech tokens; the decoupled FACodec setting is the most favorable, with cross-modal similarity increasing sharply as compression grows (Fan et al., 14 Jun 2025). Direct speech-to-speech translation uses MTP differently: MTP-S2UT treats it as an auxiliary loss that densifies intermediate decoder states, improving ASR-BLEU across French$k$7English and Spanish$k$8English settings and performing best when applied at the CTC-supervised intermediate layer (Wang et al., 11 Oct 2025). VocalNet reports that its staged, non-teacher-forced MTP design improves both speed and speech quality, with the best balance at $k$9 decoding speedup (Wang et al., 5 Apr 2025). VocalNet-M2 combines multi-codebook acoustic tokenization with future-frame prediction and reports first-chunk latency reduced from approximately 725ms to 350ms while maintaining competitive performance (Wang et al., 13 Nov 2025).

Structured non-linguistic outputs also provide favorable MTP conditions. Fast SceneScript argues that scene-layout “language” is more deterministic than open-ended natural language and therefore especially amenable to multi-token drafting, provided unreliable structured tokens are filtered before commitment (Yin et al., 5 Dec 2025). P-MTP makes the analogous claim for document serialization, where markup-heavy outputs contain long deterministic spans and token-dense structure (Xiang et al., 23 Jun 2026).

6. Limitations, misconceptions, and open problems

One common misconception is that MTP is simply a drop-in inference accelerator. Several papers argue the opposite. “On multi-token prediction for efficient LLM inference” shows that NTP-pretrained LLMs do possess latent multi-token predictive distributions via marginalization, but that explicit MTP heads attached after pretraining are hard to train because hidden states are already specialized for next-token prediction (Mehra et al., 13 Feb 2025). This means latent MTP capability and amortized, deployable MTP are distinct problems.

Another misconception is that more heads or deeper horizons are always better. Across domains, aggressive compression or drafting depth often degrades quality. In decoupled speech-language modeling, higher compression improves WER and alignment but causes a slight UTMOS decline at $g$0 compression (Fan et al., 14 Jun 2025). In FlashTTS, MTP-5 improves throughput only marginally beyond MTP-3 while worsening WER, SIM, and CMOS (Xie et al., 8 Jun 2026). In SuperThoughts, uniform two-token reasoning without adaptive fallback almost halves CoT length but induces large accuracy drops, especially at smaller model scales (Xiong et al., 11 Jun 2026). CLP reaches the opposite conclusion for frozen-backbone LLM retrofits: on larger models, shorter horizons such as $g$1 can be preferable because head accuracy is the binding constraint (Xie et al., 9 Jun 2026).

A third debate concerns what MTP should optimize. Some work treats it mainly as richer supervision for representation learning; some treats it as a lossless acceleration mechanism; some tries to do both. The tension appears clearly in curriculum learning for small models: reverse curriculum produces stronger NTP performance and output quality, but “fails to provide any self-speculative decoding benefits,” whereas forward curriculum retains self-speculative gains better (Aynetdinov et al., 28 May 2025). A related trade-off appears in world-model work. LSE-MTP argues that standard MTP induces useful representational contractivity but can also create “structural hallucinations,” because discrete token supervision permits illegal latent shortcuts toward a correct future token (Zhong et al., 7 Apr 2026).

The emerging consensus is therefore narrower than the slogan “predict more tokens at once.” The literature repeatedly points to four conditions for useful MTP: sufficiently predictable local futures, an architecture that models dependencies across drafted positions rather than only offset-wise marginals, a training recipe aligned to the intended inference mode, and an acceptance policy that adapts depth or span length to confidence (Cai et al., 16 Sep 2025, Chen, 25 Jun 2026, Xiang et al., 23 Jun 2026). A plausible synthesis is that future progress will depend less on adding ever more heads than on better head accuracy, richer but still cheap joint parameterizations, and adaptive control policies that match speculation depth to local uncertainty.