Multi-token Prediction in LLMs

Updated 3 July 2026

Multi-token prediction is a method that generalizes next-token prediction by forecasting multiple future tokens simultaneously using parallel prediction heads.
Architectural innovations like recursive head sharing and masked token strategies boost throughput and improve the coupling of long-term dependencies.
Empirical studies demonstrate that multi-token prediction can accelerate inference by up to 3× while maintaining or improving accuracy across varied tasks.

Multi-token prediction (MTP) refers to a family of methods that endow autoregressive sequence models—particularly LLMs—with the ability to predict multiple future tokens in parallel, rather than a single next token. This paradigm shift addresses fundamental bottlenecks in both data efficiency and inference throughput, enables richer modeling of structured data, and impacts representational learning, generalization, and architectural specialization. MTP is now a central component in fast generation pipelines, planning models, and auxiliary training regimes for contemporary LMs.

1. Conceptual Foundations and Taxonomy

The classical next-token prediction (NTP) objective in causal transformers (e.g., GPT) minimizes the negative log-likelihood of each token conditioned on its prefix, i.e.,

$\mathcal{L}_{\mathrm{NTP}} = -\sum_{t=1}^{T} \log P_\theta(x_{t+1} \mid x_{1:t})$

Multi-token prediction generalizes this by asking the model to forecast the next $n$ tokens at each position, using $n$ output distributions (often as parallel "heads")

$\mathcal{L}_{\mathrm{MTP}} = -\sum_{t=1}^{T-n} \sum_{k=1}^{n} \log P_\theta(x_{t+k}\mid x_{1:t})$

This fundamental mechanism is realized in various architectural and training regimes:

Parallel head architectures: $n$ independent heads atop a shared trunk each predict a separate $k$ -step-ahead token (Gloeckle et al., 2024, Zhang et al., 20 Jul 2025).
Cascaded/autoregressive MTP: Later heads condition on earlier draft tokens for richer dependencies (Zhao et al., 25 Mar 2026, Cai et al., 16 Sep 2025, Radhakrishnan et al., 27 Feb 2026).
Masked or register-based variants: Interleaved tokens or mask embeddings provide look-ahead supervision with zero inference cost (Gerontopoulos et al., 15 May 2025, Kirchenbauer et al., 5 Feb 2026, Goel et al., 18 Mar 2026).
Joint prediction via auxiliary modules: Lightweight bottlenecks, as in JTP (Ahn et al., 24 Mar 2025) or FTP (Walker, 2024).
Non-independence modeling: Circuit-based and tensor-decomposition MTP heads capture joint token dependencies (Basharin et al., 2024, Grivas et al., 14 Nov 2025).

Generations during inference may utilize blockwise or tree-based speculative decoding, accepting maximal token runs that pass parallel verification by the main model (Chen, 25 Jun 2026, Cai et al., 16 Sep 2025, Yin et al., 5 Dec 2025).

2. Architectural Realizations and Theoretical Properties

2.1 Head Structures and Training

MTP is typically implemented by augmenting a causal transformer’s output layer with $n$ parallel prediction heads, each trained via cross-entropy against its assigned $k$ -step target. Heads may be shallow affine maps, linear layers, or single-layer transformers (Gloeckle et al., 2024, Zhang et al., 20 Jul 2025). For richer joint modeling, tensor decomposition and probabilistic circuit parameterizations have been proposed (Basharin et al., 2024, Grivas et al., 14 Nov 2025).

Several advanced schemes address scalability:

Register tokens: MuToR (Gerontopoulos et al., 15 May 2025) introduces trainable tokens into input sequences during training, each predicting a future token at variable offsets, with completely standard inference pipelines.
Shared-weight recursive heads: FastMTP (Cai et al., 16 Sep 2025) uses a position-shared module $\mathcal{M}$ recursively to model dependency across consecutively forecasted tokens, dramatically raising multi-step acceptance rates.
Self-distillation: Student heads are aligned to the model’s own chain-rule distribution via Kullback-Leibler penalties or sampling-based knowledge distillation, improving head/main output consistency and overall acceptance rates (Zhao et al., 25 Mar 2026, Kirchenbauer et al., 5 Feb 2026).

2.2 Representation and Information Flow

The theoretical implications of MTP and its variants are substantial:

Gradient coupling and belief compression: MTP induces contraction among "future-equivalent" states, leading to hidden vectors that encode multi-step outcome plans (Zhong et al., 7 Apr 2026).
Planning and long-term structure: Joint prediction with a fixed bottleneck (JTP) forces hidden states to encode enough information for accurate multi-step planning; this contrasts with marginal MTP, which only encourages correct marginals for each token but not coherent joint plans (Ahn et al., 24 Mar 2025).
Latent hallucinations: Without further constraints, contractive pressure from MTP may create "shortcuts" in latent space, merging distinct history paths illegitimately—addressed by auxiliary latent-consistency regularization (Zhong et al., 7 Apr 2026).
Emergence of algorithmic and in-context reasoning: Ablations reveal earlier and more robust induction-head formation, improved arithmetic program generalization, and higher pass rates on algorithmic tasks at smaller parameter counts under MTP (Gloeckle et al., 2024).

3. Multi-Token Prediction for Accelerated Inference

3.1 Blockwise and Speculative Decoding

The principal application of MTP at inference is speculative or blockwise decoding: drafting $k$ tokens in one pass, then verifying with a strong, typically identical verifier model to guarantee output fidelity (Gloeckle et al., 2024, Cai et al., 16 Sep 2025, Kirchenbauer et al., 5 Feb 2026). The acceptance rate $n$ 0—the mean number of verified tokens per forward pass—directly dictates speedup. Notable empirical findings include:

4-token MTP delivers up to 3 $n$ 1 throughput increase on modern LLMs, with byte-level models at $n$ 2 achieving up to 6.4 $n$ 3 acceleration (Gloeckle et al., 2024).
FastMTP, via recursive head-sharing and dynamic vocabulary pruning, raises average $n$ 4 at $n$ 5 from 1.83 (vanilla MTP) to 2.62—about 2 $n$ 6 speedup at scale, losslessly (Cai et al., 16 Sep 2025).
Training-free variants, e.g., embedding-probe mask strategies, yield 8–19% throughput gains over baseline draft-free approaches (Goel et al., 18 Mar 2026).

3.2 Structural and Adaptive Innovations

Recent research has pushed beyond static, homogeneous speculative trees:

Entropy-guided depth: EntMTP (Chen, 25 Jun 2026) selects the depth of speculative drafts dynamically using the local generation entropy, maximizing expected accepted-token throughput without compromising coverage in uncertain (high-entropy) regions.
Quadratic/blockwise speculative expansion: Scheduling techniques, such as "quadratic" mask interleaving, ensure robust acceptance of $n$ 7 fresh tokens per step (Samragh et al., 16 Jul 2025).
Leap-MTP: By predicting non-adjacent, strided tokens ("leap" heads), L-MTP increases the lookahead horizon and efficiently amortizes long-range dependencies (Liu et al., 23 May 2025).

4. Empirical Results, Benchmarks, and Task Relevance

Empirical studies of MTP span code synthesis, natural language, algorithmic reasoning, visual planning, and multimodal learning:

Code and math (HumanEval, MBPP, GSM8K, MATH-500): MTP-trained 13B models solve 12% more HumanEval and 17% more MBPP tasks than NTP baselines (Gloeckle et al., 2024). Leap-MTP and FastMTP both report further accuracy and throughput improvements (Liu et al., 23 May 2025, Cai et al., 16 Sep 2025).
Summarization and open QA: Byte-level and subword MTP improve ROUGE-L by up to 1.0 and demonstrate strong few-shot accuracy on 8benchmarks (Gloeckle et al., 2024, Aynetdinov et al., 28 May 2025).
Vision and structured prediction: MuToR registers yield faster convergence for 2D image generation (ImageNet, FID 6.57 after half the steps of NTP) (Gerontopoulos et al., 15 May 2025). Fast SceneScript achieves 5 $n$ 8 scene-decode speedup with negligible F1 loss for 3D layout tasks (Yin et al., 5 Dec 2025).
Multimodal planning: Visual planning models with MTP heads outperform next-token methods by 3–7% on COIN and CrossTask (Zhang et al., 20 Jul 2025).

The table below summarizes key empirical trade-offs for representative MTP frameworks:

Method	Acceptance $n$ 9	Speedup	Task-specific Gains
Vanilla MTP ( $n$ 0)	1.83	1.2–1.6×	+5–17% (code/math)
FastMTP ( $n$ 1)	2.62	2.0–2.3×	No accuracy drop
JTP (synthetic)	N/A	N/A	100% path-finding acc.
Self-distillation MTP	$n$ 2– $n$ 3	3–5×	<5% rel. accuracy loss
Leap MTP	2 $n$ 4+ over MTP	20–30% over MTP	Avg. +1–3% reasoning tasks

5. Advanced Loss Functions, Curriculum Schemes, and Auxiliary Objectives

Numerous extensions have refined the optimization of MTP:

Curriculum schedules: Training small models with an NTP $n$ 5MTP progressive schedule closes the downstream performance gap between pure MTP and NTP, while maximizing blockwise speedups (Aynetdinov et al., 28 May 2025).
Auxiliary regularization: KL-based self-distillation optimizes main head/MTP head agreement, increasing draft acceptance by 3–8 percentage points with minimal extra cost (Zhao et al., 25 Mar 2026).
Latent/semantic anchoring: Losses penalize deviation of multi-step prediction states from teacher-forced states or target embeddings, reducing structural hallucinations (Zhong et al., 7 Apr 2026).
RL joint training: In RLVR settings, naive joint optimization of MTP and policy losses can degrade performance due to deleterious gradient interactions. An optimal per-batch coefficient (OCC) tracks alignment and adjusts weighting, yielding superior sample efficiency and accuracy in mathematical reasoning benchmarks (Wang et al., 27 May 2026).

6. Practical Considerations, Limitations, and Future Prospects

While MTP’s throughput and data-efficiency benefits are now robust across domains and model scales, the following considerations remain active areas of research:

Independence vs. expressiveness trade-off: Rank-1 and factorized MTP head architectures (efficient) cannot capture full joint future-token dependencies; richer parameterizations (tensor decomposition, probabilistic circuits (Grivas et al., 14 Nov 2025, Basharin et al., 2024)) improve expressiveness at additional computational cost.
Head and parameter overhead: Head sharing, LoRA/adapter specialization, and mask register techniques mitigate quadratic parameter growth (Yin et al., 5 Dec 2025, Gerontopoulos et al., 15 May 2025).
Compatibility and fine-tuning: Classical, head-based MTP models often face degraded transfer to downstream tasks unless extra heads are compatible with pretrained weights. Register- or masking-based approaches (e.g., MuToR, self-distilled MTP) preserve off-the-shelf compatibility (Gerontopoulos et al., 15 May 2025, Kirchenbauer et al., 5 Feb 2026).
Mitigating hallucinations and planning shortcuts: Without auxiliary trajectory or latent consistency losses, standard MTP can compress away crucial path distinctions in planning settings (Zhong et al., 7 Apr 2026).
Inference adaptivity: Scheduling speculative depth based on local entropy or learned policies enhances throughput in non-stationary or domain-heterogeneous settings (Chen, 25 Jun 2026).
Scaling and horizon: Practical block sizes typically peak near 4–8 tokens; gains flatten or reverse at greater horizon length due to increasing prediction error.

Emerging directions include integration with retrieval-augmented and Mixture-of-Experts architectures, finer adaptive control via uncertainty and bandwidth signals, and exploration of richer joint output spaces.

7. Broader Impact and Theoretical Significance

Multi-token prediction fundamentally modifies the optimization landscape and emergent properties of transformer LLMs:

It accelerates autoregressive language modeling by amortizing computation, supporting practical deployment of LLMs in high-throughput settings (Cai et al., 16 Sep 2025, Chen, 25 Jun 2026).
It acts as a representational prior for belief-state encoding, in support of planning and algorithmic reasoning (Zhong et al., 7 Apr 2026, Ahn et al., 24 Mar 2025).
It reveals nontrivial emergent capabilities in models conventionally trained for single-step prediction, as evidenced by the impressive latent performance of unmodified LLMs under mask-probing (Goel et al., 18 Mar 2026).

In summary, multi-token prediction now underpins both the training and deployment of state-of-the-art generative models, with architectural, optimization, and theoretical innovations that continue to shape the landscape of large-scale sequence modeling.