FastMTP: Enhanced Multi-Token Prediction

• FastMTP is a multi-token prediction framework that leverages recursive speculative drafting and self-distillation to accelerate large language model inference.
• It employs position-shared weights and language-aware dynamic vocabulary compression to achieve up to 2.03x speedup while maintaining lossless output quality in a 7B setting.
• FastMTP addresses autoregressive decoding bottlenecks by integrating a recursively used MTP head, resulting in significant throughput gains across diverse tasks.

FastMTP is an enhanced multi-token prediction framework for accelerating LLM inference by turning a lightweight, recursively used MTP head into an effective speculative drafter. It fine-tunes a single MTP head with position-shared weights on self-distilled data, aligns MTP training with the actual recursive inference pattern, and adds language-aware dynamic vocabulary compression to reduce drafting cost (Cai et al., 16 Sep 2025). In the reported 7B setting, FastMTP achieves an average of 2.03x speedup compared to standard next token prediction with lossless output quality, and outperforms vanilla MTP by 82% (Cai et al., 16 Sep 2025).

1. Problem formulation and motivation

FastMTP addresses the central throughput bottleneck of autoregressive decoding: one full forward pass per emitted token. For a token sequence $t_{1:i}=(t_1,\dots,t_i)$, the base model follows the standard factorization

$p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$

so generation latency scales linearly with output length. Multi-token prediction seeks to amortize this cost by learning to predict several future tokens rather than only the next one, but prior MTP systems were primarily optimized for training efficiency or final model quality rather than for sustained speculative acceptance during inference (Cai et al., 16 Sep 2025).

A recurring misconception in this literature is that an existing MTP head can simply be reused as a high-quality speculative drafter. FastMTP explicitly rejects that assumption. The reported vanilla MiMo-7B MTP checkpoint, when naively reused as a recursive drafter, shows approximately 70% acceptance at draft step 1, approximately 11% at step 2, and approximately 2% at step 3. In other words, draft quality collapses after the first speculative token. The paper attributes this to single-step training, multiple independent heads, distribution misalignment between training data and verifier behavior, and the mismatch between teacher-forced supervision and recursively drafted inference (Cai et al., 16 Sep 2025).

2. Architectural design and training objective

FastMTP is instantiated on MiMo-7B-RL, a 36-layer decoder-only transformer with a single-layer MTP module built in. The main transformer backbone, embeddings, and LM head are frozen, while only the MTP head $\mathcal{M}$ is fine-tuned. This head contains 210.8M parameters, which is less than 3% of the 7.83B total parameters. The key architectural choice is “position-shared weights”: one MTP head module is reused at every drafting step $k=1,\dots,K$, rather than allocating separate parameters to different look-ahead depths (Cai et al., 16 Sep 2025).

This shared head is used recursively. Let $\mathcal{F}$ denote the frozen backbone and $h_{1:i}=\mathcal{F}(t_{1:i})$ its hidden states. At the first recursive step, the head consumes the main-model hidden state $h_i$ together with the embedding of the shifted token $t_{i+1}$ and predicts $\hat{t}_{i+2}^1$. At later recursive steps, it consumes its own previous hidden state and the embedding of the next shifted token, thereby modeling dependencies among consecutive future tokens rather than treating each look-ahead position independently. The corresponding training loss is a multi-step cross-entropy objective,

$$\mathcal{L}_{\text{mtp}}=\sum_{k=1}^{K}\alpha_k\cdot \mathcal{L}_{\text{mtp}}^k =\sum_{k=1}^{K}\alpha_k\cdot \operatorname{CE}\big(\hat{t}^{k}_{1+k:T+1},\, t_{1+k:T+1}\big),$$

with exponentially decaying step weights

$p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$0

and $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$1 in the reported experiments. This weighting keeps near-future predictions dominant while still training deeper recursive behavior (Cai et al., 16 Sep 2025).

3. Self-distilled alignment and vocabulary compression

A defining feature of FastMTP is its use of self-distilled data. Instead of training the MTP head on external target responses, the main model first generates the responses that the drafter is expected to imitate. Prompts are collected from diverse instruction-tuning data covering general tasks, math and reasoning, code, and Chinese. MiMo-7B-RL then generates responses using temperature 0.6, top-k 20, top-p 0.95, and maximum length 4096. The resulting self-distilled dataset is cleaned with global MinHash deduplication and filters for low-quality or truncated reasoning, excessive repetition, and anomalous length, yielding approximately 389.4K samples distributed as approximately 42% general tasks, 18% math/reasoning, 13% code, and 27% Chinese. Training updates only the MTP head for 3 epochs with AdamW, peak learning rate $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$2, $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$3, $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$4, and global batch size 64, and is reported to finish in less than one day on a single NVIDIA H20 server (Cai et al., 16 Sep 2025).

FastMTP further reduces drafting overhead through language-aware dynamic vocabulary compression. For each language $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$5, it defines a high-frequency subset $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$6 and restricts the drafter’s softmax to that subset, while the verifier still operates on the full vocabulary. This preserves theoretical exactness, because any rare token omitted by the drafter can still be produced by rejection and full-vocabulary verification. The reported optimal compressed sizes differ by language: for MT-Bench, 32k tokens is optimal, whereas for C-Eval the English-centric FR-Spec statistics are replaced by Chinese token frequencies and 16k tokens is optimal. The result is an additional approximately 10–15% throughput gain on top of self-distilled FastMTP, with only a small reduction in average acceptance length (Cai et al., 16 Sep 2025).

4. Speculative decoding procedure and exactness guarantees

FastMTP uses EAGLE-style speculative decoding in a self-drafting configuration. The main model $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$7 acts as verifier, and the fine-tuned MTP head $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$8 acts as drafter. Given the current context $p(t_{1:T})=\prod_{i=1}^{T} p(t_i\mid t_{<i}),$9, the verifier first performs a standard forward pass, produces $\mathcal{M}$0, and emits the next token $\mathcal{M}$1; this token is always accepted. The MTP head then recursively drafts $\mathcal{M}$2 by repeatedly consuming its previous hidden state and the previously drafted token embedding. After that, the verifier reevaluates the draft positions in parallel and accepts the drafted tokens sequentially until the first mismatch (Cai et al., 16 Sep 2025).

The acceptance rule is the standard strict speculative decoding rule, so the output distribution remains provably identical to vanilla autoregressive sampling. FastMTP therefore claims lossless output quality not because the drafter is perfect, but because the verifier remains exact and full-vocabulary. This distinction is central: the drafter only changes efficiency, while correctness is inherited from the unchanged main model. The practical objective is to maximize the average accepted length $\mathcal{M}$3 while keeping the drafter cheap enough that the extra recursive passes do not erase the verifier-side gains (Cai et al., 16 Sep 2025).

5. Empirical performance

FastMTP is evaluated on seven tasks adapted from Spec-Bench: MT-Bench, LiveCodeBench-v6, MATH-500, Natural Questions in RAG mode, Natural Questions in direct QA mode, CNN/Daily Mail summarization, and C-Eval. All reported numbers use MiMo-7B-RL, a single NVIDIA A10 24GB GPU, single-batch greedy decoding, and maximum generation length 1024. The global averages show that naive reuse of vanilla MTP yields only modest speedup, whereas inference-pattern alignment and self-distillation substantially improve speculative utility (Cai et al., 16 Sep 2025).

Variant	Avg. $\mathcal{M}$4	Throughput / Speedup
Baseline NTP ($\mathcal{M}$5)	—	31.55 tok/s / 1.00x
Vanilla MTP reused as drafter ($\mathcal{M}$6)	1.83	38.04 tok/s / 1.21x
FastMTP, self-distilled ($\mathcal{M}$7)	2.73	57.01 tok/s / 1.81x
FastMTP + language-aware FR ($\mathcal{M}$8)	2.66	64.12 tok/s / 2.03x

Stepwise acceptance rates illustrate why the speedup changes so sharply. Vanilla MTP reports approximately 70%, approximately 10%, and approximately 0–3% acceptance at draft steps 1, 2, and 3, respectively. FastMTP raises these values to approximately 81%, approximately 56%, and approximately 36%. Task-wise, the best reported speedup with FastMTP plus language-aware FR is 2.31x on math, followed by 2.11x on code, approximately 2.06x on RAG, approximately 2.03x on MT-Bench, approximately 1.99x on Chinese knowledge, approximately 1.88x on summarization, and approximately 1.85x on QA. A separate study with a head trained up to $\mathcal{M}$9 shows that acceptance length continues to grow, but throughput peaks at $k=1,\dots,K$0 on an A100 at approximately 140 tok/s and then declines, indicating diminishing returns once drafting overhead dominates (Cai et al., 16 Sep 2025).

6. Broader MTP acceleration landscape and limitations

FastMTP belongs to a broader shift in which MTP is treated not merely as an auxiliary training objective but as an explicit inference-time draft mechanism. MiMo-V2-Flash repurposes MTP as an internal draft model for self-speculative decoding in a 309B-parameter MoE with 15B active parameters, reporting up to 3.6 acceptance length and 2.6x decoding speedup with three MTP layers (Xiao et al., 6 Jan 2026). P-MTP extends deep look-ahead MTP in document parsing through Progressive Curriculum Loss and Confidence-Gated Dynamic Drafting, reporting up to a $k=1,\dots,K$1 speedup with negligible loss in accuracy and presenting $k=1,\dots,K$2 as a practical sweet spot for token-dense parsing workloads (Xiang et al., 23 Jun 2026). MTPC, by contrast, studies the expressiveness–latency trade-off of MTP through probabilistic circuits and reports up to approximately 5.47x speedup over autoregressive decoding and approximately 1.22x over the best independence-based MTP baseline on a byte-level LLM (Grivas et al., 14 Nov 2025). These works suggest that FastMTP is part of a larger design space centered on draft quality, verifier alignment, and low-overhead look-ahead structure.

The limitations reported for FastMTP are comparatively narrow but operationally important. Speedup is task-dependent: highly structured outputs such as math and code benefit most, while QA and summarization show smaller gains. Draft lengths beyond $k=1,\dots,K$3 are not practically optimal in the reported 7B setting, even though average accepted length continues to increase, because extra drafting cost eventually outweighs verifier amortization. Language-aware vocabulary compression also depends on correctly identifying which compressed vocabulary to apply; misclassification can lower acceptance, although it does not affect correctness because verification remains full-vocabulary and exact. Finally, the reported results are specific to MiMo-7B-RL, so exact performance figures should not be transferred mechanically to other architectures without re-evaluating draft depth, compression ratio, and distributional alignment (Cai et al., 16 Sep 2025).