Ordered Multi-Token Prediction (OMTP)

• Ordered Multi-Token Prediction (OMTP) is an autoregressive extension that predicts multiple tokens in a designated order, enabling parallelized, horizon-based generation.
• OMTP employs techniques like head indexing, causal masking, and latent rollout constraints to preserve left-to-right dependencies while forecasting token blocks.
• Empirical studies show OMTP enhances sample efficiency and accelerates decoding in applications such as language modeling, code generation, recommendation, and speech processing.

Ordered Multi-Token Prediction (OMTP) denotes a family of autoregressive modeling techniques that extend next-token prediction from a one-step objective to an explicitly ordered multi-step objective. Instead of supervising only $x_{t+1}$ from a prefix $x_{\le t}$, OMTP organizes supervision over a horizon $x_{t+1:t+K}$ while preserving left-to-right structure through head indexing, causal masking, ordered losses, latent rollout constraints, or verification during decoding. Across recent work, OMTP appears both as a training objective and as an inference acceleration mechanism, with applications spanning general language modeling, code generation, recommendation reranking, speech recognition, speech generation, reasoning acceleration, world-model learning, and prompt-robust classification (Gloeckle et al., 2024, Samragh et al., 16 Jul 2025, Lin et al., 29 Oct 2025, Zuhri et al., 26 Aug 2025).

1. Conceptual definition and scope

The canonical next-token factorization used by autoregressive LLMs is

$$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$

An explicit OMTP formulation replaces one-step emission with ordered block prediction. In GReF, for example, the next $K$ positions are modeled as

$$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$

where the $j$-th head is assigned to the $j$-th future position (Lin et al., 29 Oct 2025). This head-to-offset mapping is the most direct expression of OMTP: the outputs are parallelized, but their semantics remain ordered.

The term is broader than any single architecture. In the large-language-model literature, some papers describe the same pattern as “multi-token prediction” rather than OMTP, yet still implement an ordered objective because head $i$ is trained on token $x_{t+i}$ and decoding accepts tokens in prefix order. The multi-head shared-trunk formulation in large decoder-only LLMs is therefore ordered by construction even when the label OMTP is not used explicitly (Gloeckle et al., 2024).

A further distinction is between exact-token OMTP and order-only supervision. Token Order Prediction (TOP) argues that exact future-token prediction can be an overly difficult auxiliary loss for standard NLP tasks and instead trains a model to rank upcoming vocabulary items by proximity using a listwise learning-to-rank loss. In that setting, OMTP becomes an ordering problem over future tokens rather than a direct prediction of their exact identities (Zuhri et al., 26 Aug 2025).

2. Architectural realizations

Recent OMTP systems differ primarily in how they expose future positions and how they encode ordering.

Architecture pattern	Representative papers	Ordering mechanism
Shared trunk with multiple future heads	(Gloeckle et al., 2024, Lin et al., 29 Oct 2025, Wang et al., 5 Apr 2025)	Head index corresponds to horizon
Masked future slots appended to prefix	(Samragh et al., 16 Jul 2025, Kirchenbauer et al., 5 Feb 2026)	Causal masking, sampler, or verification
Position-shifted special tokens	(Liu et al., 16 Apr 2026)	Offset encoded by position IDs
Bottlenecked teacher-forced future conditioning	(Ahn et al., 24 Mar 2025)	Chain-rule conditioning through Fetch
Per-position semantic-state decoder	(Walker, 2024)	Causal decoder over future horizon
Frozen-backbone replicated final heads	(Mehra et al., 13 Feb 2025)	One replicated head per future position

The masked-input family appends multiple future masks to a common prefix and constrains their visibility. “Your LLM Knows the Future” equips a pretrained decoder-only LLM with gated LoRA, a masked-input formulation, a sampler MLP that conditions on the previously chosen token, and speculative verification. The gated LoRA path activates only on masked positions, so next-token behavior is preserved while masked positions become multi-token predictors (Samragh et al., 16 Jul 2025). “Multi-Token Prediction via Self-Distillation” likewise inserts $x_{\le t}$0 special <[MTP](https://www.emergentmind.com/topics/multi-token-prediction-mtp-4f2bcf24-fd36-4312-a561-ac31459a8c16)> tokens after a prefix and uses blocked causal attention so each masked slot attends to the prefix and earlier slots in the same block, but it keeps the base transformer and unembedding head unchanged apart from the learned <MTP> embedding (Kirchenbauer et al., 5 Feb 2026).

Other systems encode order through positional layout rather than explicit heads. MemoSight inserts foresight tokens $x_{\le t}$1 immediately before reasoning tokens and shifts their position IDs forward so each carrier token predicts a token $x_{\le t}$2 steps ahead; memory tokens then compress past reasoning steps for KV-cache reduction. Here the ordered offsets are represented by position-aware alignment rather than by separate horizon heads (Liu et al., 16 Apr 2026).

A different line of work uses an explicit bottleneck to prevent future-token leakage into the main representation. JTP predicts future tokens with teacher forcing, but future symbols enter only through a lightweight Fetch module attached to the base hidden state. This is designed to make the base hidden state a short-horizon belief state rather than allowing a deeper auxiliary stack to solve the task independently (Ahn et al., 24 Mar 2025). Future Token Prediction (FTP) pushes this further by projecting each encoder hidden state into a pseudo-sequence that a small causal decoder cross-attends to when predicting the next $x_{\le t}$3 tokens, thereby making each token position carry a semantic state predictive of an ordered horizon (Walker, 2024).

Adaptation of frozen pretrained LLMs has also been investigated directly. “On multi-token prediction for efficient LLM inference” replicates the final transformer layer into horizon-specific heads and shares the unembedding matrix, but finds that hidden layers are already strongly specialized for next-token prediction, making OMTP adaptation non-trivial even with weighted hidden-state mixing and LoRA-based joint finetuning (Mehra et al., 13 Feb 2025). In speech generation, VocalNet instead uses a depth-ordered decoder: the base speech decoder emits the first next-token distribution, and $x_{\le t}$4 additional one-layer MTP modules transform that state sequentially so deeper heads predict farther future speech tokens in order (Wang et al., 5 Apr 2025).

3. Objectives and mechanisms for enforcing order

The simplest OMTP objective is a sum of per-horizon cross-entropies. In large text LMs, this appears as the standard multi-token auxiliary loss over independent future heads; in recommendation reranking, it becomes a blockwise ordered factorization over items in a slate (Gloeckle et al., 2024, Lin et al., 29 Oct 2025). GReF makes the ordering explicit by combining multi-token cross-entropy with an ordered preference loss,

$x_{\le t}$5

where $x_{\le t}$6 applies a pairwise logistic loss to preferred and less-preferred permutations of the predicted block (Lin et al., 29 Oct 2025). This is not merely a speed device: it imposes exposure-order consistency for reranking.

Several systems augment cross-entropy with consistency constraints that tie ordered predictions back to autoregressive behavior. In “Your LLM Knows the Future,” Latent Consistency Matching aligns masked-position hidden states with the corresponding next-token hidden states from the frozen base model, so the multi-token branch is explicitly self-distilled toward the autoregressive trajectory (Samragh et al., 16 Jul 2025). LSE-MTP introduces a stronger version of this idea by penalizing the distance between multi-step predictive latents and teacher-forced future hidden states,

$x_{\le t}$7

and also aligns those predictive latents to target token embeddings. This addresses what the paper terms structural hallucinations, in which discrete token supervision encourages illegal latent shortcuts (Zhong et al., 7 Apr 2026).

A different training principle is online self-distillation. “Multi-Token Prediction via Self-Distillation” lets the student propose an ordered rollout $x_{\le t}$8 in one pass, then scores that rollout with a frozen next-token teacher using the chain rule. The student objective is

$x_{\le t}$9

so coherent low-entropy ordered rollouts receive higher weight (Kirchenbauer et al., 5 Feb 2026).

Order can also be weakened from exact-token supervision to ranking supervision. TOP defines a target vector over the vocabulary that ranks tokens by how soon they will next appear and optimizes a ListNet-style listwise objective

$x_{t+1:t+K}$0

This preserves the notion of ordered anticipation while discarding the requirement to predict exact future identities (Zuhri et al., 26 Aug 2025).

4. Decoding, verification, and efficiency

Inference procedures vary sharply across OMTP systems. One family uses propose-and-verify decoding. In “Your LLM Knows the Future,” a masked-input model first generates an ordered chain with a sampler conditioned on the previous chosen token, then verifies the speculative continuation either linearly or quadratically. Acceptance is measured as $x_{t+1:t+K}$1, the number of committed tokens divided by generation steps; on Tulu3-8B with eight masks, the paper reports acceptance-rate speedups up to $x_{t+1:t+K}$2 on GSM8K, $x_{t+1:t+K}$3 on HumanEval, and $x_{t+1:t+K}$4 on AlpacaEval, with knowledge tasks reaching about $x_{t+1:t+K}$5 and safety sets up to about $x_{t+1:t+K}$6 (Samragh et al., 16 Jul 2025).

Speech-LLaMA uses two explicit accept strategies. Verification-based decoding predicts $x_{t+1:t+K}$7 tokens in parallel and accepts the longest prefix whose tokens match the main head’s autoregressive rescoring, optionally under top-$x_{t+1:t+K}$8 rather than strict top-1 agreement. Threshold-based decoding instead accepts the longest prefix whose per-head confidences exceed a threshold $x_{t+1:t+K}$9. On LibriSpeech, OMTP with four heads reduces decoder calls by about $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$0 while maintaining or improving WER, and the latent-head variant lowers decoder RTF relative to separate projection heads (Raj et al., 2024).

Self-distilled standalone OMTP removes external verification entirely at deployment time. The model accepts a confidence-adaptive prefix of its own block prediction: if the first $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$1 per-slot top-1 probabilities exceed a threshold $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$2, those tokens are committed; otherwise decoding falls back to a shorter accepted prefix. On GSM8K, the paper reports more than $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$3 average speedup at less than $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$4 drop in accuracy relative to single-token decoding for one model, while more permissive thresholds push acceleration into the $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$5–$$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$6 range with larger quality trade-offs (Kirchenbauer et al., 5 Feb 2026).

In recommendation reranking, OMTP is used to reduce the number of autoregressive steps rather than to speculate over open-ended text. GReF advances the slate by $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$7 items per step using parallel heads, masked softmax over the in-context candidate set, and deduplication by binary masks. On the Kuaishou dataset, average single-sample latency drops from $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$8 ms for GReF without OMTP to $$p_\theta(y \mid x, X)=\prod_{t=1}^{L} p_\theta(y_t \mid y_{<t}, x, X).$$9 ms with OMTP, close to the $K$0 ms of the non-autoregressive baseline NAR4Rec, while maintaining stronger AUC and NDCG (Lin et al., 29 Oct 2025).

MemoSight combines OMTP with context compression. Its foresight tokens draft ordered future reasoning tokens, while memory tokens replace verbose prior steps in the KV cache. The reported result is a KV-cache footprint reduction of up to $K$1 and an inference acceleration of $K$2, with additional speculative decoding speedups raising average throughput from $K$3 to $K$4 tokens/s on Qwen2.5-7B and from $K$5 to $K$6 tokens/s on Llama-3.1-8B (Liu et al., 16 Apr 2026).

5. Empirical effects on capability and representation

In large text and code models, OMTP is repeatedly associated with gains beyond latency. “Better & Faster LLMs via Multi-token Prediction” reports that 13B parameter models solve $K$7 more problems on HumanEval and $K$8 more on MBPP than comparable next-token models, and attributes part of the advantage to improved sample efficiency, stronger induction-head formation, and better algorithmic reasoning on synthetic tasks (Gloeckle et al., 2024). Those results are echoed by masked-input OMTP systems that preserve next-token behavior while adding ordered future prediction (Samragh et al., 16 Jul 2025).

Some work emphasizes representational rather than purely generative consequences. FTP trains each token position to decode an ordered horizon from a pseudo-sequence derived from a single hidden state, and reports smoother top-layer embedding trajectories, improved topic coherence, stronger downstream classification with frozen embeddings, and better performance on a complex toy coding problem than a GPT baseline trained to similar next-token validation loss (Walker, 2024). JTP makes a related claim in a more controlled synthetic setting: by forcing future-token information through a minimal Fetch bottleneck, it induces a short-horizon belief-state representation that solves star-graph navigation tasks more reliably than marginal multi-token baselines (Ahn et al., 24 Mar 2025).

World-model papers frame OMTP as a structural bias on latent dynamics. LSE-MTP argues that standard multi-token supervision creates gradient coupling that promotes representational contractivity toward future-equivalent states, but also raises the risk of illegal latent shortcuts. On synthetic graphs and Manhattan Taxi Ride, latent semantic enhancement improves representation alignment, reduces Illegal Shortcut Probability, and raises metrics such as Valid Trajectories, State-wise Similarity, and Detour Robustness relative to next-token and standard MTP baselines (Zhong et al., 7 Apr 2026).

Order-only auxiliaries can outperform exact-token OMTP on standard NLP evaluation. TOP reports that, across 340M, 1.8B, and 7B models trained on FineWeb-Edu, the ranking-based auxiliary generally surpasses both NTP and exact MTP on eight standard NLP benchmarks, with especially strong gains on LAMBADA, HellaSwag, ARC Challenge, SciQ, NaturalQuestions Open, and TriviaQA at larger scale (Zuhri et al., 26 Aug 2025). At inference time, P3 reaches a related goal without retraining: by appending placeholders and reading many future-position distributions in one pass, it improves zero-shot classification accuracy and cuts cross-prompt variance by up to $K$9, while even no-prompt performance remains comparable to prompted evaluation (Qian et al., 4 Apr 2025).

6. Limitations, controversies, and open directions

A central unresolved issue is whether OMTP should be viewed as exact future-token prediction, an ordered approximation to the chain rule, or a softer representation-learning objective. The adaptation study of frozen pretrained LLMs shows that standard next-token models already possess some two-step predictive capability via numerical marginalization, but their hidden layers are strongly specialized for NTP, and adding OMTP heads cannot fully overcome that barrier even with LoRA, head warmup, differential learning rates, and weighted hidden states (Mehra et al., 13 Feb 2025). This directly challenges the assumption that OMTP is a trivial post hoc extension of NTP models.

Another controversy concerns task dependence. TOP explicitly argues that exact future-token prediction is too difficult as an auxiliary loss for standard NLP benchmarks, which explains why MTP can help code generation yet regress on some natural-language tasks at larger scales (Zuhri et al., 26 Aug 2025). This criticism is consistent with speech and text results showing diminishing returns as horizon size grows. Speech-LLaMA reports that efficiency improves with more heads but saturates after $$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$0, with WER staying strong up to $$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$1 and then degrading; multilingual settings show substantial speed gains but a small average WER increase, attributed partly to decoder capacity (Raj et al., 2024). VocalNet likewise finds that very large $$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$2 harms alignment, and that sequential ordered modules outperform both parallel heads and teacher-forced deeper modules at aggressive speedups (Wang et al., 5 Apr 2025).

Long-range uncertainty remains a practical failure mode. Masked-input OMTP systems report lower acceptance in high-entropy regions, abrupt topic shifts, and rare-token regimes, as well as sublinear gains when $$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$3 increases; quadratic decoding stabilizes acceptance but raises per-step sequence length by $$p_\theta(y_{t+1:t+K} \mid y_{\le t}, x, X)=\prod_{j=1}^{K} p_\theta^{(j)}(y_{t+j}\mid h_{\le t}(x, X, y_{\le t})),$$4 (Samragh et al., 16 Jul 2025). World-model analysis adds a deeper concern: stronger multi-step pressure can produce teleological bias, in which the model favors distant consistency over locally legal transitions, leading to structural hallucinations unless latent trajectories are explicitly anchored (Zhong et al., 7 Apr 2026).

Inference-time approximations also vary in fidelity. Placeholder-based P3 is highly effective for classification robustness, but it does not condition later positions on realized earlier outputs, so it is not a strict chain-rule OMTP system. This suggests a broader interpretation: OMTP is less a single algorithm than a design space for imposing ordered multi-step structure on autoregressive models, with different points in that space trading off exactness, efficiency, representational pressure, and deployment complexity (Qian et al., 4 Apr 2025).