---
title: Next-Token-Pair Prediction (NTPP)
url: https://www.emergentmind.com/topics/next-token-pair-prediction-ntpp-7f3d191b-fd1c-40c2-a36c-75662fcbd5c3
type: topic
---

# Next-Token-Pair Prediction (NTPP)

Next-Token-Pair Prediction (NTPP) is a generative modeling paradigm specialized for predicting two tokens simultaneously from a given context, rather than the standard sequential single-token prediction of traditional autoregressive language models. NTPP is leveraged for both text-based large language models (LLMs) and for dual-channel speech modeling, providing a pathway toward accelerated inference, improved parallelism, and richer, speaker-independent conversational dynamics. NTPP's mathematical, architectural, and training underpinnings are precise, enabling rigorous evaluation and benchmarking in both domains [2507.11851][2502.09419][2506.00975].

## 1. Mathematical Foundation and Objective Formulation

NTPP formalizes the prediction of token pairs $(x_{t+1}, x_{t+2})$ given a context $\mathcal{X}_{\leq t}$ via the joint probability:
\[
p(x_{t+1},x_{t+2} \mid \mathcal{X}_{\leq t};\theta) = p(x_{t+1} \mid \mathcal{X}_{\leq t};\theta)\;p(x_{t+2} \mid \mathcal{X}_{\leq t},x_{t+1};\theta)
\]
This can be generalized to $k$ tokens by recursively marginalizing over all intermediate states as in
\[
p(\mathcal{X}_{t:t+K} \mid \mathcal{X}_{\leq t}; \theta)
= p(x_{t+1}\mid \mathcal{X}_{\leq t})
\prod_{k=2}^{K}
\sum_{s_{1:k-1}\in \mathbb{V}^{k-1}}
p\bigl(x_{t+k},s_{1:k-1}\mid \mathcal{X}_{\leq t}\bigr)
\]
[2502.09419].

For dual-channel speech (SLMs), NTPP is formally defined over paired token streams $(s_t^{(A)}, s_t^{(B)})$, factorizing as
\[
p(S^{(A)}, S^{(B)}) = \prod_{t=1}^T p(s_t^{(A)}, s_t^{(B)} \mid s_{<t}^{(A)}, s_{<t}^{(B)}; \theta)
\]
with conditional independence at each time step:
\[
p(s_t^{(A)}, s_t^{(B)} \mid \text{context})
=
p(s_t^{(A)} | \text{context}) \times p(s_t^{(B)} | \text{context})
\]
and the training objective
\[
L_{NTPP}(\theta) = -\sum_{t=1}^T \left[ \log p(s_t^{(A)} | \cdots) + \log p(s_t^{(B)} | \cdots) \right]
\]
[2506.00975].

## 2. Architectural Strategies for Text-Based LLMs

In text-based LLMs, NTPP is instantiated via a masked-input formulation combined with specialized prediction heads and gated LoRA adaptation [2507.11851]. The procedure includes:

- Masked Input: Augment the context $X = [x_1, ..., x_n]$ by appending $k=2$ mask tokens $m_1, m_2$, producing $X_m = [x_1, ..., x_n, m_1, m_2]$.
- Frozen-Base Transformer: The base model's parameters are frozen during fine-tuning.
- Gated LoRA Modification: Low-rank adapters (A,B) are inserted in parallel to each linear layer, gated such that adaptation occurs only for mask positions. The output is
  \[
  y_t = W x_t + I(t) (AB x_t)
  \]
  where $I(t)$ is $1$ if t is a mask, and $0$ otherwise.
- Token Prediction Heads: Split into the base unembedding head $W$ (classic next-token prediction) and a sampler MLP head for the mask positions, which conditions on both $z_t$ and the embedding of the previously sampled token $E_{y_{t-1}}$.

Per-token outputs and losses are:
\[
p_t^b = \text{softmax}(W z_t), \quad p_t^s = \text{softmax}(W \,\text{MLP}(z_t, E_{y_{t-1}}))
\]
\[
\mathcal{L}_t^b = -\log p_t^b(y_t), \quad \mathcal{L}_t^s = -\log p_t^s(y_t)
\]
with auxiliary latent consistency matching (LCM) to align representations.

## 3. Marginalization and MTP Heads in Pretrained Models

Marginalization computes the exact joint distribution by summing over all intermediate token candidates:
\[
p(x_{t+2} \mid c) = \sum_{y \in \mathbb{V}_\text{top}} p(x_{t+2} \mid c, y) \; p(y \mid c)
\]
and is the baseline for multi-token prediction quality [2502.09419]. For practical amortization, models append “MTP heads”—parallel transformer layers dedicated to future tokens—on top of the frozen backbone. Each head predicts $p(x_{t+n} \mid \mathcal{X}_{\leq t})$ via
\[
p(x_{t+n}\mid \mathcal{X}_{\leq t}) = \text{softmax}(f_u(f^L_n(z^{L-1}_{1:t})))
\]
where $f^L_n$ is the $n$-th replicated final layer and $f_u$ the shared unembedding.

Empirical findings indicate that while such MTP heads increase throughput, performance lags behind marginalization, especially when the backbone is strongly specialized for NTP. Joint training (with LoRA and weighted hidden states) narrows the gap but does not close it [2502.09419].

## 4. Training Workflow and Loss Schemes

Fine-tuning protocols for text-based NTPP introduce several coordinated loss terms [2507.11851]:

- Base loss $\mathcal{L}^b_t$ on all tokens (unembedding head).
- Sampler loss $\mathcal{L}^s_t$ on mask tokens (sampler MLP).
- Latent Consistency Matching (LCM) loss $\mathcal{L}_t^{\mathrm{lcm}}$ aligns masked representations with their autoregressive equivalents.

Overall objective for $k=2$:
\[
\mathcal{L}
=
\frac{1}{|T_\mathsf{orig} \cup T_\mathsf{mask}|}
\sum_{t \in T_\mathsf{orig} \cup T_\mathsf{mask}}
(\mathcal{L}^b_t + \mathcal{L}^s_t)
+
\frac{1}{|T_\mathsf{orig}|}\sum_{t \in T_\mathsf{orig}}\!\mathcal{L}_t^{\mathrm{lcm}}
\]

For MTP head-based approaches, loss summation and differential learning rates (heads at $4\times$ backbone LR), as well as head warmup protocols, are used to maintain stability and balance between prediction heads [2502.09419].

## 5. Decoding Strategies and Inference Efficiency

Speculative decoding with quadratic expansion enables NTPP to maximize inference throughput [2507.11851]:

- Beginning from verified history, two mask tokens are appended and processed to predict both verified and speculative tokens.
- Quadratic verification interleaves new masks after each speculative token, preventing depletion of viable sequence branches.
- Expected acceptance rate empirically approaches $k+1$ (i.e., $~3$ in the $k=2$ case), with practical speedups of $5\times$ on code/math and $2.5\times$ on chat for Tulu3-8B on standard benchmarks.

Marginalization-based approaches in next-token models, while exact, incur a significant computational overhead and scale linearly with vocabulary candidates, restricting practical acceleration unless batched with further speculative strategies [2502.09419].

## 6. Extensions to Dual-Channel Speech Language Modeling

NTPP is also applied to dual-channel speech dialogue modeling [2506.00975]:

- Input: Two continuous speech streams (A, B) are quantized via VQ or RVQ into discrete token sequences.
- Architecture: Decoder-only transformers process interleaved token pairs, using rotary positional and channel embeddings. In RVQ, cyclic depth embeddings are added.
- Attention masking ensures each pair is processed independently at each time step, preventing cross-attention between speakers.
- Training: A two-stage protocol—pretraining with next-token prediction, then fine-tuning with NTPP—on large-scale single-channel speech followed by paired conversational corpora (e.g., Fisher).
- Inference: NTPP uses a single KVCache, maintaining sublinear inference latency and outperforming cascaded ASR→LM→TTS approaches and dual-cache models on turn-taking statistics (lower disruptions, more natural overlaps).

## 7. Benchmark Results, Comparisons, and Practical Considerations

Empirical results on NTPP-via-masked-input for Tulu3-8B in text generation tasks [2507.11851]:

| Task (k=8)   | Acceptance Rate (Speedup) |
|--------------|---------------------------|
| Math         | 5.22×                     |
| Code         | 5.35×                     |
| Chat         | 2.52×                     |
| Knowledge    | ∼2.38×                    |

Ablations demonstrate additive gains from linear decoding, quadratic expansion, sampler head, and LCM loss components. Even low-rank ($r=4,16$) LoRA modules maintain $~4\times$ speedup.

For speech, turn-taking and inference efficiency metrics (per minute; NTPP vs. baselines):

| Model         | #IPU | #Pause | dur_IPU | dur_Pause |
|---------------|------|--------|---------|-----------|
| NTPP (T=0.5)  | 1.5  | 1.9    | 2.9     | 3.0       |
| dGSLM         | 1.6  | 3.4    | 4.6     | 3.6       |
| LSLM          | 2.2  | 3.6    | 4.1     | 3.4       |
| Cascaded      | 4.1  | 7.0    | 4.3     | 5.5       |

NTPP achieves lower pause/gap rates and maintains sub-220 ms latency across multi-turn dialogues.

Across implementations:

- NTPP introduces minimal overhead (e.g., two mask tokens, sampler head, gated LoRA modules).
- All code and hyperparameters are PyTorch/LoRA compatible.
- Speech models benefit from VAD-free turn-taking and robust speaker independence, enabled by the paired causal mask and unified KVCache.

## 8. Limitations, Open Questions, and Future Directions

Pretrained next-token models are strongly specialized for autoregressive targets; adapting them for multi-token prediction (MTP/NTPP) incurs performance degradation compared to theoretically exact marginalization [2502.09419]. Even joint LoRA training with weighted hidden states leaves a gap, suggesting that fully multi-token pretraining or deeper architectural changes may be required for optimal pairwise prediction.

For text-based models, extension to $k>2$ is straightforward in notation and implementation, although quadratic speculative decoding trees will require careful resource management. In speech modeling, integrating NTPP into end-to-end SLMs promises further latency reductions and improved turn-taking, but challenges remain in scaling, multilingual compatibility, and long-context alignment.

A plausible implication is that NTPP will become central for efficient generative modeling in future multimodal dialog agents and real-time inference engines, contingent on further advances in backbone adaptation and parallel decoding techniques.

Source: https://www.emergentmind.com/topics/next-token-pair-prediction-ntpp-7f3d191b-fd1c-40c2-a36c-75662fcbd5c3