Conditional MTP Projector (cMTPP)

• The Conditional MTP Projector (cMTPP) is a lightweight, parameter-efficient module that generates dynamic multi-token predictions via adaptive normalization and gated MLP transformations.
• It integrates with the Temporal Guidance framework to contrast 'amateur' logits with expert predictions, boosting generation quality while minimizing computational and memory overhead.
• Empirical evaluations demonstrate improved benchmark performance with reduced VRAM usage and latency compared to traditional dual-model contrastive decoding methods.

The Conditional MTP Projector (cMTPP) is a parameter-efficient, auxiliary projection module introduced for self-contrastive decoding in LLMs within the Temporal Guidance (TeGu) framework. Its core function is to enable dynamic multi-token prediction (MTP) for arbitrary temporal offsets $k$ without the overhead of multiple independent networks or full auxiliary models. By leveraging lightweight transformations and adaptive normalization of cached hidden states, cMTPP facilitates the construction of "amateur" predictions for contrastive learning in LLM decoding processes, yielding measurable improvements in generation quality while maintaining a compact computational and memory footprint (Zheng et al., 29 Jan 2026).

1. Role and Motivation within Temporal Guidance

Temporal Guidance (TeGu) implements self-contrastive decoding by contrasting predictions from an "expert" (standard forward LLM prediction) with those of an "amateur" model, conditioned on positions $k$ steps in the past. At each generation step $t$, the scheme contrasts

• Expert logits: $\log P_\mathrm{exp}(x_t \mid x_{<t})$
• Amateur logits: $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$

Guided scores are computed as:

$$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$

However, native LLM architectures typically lack flexible MTP heads capable of generating predictions for arbitrary $k$. The cMTPP module addresses this gap by providing a compact, shared mechanism for producing "amateur" logits given a cached hidden state $h_{t-1-k}$ and a step-offset $k$, circumventing the need for extra full-sized models or multiple MTP heads. The output is passed through the frozen LLM (LM) head, ensuring compatibility and efficiency.

2. Architecture and Workflow

The cMTPP module transforms a cached last-layer hidden state and a temporal step-index into logits approximating $P(x_t \mid x_{<t-k})$. The principal components and data flow are:

• Input representations:

(1) $k$0: cached hidden state. (2) $k$1: integer offset.

• Step-ID Embedding:

$k$2

• Adaptive LayerNorm (AdaLN):

Compute scale $k$3 and bias $k$4 via an MLP over $k$5

$k$6

• Gated Feed-Forward Network ("SwiGLU" variant):

$k$7 $k$8

$k$9

• Down-projection:

$t$0

• LM head projection:

$t$1

Parameters:

• $t$2, $t$3, $t$4 (expansion factor $t$5)
• AdaLN and StepEmbed parameters are minor in scale

The full LM backbone and its output projection $t$6 are frozen during cMTPP training.

3. Mathematical Representation

Let $t$7, $t$8 the offset index. The transformations are:

1. Adaptive normalization:

$t$9

2. Gated MLP with SwiGLU:

$\log P_\mathrm{exp}(x_t \mid x_{<t})$0

where $\log P_\mathrm{exp}(x_t \mid x_{<t})$1

3. Down-projection:

$\log P_\mathrm{exp}(x_t \mid x_{<t})$2

4. Final logits via frozen head:

$\log P_\mathrm{exp}(x_t \mid x_{<t})$3

These "amateur" logits are used for contrastive adjustment of the expert logits as per TeGu.

4. Training Objective and Optimization

All parameters of cMTPP are trained while freezing the LLM backbone. For offsets $\log P_\mathrm{exp}(x_t \mid x_{<t})$4 at training position $\log P_\mathrm{exp}(x_t \mid x_{<t})$5:

• Cross-entropy loss (CE):

$\log P_\mathrm{exp}(x_t \mid x_{<t})$6

• KL-Distillation loss (KD):

$\log P_\mathrm{exp}(x_t \mid x_{<t})$7

• Total loss:

$\log P_\mathrm{exp}(x_t \mid x_{<t})$8

$\log P_\mathrm{exp}(x_t \mid x_{<t})$9 controls CE/KD mixing (e.g., $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$0); KD can use temperature $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$1.

Ablation experiments indicate that combining CE and KD objectives stabilizes and enhances benchmark performance, whereas CE-only objectives yield degradation or instability.

5. Efficiency: Parameter, Memory, and Computation Analysis

A summary of cMTPP’s computational cost and memory usage:

• Parameter count:

$\log P_\mathrm{amt}(x_t \mid x_{<t-k})$2 collectively: $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$3 parameters ($\log P_\mathrm{amt}(x_t \mid x_{<t-k})$4). For $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$5, overhead is $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$6 million params ($\log P_\mathrm{amt}(x_t \mid x_{<t-k})$7 for 8B models).

• FLOPs per inference step:

Three matrix multiplications: $\log P_\mathrm{amt}(x_t \mid x_{<t-k})$8 FLOPs per token, substantially less than a full second forward through the backbone.

• Additional memory and latency:

On Qwen3-8B, cMTPP increases VRAM usage from 17.72 GB (greedy) to 19.72 GB ($\log P_\mathrm{amt}(x_t \mid x_{<t-k})$9), and time by just 2%.

Decoding Mode	VRAM (GB)	Latency (×)
Greedy	17.72	1.0
Standard CD (1.7B)	23.11	1.2
DoLa	19.22	1.04
TeGu + cMTPP	19.72	1.02

This overhead is substantially lower than that of dual-model contrastive decoding and close to DoLa.

6. Empirical Evaluation and Ablations

Main Results

• On Qwen3-1.7B (α=0.2): GSM8K: $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$0, IFEval: $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$1
• On Qwen3-8B (α=0.5): Math500: $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$2, IFEval: $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$3

Key Ablation Insights

• CE-only training leads to instability; the inclusion of KL-distillation is critical for robust task performance across all evaluated benchmarks.
• Single-step offset ($$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$4) as the amateur outperforms higher $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$5 or mixtures: larger $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$6 dilute the contrastive effect and reduce accuracy.
• Optimal guidance strength $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$7 depends on model size: smaller models peak at $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$8, while larger models tolerate up to $$V(x_t) = \log P_\mathrm{exp}(x_t) + \alpha \left[ \log P_\mathrm{exp}(x_t) - \log P_\mathrm{amt}(x_t) \right]$$9.
• Amateur logits generated by cMTPP demonstrate elevated entropy relative to the expert head, confirming their function as high-uncertainty, "amateur" projections.

7. Implementation Considerations and Pseudocode

Principal hyperparameters include expansion ratio $k$0, loss weights (CE: 0.3, KD: 0.7), KD temperature $k$1, optimizer AdamW (peak LR 2e–4), cosine schedule, and $k$2 warmup.

cMTPP is used during inference as follows:

$k$3

The cMTPP forward step is:

$k$4

All LLM backbone parameters remain frozen throughout cMTPP training; only cMTPP parameters are updated (Zheng et al., 29 Jan 2026).