Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

Abstract: Reinforcement learning (RL) has become a key component in modern LLMs, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance. To address this bottleneck, we present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines. First, we reveal that the MTP acceptance rate is fundamentally bounded by the fluctuation of model entropy, which demonstrates a clear negative linear relationship with the rise of entropy in the RL stage. Second, we show that probabilistic rejection sampling largely alleviates the disturbance introduced by entropy in RL compared to greedy draft sampling. We further identify that the conventional MTP training objectives (cross-entropy or KL) are suboptimal in such settings, and therefore we propose a novel end-to-end TV loss that directly optimizes multi-step rejection sampling acceptance rate, yielding ~10% acceptance rate improvements, achieving up to 95% acceptance rates and up to 25% extra inference throughput gains across mathematical reasoning, code generation, and agentic tasks. Third, we test various online MTP training strategies during RL and show that pre-RL MTP training with e2e TV loss and rejection sampling achieves a consistent acceptance rate and speedup throughout the entire RL, eliminating the need for costly online MTP updating. We provide extensive experiments and analysis that validate our findings. Experimental results show our method achieves up to 1.8x end-to-end acceleration in async RL training of Qwen3.5, Qwen3.6, and Qwen3.7 models.

• The paper demonstrates that policy entropy, rather than policy–draft mismatch, fundamentally limits MTP acceptance during RL training.
• It introduces an end-to-end Total Variation loss paired with probabilistic rejection sampling, resulting in up to 25% throughput gains and 1.8x wall-clock speedup.
• Empirical analysis across Qwen3.5 to Qwen3.7 models confirms that the Bebop framework decouples acceptance rates from entropy dynamics, enhancing RL scalability.

Breaking Entropy Bounds in RL LLM Training: The Bebop Framework for MTP Acceleration

Introduction

The paper "Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling" (2606.12370) presents a comprehensive analysis and system for integrating Multi-Token Prediction (MTP) into reinforcement learning (RL) pipelines for LLM post-training. At its core, the study identifies entropy fluctuations in policy models during RL as the primary limiting factor for MTP acceptance rates, rather than the traditionally presumed policy–draft distribution mismatch. The authors introduce a novel end-to-end (e2e) Total Variation (TV) loss for MTP pretraining and empirically demonstrate that pairing this with probabilistic rejection sampling, rather than greedy verification, largey decouples acceptance rate from entropy trends during RL, yielding up to 25% inference throughput gains and $1.8\times$ wall-clock RL acceleration.

MTP Acceptance Rate Bottlenecks: Entropy Versus Distributional Mismatch

Extensive theoretical and experimental analysis establishes that RL-induced policy entropy fluctuations drive a fundamentally linear degradation in MTP acceptance rate. The empirical data reveal that as the policy exploration increases entropy, MTP's ability to accept speculative drafts diminishes in proportion, and this degradation dominates over effects caused by draft–policy drift (i.e., distributional mismatch due to frozen MTP heads during RL fine-tuning):

Figure 1: (a) MTP acceptance rates degrade linearly with policy entropy fluctuation in RL; training MTP with the e2e TV loss largely eliminates this entropy dependence under rejection sampling. (b) The TV-trained MTP achieves substantially better distributional overlap with the policy model, yielding superior acceptance rate and speedup.

The paper provides a decomposition analysis (see Figure 2 in the paper), empirically showing the dominance of entropy-driven acceptance drops over mismatch-driven drops across tasks and model scales. This insight directly challenges recent proposals to continuously co-train MTP heads online during RL, which incurs significant compute overhead with marginal benefit.

Figure 3: Per-step MTP acceptance rates during SWE-bench RL training with Qwen3.5-3.6 Plus. Later MTP steps exhibit progressively larger degradation, with stepwise losses accumulating over generation.

Figure 2: Decomposition of acceptance length changes during RL training. The entropy-driven component dominates total acceptance loss, while mismatch between draft and policy head accounts for little degradation.

Acceptance Methods: Greedy Target-Only Versus Probabilistic Rejection Sampling

The study contrasts two verification approaches for MTP:

• Target-Only Sampling: Accepts draft tokens if they are the $\arg\max$ under policy probability $p$, but is fundamentally capped by $\max_y p(y)$, which degrades rapidly as entropy rises.
• Rejection Sampling: Accepts a multinomial sample from the draft with probability $\min(1, p(\hat{y}) / q(\hat{y}))$, making acceptance proportional to $\sum_v \min(p(v), q(v))$, i.e., distributional overlap, and thus preserving higher acceptance under high-entropy regimes.

Empirically, switching to rejection sampling raises the acceptance plateau and smooths entropy-induced variance, provided the draft model is well aligned.

Figure 4: Accept length during RL training across different workloads using Qwen3.6-Plus and Qwen3.7-Max. Rejection sampling with TV loss (RS w/ TV) sustains higher accept lengths than target-only or RS with conventional CE loss.

Novel e2e TV Loss: Directly Maximizing Multi-Step Overlap

Conventional MTP objectives, e.g., cross-entropy (CE) or KL divergence, only minimize a loose upper bound (via Pinsker’s inequality) on the TV distance, and their gradients distribute learning uniformly across the vocabulary, leading to inefficient capacity allocation and entropy-sensitive acceptance drops.

To address this, the authors propose an e2e TV loss, which directly maximizes the multi-step acceptance length:

$$\mathcal{L}_{\mathrm{e2e}} = 1 - \frac{1}{\gamma} \sum_{j=1}^{\gamma} \prod_{i=1}^j \alpha_i$$

where $\alpha_i = 1 - d_\mathrm{TV}(p_i, q_i)$ at each step $i$, and $\gamma$ is the speculative generation depth.

This TV loss possesses key properties:

• Gradients are concentrated on tokens with high probability mass.
• It yields probability-proportional mismatch, allocating more fine-tuning effort to high-likelihood modes and largely ignoring the tail.
• The acceptance rate becomes nearly independent of entropy, as shown empirically and predicted by their analysis.

Figure 5: CE loss (solid) vs. TV loss (dashed) during SFT training. TV loss consistently delivers higher acceptance rates, with gains most pronounced on complex agentic tasks.

Empirical Gains and Scaling Results

Extensive experiments on Qwen3.5, Qwen3.6, and Qwen3.7 families validate the practical performance of Bebop:

• e2e TV loss plus rejection sampling yields up to 8% higher acceptance rates than CE/KL baselines across reasoning, code, and agentic benchmarks.
• The throughput of async RL steps increases by 25%, translating to up to $\arg\max$0 total wall-clock RL speedup.
• The method scales effectively: as model size grows to Qwen3.7-Max, per-step acceptance rates approach 95%, especially on highly agentic or structured outputs.

Figure 6: MTP with rejection sampling (RS w/ TV) reduces RL step latency compared to baselines, yielding substantial acceleration.

Figure 7: Entropy loss vs. accept length across RL workloads. TV-trained drafts (RS w/ TV) show entropy-invariant acceptance, while CE-trained models degrade linearly with $\arg\max$1. This confirms that e2e TV loss breaks the entropy–acceptance coupling.

The authors further show that no significant gain is afforded by updating MTP heads during RL, as entropy-driven effects subsume residual mismatch.

Figure 8: (a) Accept length during RL with and without MTP weight updates. TV loss mitigates acceptance drops even without online updates. (b) Acceptance rate delta correlates strongly with speedup, confirming throughput scaling is driven by acceptance improvements.

Robustness, Generalization, and Ablations

The study demonstrates several robustness characteristics:

• The e2e TV loss grants stable acceleration benefits throughout RL, without retraining, as entropy or temperature schedules vary.
• Rejection sampling generally outperforms target-only methods except in extreme distributional drift, which is rare when pretraining includes sufficiently varied SFT data.
• Generalization to new domains is robust: models pretrained without agent data still achieve substantial acceptance gains on agentic workloads due to the distributional overlap optimization.

Figure 9: (a) Acceptance as a function of sampling temperature. Rejection sampling sustains stable acceptance as temperature increases, whereas target-only degrades rapidly. (b) Acceptance rate is robust across generation positions.

Figure 10: RS decision boundary across models. Nearly all production-grade MTP models fall in the RS-superior region.

Implications and Future Directions

The Bebop framework provides actionable guidance for large-scale LLM RL, showing that:

• MTP can be seamlessly integrated into RL rollouts for significant acceleration, without introducing optimization instabilities or necessitating online draft co-training.
• Direct TV loss optimization should replace CE/KL objectives in practical MTP deployments, especially under distributional shift and high-entropy exploration phases.
• Rollouts for long-horizon, agentic, or tool-use scenarios particularly benefit from entropy-invariant MTP acceleration, suggesting further application in advanced dialog and planning settings.

Remaining open questions include formal tightening of the assumed proportional-mismatch model underlying TV-trained acceptance stability, as well as adapting MTP–TV loss to out-of-distribution or extreme-entropy RL phases potentially by dynamic online adaptation.

Conclusion

This work rigorously establishes that model entropy, not just policy–draft mismatch, is the fundamental ceiling on MTP acceptance in RL pipelines for LLMs. By directly learning to maximize distributional overlap through e2e TV loss and employing rejection sampling, Bebop overcomes these entropy-imposed bottlenecks—providing stable, scalable, and highly efficient RL training acceleration. These findings carry practical import for the design of next-generation LLM RL pipelines and inform future approaches to speculative decoding, policy optimization, and large-scale LLM training.