Bebop Framework for MTP Acceleration

• The paper demonstrates that optimizing total variation loss in the Bebop framework achieves up to 95% MTP acceptance and 1.8× RL acceleration on large-scale LLMs.
• Bebop employs a novel probabilistic rejection sampling mechanism with entropy-bounded acceptance to robustly overcome rollout efficiency bottlenecks.
• The framework simplifies integration by requiring a single pre-RL MTP training phase, yielding substantial throughput gains and improved training stability.

Bebop is a systematic framework for accelerating reinforcement learning (RL) training in LLMs via Multi-Token Prediction (MTP) integrated with probabilistic rejection sampling. It addresses the bottleneck imposed by rollout efficiency in RL pipelines, providing a principled solution that achieves high acceptance rates and substantial inference throughput gains by optimizing for total variation (TV) distance, rather than conventional cross-entropy (CE) or KL objectives. Bebop enables stable, high-throughput training by decoupling acceptance rates from policy entropy fluctuations and is validated on large-scale Qwen model families, delivering up to 1.8× end-to-end RL acceleration and 95% MTP acceptance rates (Li et al., 10 Jun 2026).

1. Entropy-Bounded Acceptance in MTP Rollouts

MTP speculatively generates sequences of tokens (drafts), which are then accepted or rejected based on agreement with the target policy distribution. Acceptance rates are fundamentally limited by the entropy of the RL policy’s next-token distribution $p \in \Delta^{|V|}$, where $H(p) = -\sum_{v \in V} p(v)\log p(v)$.

Two acceptance schemes are compared:

• Target-Only (TO) Sampling: The acceptance rate is $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$.
  • It holds that $\max_y p(y) \geq \exp(-H(p))$.
  • Empirically, $$\alpha^{\rm TO} \approx a^{\rm TO} - b^{\rm TO} H(p)$$, showing a near-linear negative relationship with $H(p)$.
• Rejection Sampling (RS): The acceptance rate becomes $$\alpha^{\rm RS} = \sum_v \min\{p(v), q(v)\} = 1 - d_{\rm TV}(p, q)$$.
  • Under standard CE/KL training, acceptance rates degrade as $H(p)$ increases: $$\alpha^{\rm RS} \approx a^{\rm RS} - b^{\rm RS} H(p)$$.
  • In both schemes, higher entropy in $p$ compresses the single-step acceptance rate.

The negative linear dependency between model entropy and acceptance rate establishes a theoretical upper bound for MTP-based acceleration under naive training objectives.

2. Probabilistic Rejection Sampling Mechanism

Bebop replaces greedy (target-only) validation with a complete probabilistic rejection sampling regime. At each token generation step, for MTP rollout horizon $H(p) = -\sum_{v \in V} p(v)\log p(v)$0:

• At context $H(p) = -\sum_{v \in V} p(v)\log p(v)$1, the MTP draft head outputs logits $H(p) = -\sum_{v \in V} p(v)\log p(v)$2, yielding $H(p) = -\sum_{v \in V} p(v)\log p(v)$3.
• For step $H(p) = -\sum_{v \in V} p(v)\log p(v)$4:
  • Draw $H(p) = -\sum_{v \in V} p(v)\log p(v)$5.
  • Accept with probability $H(p) = -\sum_{v \in V} p(v)\log p(v)$6.
  • On rejection, sample the next token from the residual $H(p) = -\sum_{v \in V} p(v)\log p(v)$7.
• If all $H(p) = -\sum_{v \in V} p(v)\log p(v)$8 tokens are accepted, a "bonus" token is drawn from $H(p) = -\sum_{v \in V} p(v)\log p(v)$9.

Pseudo-code (condensed):

$\max_y p(y) \geq \exp(-H(p))$9 Rejection sampling yields higher acceptance rates relative to target-only approaches, particularly when the MTP head is appropriately aligned to the target via TV-based optimization.

3. End-to-End TV Loss for MTP Training

Bebop introduces an end-to-end TV (Total Variation) loss that directly optimizes rejection sampling acceptance, which is governed by the overlap between $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$0 and $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$1.

• Single-step TV loss:

$$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$2

• Multi-step (e2e) TV loss: For $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$3th draft step, $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$4. The expected accepted draft length (normalized) is

$$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$5

Bebop minimizes:

$$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$6

• Gradient structure:
  • CE/KL gradient: uniform mismatch per token, $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$7.
  • TV gradient: $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$8-proportional, concentrating on high-probability regions and yielding bounded magnitude for stable updates:

  $$\alpha^{\rm TO} = p(\arg\max_y q(y)) \approx \max_y p(y)$$9 - Reverse KL is mode-seeking and may zero out low-mass modes, decreasing $\max_y p(y) \geq \exp(-H(p))$0/$\max_y p(y) \geq \exp(-H(p))$1 overlap and harming acceptance. Only TV loss directly maximizes the desired acceptance metric.

4. Training and Integration Strategy

Bebop requires only a single pre-RL MTP training phase, simplifying pipeline integration.

• Pre-RL MTP Training:

  • Conducted after supervised fine-tuning (SFT), with the LLM backbone frozen.
  • Only the MTP heads are trained, using the e2e TV loss.
  • Example hyperparameters (Qwen3.5-35A3B): batch size 256, sequence length 256k, 1 epoch, learning rate $\max_y p(y) \geq \exp(-H(p))$2 with 3% warmup, multi-step horizon $\max_y p(y) \geq \exp(-H(p))$3.
  • Utilizes Megatron infrastructure and a fused full-vocabulary TV kernel.
• Online MTP Training:
  • Not required. The decomposition $\max_y p(y) \geq \exp(-H(p))$4 reveals that, with rejection sampling, $\max_y p(y) \geq \exp(-H(p))$5 remains near zero.
  • Acceptance drift during RL is accounted for by entropy changes, not head mismatch.
  • Online TV loss updates confer negligible benefit except when RL policy entropy moves substantially outside SFT regime.

A single pre-RL TV-trained MTP head suffices for stable operation throughout RL.

5. Empirical Validation and Quantitative Analysis

Comprehensive experiments on Qwen model families demonstrate Bebop’s efficacy across tasks.

• SFT Acceptance Gains (γ=3, RS):

| Task | CE | e2e TV | Δ | |----------|------|--------|-------| | Math | 75.0 | 78.0 | +3.0 | | Code | 71.3 | 74.6 | +3.3 | | SWE | 75.1 | 83.1 | +8.0 | | Agent | 90.3 | 97.0 | +6.7 | | MT-Bench | 65.3 | 67.6 | +2.3 |

Bebop’s e2e TV loss delivers consistent 3–8 point improvements in acceptance rates across mathematical reasoning, code generation, and agentic environments.

• Throughput Gains:
  • Rejection sampling throughput tracks acceptance rate linearly.
  • e2e TV yields up to 25% more tokens/sec relative to CE.
• RL Training Acceleration (Async Rollout):

| Model | Speedup | |-----------------|---------| | Qwen3.5 (SWE) | 1.7× | | Qwen3.6 (Hybrid)| 1.5× | | Qwen3.7 (Agent) | 1.8× |

End-to-end wall-clock accelerations reach 1.8× on large-scale models.

• Ablations and Analyses:
  • TV training reduces the negative entropy-acceptance slope from approximately −1.68 (CE/KL) to approximately −0.06 (RS + TV).
  • Decomposition confirms $\max_y p(y) \geq \exp(-H(p))$6 under RS + TV.
  • Online CE updates during RL degrade $\max_y p(y) \geq \exp(-H(p))$7 back toward the CE baseline; additional online TV co-training yields no further advantage.
  • Empirically, RS outperforms TO on nearly all native drafts as $\max_y p(y) \geq \exp(-H(p))$8.

6. Significance and Implications

Bebop establishes a new regime for efficient RL training in LLMs by demonstrating that MTP acceleration is fundamentally entropy-bounded unless the head-target overlap is maximized via TV optimization and that practical wall-clock gains accrue by decoupling acceptance from entropy fluctuations through probabilistic rejection sampling. The framework enables up to 95% acceptance rates, 25% throughput increases, and 1.8× end-to-end RL acceleration without the need for online MTP co-training, representing a robust advance in scalable RL for LLMs (Li et al., 10 Jun 2026). A plausible implication is the elimination of rollout as a systemic bottleneck in large-scale RL pipelines for LLM development, provided model entropy remains within SFT-calibrated regimes.