Group-Based Draft Policy Training

• Group-based draft policy training is a family of methods that use group comparisons and tree-level rewards, moving beyond traditional token-level supervision.
• The approach aligns training with deployment by integrating speculative decoding, group-standardized advantages, and adaptive control strategies to improve acceptance length and speed.
• These methods, applied in settings like reinforcement learning and error replay, yield significant throughput gains and performance boosts across dialogue, code, and mathematical reasoning benchmarks.

Searching arXiv for the specified papers and closely related work on group-based draft policy training. Group-based draft policy training denotes a family of optimization procedures in which a draft component is trained through grouped comparisons rather than isolated next-token prediction. In the speculative decoding setting, a lightweight draft model proposes blocks or trees of candidate tokens that a larger target model verifies in parallel; in draft-conditioned reinforcement learning, externally supplied drafts are injected into the learner’s context during on-policy optimization. Across these settings, recent work replaces single-path supervision with tree-level rewards, group-standardized advantages, replay from draft-induced error positions, or co-adaptive control over drafting and verification, with the explicit aim of aligning training with deployment-time behavior and improving acceptance length, throughput, or downstream reasoning performance (Hu et al., 26 Sep 2025, Lei et al., 28 May 2026).

1. Problem setting and motivating mismatches

In speculative decoding, a target model is paired with a lightweight draft model whose proposed tokens are verified in parallel. A recurrent observation in this literature is that the deployment policy is structurally richer than the training signal. Existing objectives often optimize a single greedy draft path, whereas decoding-time execution follows a tree policy that re-ranks and verifies multiple branches. GTO identifies this as a draft policy misalignment: training effort is concentrated on paths that are often not the ones ultimately accepted at inference, and less than 50% of accepted inference branches match the training-time greedy path (Hu et al., 26 Sep 2025).

A second mismatch arises from data collection. Draft models such as EAGLE-3 or DFlash are commonly trained by supervised fine-tuning on target-generated trajectories. Draft-OPD reports that this procedure quickly plateaus: the draft model’s acceptance length on test data stops improving because the drafter is trained on fixed target-generated states, while inference evaluates blocks proposed under the drafter’s own policy. Traditional on-policy distillation addresses exposure bias for standalone autoregressive students, but block-based or tree-based drafters cannot reliably self-roll out complete sequences, and target-assisted generation alone removes the on-policy signal that would expose the drafter’s failure modes (Lei et al., 28 May 2026).

These observations motivate group-based training signals. Instead of asking whether a single token was correct under teacher forcing, recent methods ask which branch in a tree produced the highest expected acceptance length, which draft-induced states caused rejection, or which coordination of draft depth and verification budget maximized tokens per second. This suggests that draft policy training is increasingly being defined by deployment-aligned grouping structures rather than by token-level imitation alone.

2. Tree-aligned objectives and group-standardized optimization

Group Tree Optimization (GTO) makes the tree policy itself the training target. For a prefix $\mathbf{x}_{1:t}$, a draft tree $\mathbf{T}_t$ is constructed by the draft model under the same layer-wise top-$k$ expansion, pruning, and re-ranking policy used at inference. The central objective is the Draft Tree Reward, a sampling-free quantity equal to the expected acceptance length of the draft tree under the target model:

$$\mathbf{r}_t = \mathcal{R}(\mathbf{T}_t; \eta) = \frac{1}{\eta}\log\left(\sum_{i=1}^N \exp(\eta\,\mathbf{L}_{t,i})\right),$$

where $\mathbf{L}_{t,i}$ is the expected acceptance length of branch $i$ and $\eta$ is a smooth-max temperature. The paper states that increasing this reward provably improves acceptance length and speedup (Hu et al., 26 Sep 2025).

GTO then stabilizes optimization through a reference-model contrast. A frozen draft model $\mathcal{M}_0$ generates reference trees $\bar{\mathbf{T}}_i$, and the reward used for optimization is debiased by subtraction:

$$\mathbf{R}_i = \mathcal{R}(\mathbf{T}_i) - \mathcal{R}(\bar{\mathbf{T}}_i).$$

Within each group $\mathbf{T}_t$0 of nearby positions, rewards are standardized as

$\mathbf{T}_t$1

and a PPO-style clipped surrogate is applied along the longest accepted sequence:

$\mathbf{T}_t$2

The final objective combines this groupwise surrogate with a token-level loss. The method is explicitly described as Group-based Draft Policy Training, and its role is not merely variance reduction but alignment: the same tree policy used at inference defines the reward surface during training. Empirically, GTO reports a 7.4% increase in acceptance length and an additional 7.7% speedup over EAGLE-3 across dialogue, code, and math benchmarks, including LLaMA-3.1-8B, LLaMA-3.3-70B, Vicuna-1.3-13B, and DeepSeek-R1-Distill-LLaMA-8B (Hu et al., 26 Sep 2025).

3. On-policy distillation, error replay, and adaptive draft models

Draft-OPD addresses the offline-to-inference mismatch by combining target-assisted rollout with replay from verification-exposed error positions. During speculative decoding, the draft model proposes $\mathbf{T}_t$3-token blocks and the target model verifies them. After each verification, the method records the anchor position where the block began. Training then replays drafting from those anchors, computes draft and target log-probabilities for each token, and partitions tokens into accepted and rejected sets. Accepted tokens are optimized with forward KL,

$\mathbf{T}_t$4

while rejected tokens use reverse KL with exponentially decaying weights $\mathbf{T}_t$5,

$\mathbf{T}_t$6

The total loss averages these two terms. The rationale given in the paper is that forward KL is data-efficient on accepted states, whereas reverse KL penalizes the drafter’s confident mistakes on rejected states. Draft-OPD reports over $\mathbf{T}_t$7 lossless acceleration for thinking models, with improvements over EAGLE-3 and DFlash by 23% and 13%, and higher throughput in SGLang by 7–17% than DFlash even at high concurrency (Lei et al., 28 May 2026).

FastGRPO treats the draft model as a dynamically changing component inside group-based RL training. It identifies generation as the primary bottleneck in GRPO, accounting for over 90% of end-to-end training time, and argues that direct application of speculative decoding is limited in high-concurrency training regimes. Its concurrency-aware speculative decoding adjusts drafting and verification parameters according to the current number $\mathbf{T}_t$8 of unfinished sequences:

$\mathbf{T}_t$9

$k$0

The second component, online draft learning, continuously updates the draft model on outputs generated by the evolving target policy so that acceptance rates do not decay as GRPO changes the target distribution. Reported end-to-end speedups are 2.35x to 2.72x across mathematical reasoning datasets and models, and the paper states that online draft learning adds a further 0.7x–0.9x generation speedup in ablations (Zhang et al., 26 Sep 2025).

Together, these methods establish a common technical theme: draft models are most effective when training explicitly targets draft-induced states, rejected proposals, and nonstationarity in the target policy rather than treating the drafter as a static auxiliary model.

4. Co-adaptive control of drafting and verification

Learning to Draft (LTD) reframes speculative decoding as a reinforcement learning environment in which two lightweight policies jointly control the draft-and-verify cycle. One policy chooses draft depth—whether to continue or stop expanding the draft tree—and the other chooses verification size, selecting how many candidate tokens to submit for target-model verification. The reward is direct throughput rather than a proxy metric:

$k$1

where $k$2 is the acceptance length and $k$3, $k$4 are drafting and verification latencies (Zhang et al., 2 Mar 2026).

The training procedure is explicitly co-adaptive. LTD first trains the size policy with randomly sampled draft depths and the depth policy with a fixed verification size. It then alternates optimization rounds while holding the other policy fixed, so that each controller adapts to the current behavior of the other. PPO is used for policy updates, and real wall-clock measurements define the reward. The paper emphasizes that prior dynamic methods typically optimize acceptance length or rely on hand-tuned static schedules, whereas LTD directly optimizes throughput and treats draft and verification phases as a joint decision problem (Zhang et al., 2 Mar 2026).

Reported speedup ratios range from 2.24x to 4.32x across five open-source LLMs and four tasks. Against Eagle3, LTD reports gains of up to 36.4%. An important empirical nuance is that higher acceptance length is not treated as sufficient: LTD can outperform grid-searched baselines in throughput even when its average acceptance length is slightly lower, because it avoids spending verification budget on low-value candidates. The policy-decision overhead is reported as less than 1.5% of generation time even for the smallest LLM evaluated (Zhang et al., 2 Mar 2026).

A plausible implication is that group-based draft policy training extends beyond learning the token proposer itself. It can also include learning control policies over computational allocation, provided that those policies are trained against grouped speculative-decoding outcomes rather than token-level cross-entropy.

5. Draft-conditioned reinforcement learning and weak-to-strong elicitation

A distinct use of drafts appears in Weak-to-Strong Elicitation via Mismatched Wrong Drafts, where the draft is not a speculative decoder but an off-policy reasoning trace injected into the strong learner’s prompt during GRPO fine-tuning. The setup uses Mathstral-7B-v0.1 as the learner, Qwen2.5-Math-1.5B as the draft model, approximately 8.8K Level 3–5 MATH problems for training, MATH-500 held out for validation, and Dr. GRPO with binary outcome-only reward, LoRA rank 16, AdamW, learning rate $k$5, and group size 16, all on a single NVIDIA B200 GPU (Deng, 17 May 2026).

For each training problem, 32 draft completions are sampled from the weaker model. A non-trivially wrong draft is selected, and then drafts are assigned either to the original problem (matched) or to a different problem via a random 1-1 derangement (mismatched). The prompt explicitly tells the learner that the thinking section may contain errors and asks it to write its own correct solution. At evaluation, the draft field is replaced with the literal string “N/A”. No SFT, no reward models, no synthesized data, and no produce-critique-revise inner loop are used (Deng, 17 May 2026).

The empirical result is that mismatched wrong drafts outperform all tested variants. On MATH-500, the paper reports 71.98% pass@1 for the mismatched-wrong variant, compared with 70.82% for no-draft GRPO, 70.36% for matched-wrong, and 69.02% for mismatched-correct. On AIME 2025 and 2026, the mismatched-wrong variant is reported to uniquely lift pass@$k$6 above both the Mathstral-7B base model and the Qwen2.5-Math-1.5B draft model at every sample budget from $k$7 to $k$8 across 2 seeds, with gains of +14.2 percentage points on 2025 and +9.0 percentage points on 2026 at pass@1024 over Mathstral-7B (Deng, 17 May 2026).

The paper’s interpretation is specific: neither wrongness alone nor mismatch alone is sufficient. Matched-correct drafts create copying shortcuts; matched-wrong drafts create anchoring traps; mismatched-correct drafts can induce analogy overfit. Mismatched-wrong drafts, by contrast, are argued to force the learner to rely on its own reasoning while still shifting the context distribution. This suggests that “draft policy training” can also refer to the use of external drafts as structured perturbations in on-policy RL, not only to optimization of speculative-decoding drafters.

6. Empirical landscape, recurrent assumptions, and design tensions

Across recent work, group-based draft policy training has split into several operational regimes. The table summarizes representative methods and their reported effects.

Method	Training signal	Reported effect
GTO (Hu et al., 26 Sep 2025)	Draft Tree Reward, reference-tree debiasing, group-standardized PPO surrogate	+7.4% acceptance length; +7.7% speedup over EAGLE-3
Draft-OPD (Lei et al., 28 May 2026)	Forward KL on accepted tokens, reverse KL on rejected tokens, replay from anchors	Over $k$9 lossless acceleration; +23% over EAGLE-3; +13% over DFlash
FastGRPO (Zhang et al., 26 Sep 2025)	Concurrency-aware speculative decoding and online draft learning	2.35x to 2.72x end-to-end speedup
LTD (Zhang et al., 2 Mar 2026)	PPO on throughput reward for draft depth and verification size	2.24x to 4.32x speedup; up to 36.4% over Eagle3
Mismatched wrong drafts (Deng, 17 May 2026)	Dr. GRPO with injected mismatched wrong drafts	71.98% MATH-500 pass@1; +14.2pp and +9.0pp at AIME 2025/2026 pass@1024

Several recurrent assumptions are directly challenged by these results. One assumption is that training the greedy draft path is an adequate proxy for deployment-time decoding. GTO disputes this by showing that the decoding-time tree policy can diverge sharply from the training-time greedy path and by optimizing the expected acceptance length of the full tree instead (Hu et al., 26 Sep 2025). A second assumption is that more offline supervised data is sufficient to improve draft models. Draft-OPD disputes this by locating the bottleneck in draft-induced errors rather than in teacher-forced target trajectories (Lei et al., 28 May 2026).

A third assumption is that acceptance length is the correct optimization target for efficiency. LTD disputes this by optimizing throughput directly and showing that verification cost can dominate the utility of longer accepted spans (Zhang et al., 2 Mar 2026). A fourth assumption is that a fixed draft model remains adequate during RL post-training. FastGRPO disputes this by identifying distributional drift between the evolving target model and the fixed drafter and by updating the draft model online (Zhang et al., 26 Sep 2025). A fifth assumption is that helpful drafts should be correct and matched to the current problem. The weak-to-strong results dispute this in the specific domain of mathematical reasoning, where mismatched wrong drafts were the most effective variant tested (Deng, 17 May 2026).

The principal design tension running through the literature is between alignment to deployment and stability of optimization. Tree-level and throughput-level objectives improve deployment fidelity but introduce sparse, nonlocal, or hardware-dependent reward surfaces. Group standardization, reference-model debiasing, clipped surrogates, and mixed-KL losses are the mechanisms currently used to control that instability. Another tension is between acceptance-oriented and time-oriented criteria: acceptance length, accepted-token probability, and wall-clock throughput are related but not identical objectives. Recent work increasingly treats this distinction as fundamental rather than cosmetic.

A plausible broader implication is that draft policy training is becoming a distinct subfield within LLM optimization. Its unit of analysis is no longer just the next token, but the draft-induced computation graph: branches, blocks, verification budgets, rejection points, and even off-policy draft traces embedded in the learner’s context.