PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length

Abstract: Speculative decoding (SD) is a powerful technique for accelerating the inference process of LLMs without sacrificing accuracy. Typically, SD employs a small draft model to generate a fixed number of draft tokens, which are then verified in parallel by the target model. However, our experiments reveal that the optimal draft length varies significantly across different decoding steps. This variation suggests that using a fixed draft length limits the potential for further improvements in decoding speed. To address this challenge, we propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer. This layer pre-verifies draft tokens blockwise before they are sent to the target model, allowing the draft model to stop token generation if the blockwise pre-verification fails. We implement Pacer on multiple SD model pairs and evaluate its performance across various benchmarks. Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding. Furthermore, when integrated with Ouroboros, Pacer attains up to 3.09x Speedup.

• The paper introduces a blockwise pre-verification layer on top of the draft model to dynamically decide when to stop the speculative draft, improving decoding efficiency.
• The preverification layer offers speedups of up to 2.66x alone and 3.09x when combined with Ouroboros.
• Pacer’s effectiveness is shown in multiple benchmarks, preserving draft quality while offering a significant speed benefit.

Pacer addresses a well-known inefficiency in speculative decoding (SD): the draft window size $\gamma$ is fixed across all decoding steps, even though the number of draft tokens the target model will accept varies substantially from step to step. The paper proposes a trainable, lightweight pre-verification layer that runs atop the draft model and decides blockwise whether drafting should continue before the target model is invoked. Across four model families and multiple benchmarks, Pacer achieves up to $2.66\times$ speedup over autoregressive decoding and up to $3.09\times$ when combined with Ouroboros (2602.01274).

Motivation: fixed window sizes are suboptimal

The authors first establish empirically that optimal draft lengths are highly variable. Using DeepSeek-Coder 1.3B/33B on HumanEval with an oversized draft window, they measure the maximum acceptance length $L_A^\star$ at each decoding step and observe large fluctuations around the best fixed window ($\gamma=9$). Two failure modes follow from fixing $\gamma$: when $\gamma > L_A^\star$, computation is wasted on drafts that are rejected; when $\gamma < L_A^\star$, drafting terminates prematurely and incurs additional expensive target-model forward passes. An oracle experiment using the per-step optimal dynamic window reduces target forward passes from 3,047 to 1,150 relative to fixed-window SD, yielding a $1.4\times$ throughput improvement. This oracle result motivates the entire framework, though it also defines an upper bound that Pacer can only approximate.

A third observation shapes the architecture: acceptance rates decline sharply with draft position, so positional information is a strong predictor of acceptance. This directly justifies adding position embeddings to the pre-verification layer.

Method

Pacer inserts a single-layer Transformer $M_B$ between the draft model $2.66\times$0 and the target model $2.66\times$1. Drafting proceeds in blocks of size $2.66\times$2: after each block, $2.66\times$3 consumes the draft hidden states augmented with positional embeddings, together with hidden states of previously accepted tokens, and outputs estimated acceptance rates $2.66\times$4. If the mean predicted acceptance over the block falls below a threshold $2.66\times$5, drafting stops and the accumulated $2.66\times$6 tokens are verified by the target model; otherwise another block is generated. The threshold is multiplied by a growth factor $2.66\times$7 at each round, making early stopping progressively easier for longer drafts — a heuristic correction for the fact that training data is dominated by short drafts, causing the predictor to under-estimate acceptance at long positions.

Blockwise operation serves two purposes. It amortizes the latency of the pre-verification layer (measured at 1.81 ms per forward pass versus 16.52 ms for the 1.3B draft model), and it avoids misclassification of individual tokens by aggregating evidence over a block.

Training data is constructed to match inference exactly: the target model generates reference responses, then SD with a large window ($2.66\times$8) produces drafts labeled accepted or rejected based on token-level agreement with the target output. The layer is trained with cross-entropy loss. Because successive draft steps share prefixes, the authors pack all draft steps for a prompt into one sequence with a custom attention mask restricting each draft token to its own prefix and preceding drafts, which substantially improves training efficiency. Total training time ranges from 18 to 47 minutes per model pair on 8×A800 GPUs — a modest cost.

Main results

Pacer was evaluated on HumanEval, MBPP, GSM8K, and CNN/DM using DeepSeek-Coder (1.3B/33B, 6.7B/33B), Llama-2-chat (7B/70B), and Qwen2.5 (1.5B/32B) pairs, against vanilla autoregressive decoding, standard SD (with the best fixed window found via sweep), lookahead decoding, assist generation, and REST. Representative speedups:

Benchmark	Model pair	Best baseline	Pacer
HumanEval	DeepSeek 1.3B/33B	2.07× (SD)	2.31×
HumanEval	Llama-2 7B/70B	2.33× (SD)	2.66×
MBPP	Llama-2 7B/70B	2.01× (SD)	2.14×
GSM8K	DeepSeek 1.3B/33B	1.92× (SD)	2.17×
CNN/DM	Llama-2 7B/70B	1.70× (SD)	1.71×

The gains are smallest on CNN/DM, where average acceptance lengths are inherently short (~1.8–1.9); this is consistent with the paper's later admission that Pacer's benefit scales with draft quality. Against other adaptive-length methods on HumanEval with Llama-2 7B/70B, Pacer reaches 24.20 tokens/s with average acceptance length $2.66\times$9, compared to AdaEDL (22.53 tokens/s, $3.09\times$0) and SpecDec++ (21.17 tokens/s, $3.09\times$1). On SpecBench across six task categories, Pacer attains the highest average speed (17.95 tokens/s).

Compatibility with draft-quality methods

Because Pacer modulates draft length rather than draft content, it composes with methods that improve drafting itself. Integrated with Ouroboros (phrase-level n-gram drafting) on DeepSeek-Coder 1.3B/33B, speedup rises from 2.71× to 2.82× on HumanEval, from 2.74× to 2.86× on MBPP, and from 2.89× to 3.09× on GSM8K, with acceptance lengths increasing correspondingly (e.g., 8.36 → 10.90 on HumanEval). This orthogonality claim is supported only for Ouroboros; combinations with EAGLE-style feature-level drafters or Medusa-style self-drafting are not evaluated.

Ablations and design analysis

Three ablations isolate the sources of improvement:

• Position embeddings: removing them drops HumanEval speed from 41.80 to 39.99 tokens/s, confirming the positional signal identified in the observational analysis.
• Growth factor: removing it increases $3.09\times$2 but lowers speed (40.36 vs. 41.80 tokens/s), because the predictor under-estimates acceptance at long positions and drafts become excessively long.
• Halting criterion: the mean-token criterion outperforms stricter any-token (38.96 tokens/s) and last-token (40.11 tokens/s) rules, since averaging suppresses occasional low-confidence predictions that would otherwise halt drafting prematurely.
• Attention scope: full-context attention in $3.09\times$3 yields 24.20 tokens/s versus ~22.3–22.6 for local-block or local-draft attention, indicating that prefix context matters for reliable acceptance prediction.

Hyperparameter sweeps show moderate sensitivity: performance peaks at $3.09\times$4, $3.09\times$5, and $3.09\times$6 on HumanEval with DeepSeek-Coder 1.3B/33B, degrading by roughly 10% at the extremes. The runtime breakdown shows the pre-verification layer accounts for only 1.30–2.10% of total inference time, so its overhead is small relative to the savings in draft and target forward passes.

Limitations and open questions

The paper concedes several constraints. Pacer's gains depend on draft quality: tasks where drafts diverge quickly (CNN/DM summarization) see minimal improvement because optimal drafts are short regardless. When both models are small, the fixed overhead of the pre-verification layer becomes proportionally more significant. The hyperparameters $3.09\times$7, $3.09\times$8, and $3.09\times$9 are tuned per task and model pair rather than learned, leaving open whether they can be set automatically or robustly transferred across distributions. The growth factor is explicitly a heuristic patch for a distribution-shift problem in the training data (short drafts dominate); a principled fix, such as curriculum sampling over draft positions, remains unexplored. Finally, the oracle analysis shows a $L_A^\star$0 headroom over the best fixed window, but the paper does not quantify how close Pacer's predictions come to the oracle stopping decisions, so the remaining gap between Pacer and optimal dynamic drafting is not characterized.

Conclusion

Pacer replaces the fixed draft window of speculative decoding with a trainable blockwise pre-verification mechanism that uses draft context and positional information to decide when to stop drafting. It delivers consistent improvements over fixed-window SD and prior adaptive-length heuristics across code generation, arithmetic reasoning, and summarization benchmarks, with negligible overhead and demonstrated composability with draft-quality enhancements. Its effectiveness is bounded by draft quality and requires per-task hyperparameter tuning, questions the paper identifies but does not resolve.