Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding

Abstract: Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models. As the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost. We propose EVICT, a training-free, hyperparameter-free, and lossless adaptive verification method for MoE speculative decoding. EVICT makes every verified token count by truncating the draft tree before target verification and retaining only the cost-effective prefix. It leverages fine-grained drafter signals to estimate candidate benefit, combines them with offline-profiled verification cost, and remains highly compatible with the high-performance graph-based serving framework SGLang. Extensive experiments on diverse MoE backbones and benchmarks show that EVICT achieves up to 2.35x speedup over autoregressive decoding and an average 1.21x speedup over the state-of-the-art baseline EAGLE-3, while significantly reducing unnecessary expert activations during verification.

• The paper introduces EVICT, an adaptive, lossless verification approach that selects optimal draft tree prefixes based on a cost-benefit ratio.
• EVICT achieves up to 2.35× speedup by reducing unnecessary expert activations, trading a slight drop in accepted tokens for overall throughput gains.
• The method integrates with CUDA graph fusion to eliminate CPU bottlenecks, ensuring efficient deployment in sparse MoE models.

Adaptive Verification for MoE Speculative Decoding: An Expert Analysis of EVICT

Motivation and Challenges in MoE Speculative Decoding

Sparse Mixture-of-Experts (MoE) models are increasingly dominant for scaling LLMs, balancing parameter capacity with manageable per-token compute. Speculative decoding, including tree-based strategies, has provided substantial reductions in autoregressive inference latency, especially when integrated with high-performance inference systems. However, unique cost characteristics arise for MoE models. Each token only activates a subset of experts, but parallel tree-based verification aggregates all experts required by the tokens under verification, causing a rapid increase in the number of activated experts and inflating latency, particularly in low-latency and small-batch inference regimes.

Figure 1: Average activated experts and per-iteration latency versus tree size of Qwen3-30B-A3B, averaged across a full inference run. Larger trees activate more experts, thus incur higher iteration latency.

Figure 2: Per-iteration latency breakdown averaged across a full inference run. In all cases, Target Verify dominates the latency budget, highlighting target verification as the main bottleneck.

These phenomena, depicted in Figures 1 and 2, highlight the growing union of activated experts with draft tree size and the dominance of verification latency in end-to-end performance. Conventional tree-based speculative decoding approaches, which maximize draft acceptance, are suboptimal for MoE serving due to this verification overhead and expert inflation.

The EVICT Method: Adaptive, Cost-Aware Verification

EVICT (Expert-aware Verification via Identifying Cost-effective Tree Prefixes) addresses the crucial need to adapt verification not simply to acceptance likelihood, but to a cost-aware measure that explicitly considers expert activation overhead. EVICT introduces a lossless, training-free, and hyperparameter-free approach that selects, at each speculative step, the prefix of the draft tree that maximizes the utility measured as the ratio of expected accepted tokens to verification cost.

The method proceeds as follows:

1. Accepted Length Estimation: For each candidate node in the draft tree, EVICT computes a pre-verification score via the product of drafter probabilities along its path, using the well-calibrated outputs of the drafter model.
2. Utility-Guided Truncation: Rather than verifying a fixed number of nodes, EVICT computes, for each prefix length $k$, the ratio of expected total accepted tokens over empirically profiled verification cost, and selects the $k$ maximizing this ratio.
3. Offline Cost Profiling: EVICT profiles the cost of verifying each possible prefix offline, storing latencies for lookup, thus reducing online decision latency.
4. Framework Co-design: The method is implemented in SGLang with graph-captured kernels for each prefix size to avoid CPU-GPU synchronization or custom kernels, enabling full compatibility with CUDA graphs.

   Figure 3: Overview of tree-based speculative decoding, including draft-tree construction, target verification, and tree-based sampling.

   

   Figure 4: (a) EVICT selects the verification budget by balancing the estimated accepted length against the profiled verification cost, and chooses the utility-maximizing prefix size. (b) Given the selected budget, EVICT truncates the draft tree into an ancestor-closed cost-effective prefix, while EAGLE-3 verifies additional low-utility nodes under a fixed tree budget. (c) The retained prefix is verified in a single target-model forward pass using a tree attention mask. (d) By verifying fewer draft nodes, EVICT reduces the union of activated experts in the MoE target model, thereby lowering target-side verification cost.

This architecture is strictly lossless: the distribution of outputs is unchanged relative to vanilla target-model sampling. Unlike prior work (e.g., EAGLE-3, OPT-Tree), EVICT does not require hand-tuned hyperparameters or quality/efficiency trade-offs; all decisions are contextually determined by cost-benefit tradeoffs.

Numerical Results and Empirical Claims

Extensive experiments across diverse state-of-the-art MoE backbones (e.g., Qwen3-30B-A3B) and standard benchmarks (MT-Bench, Alpaca, GSM8K, HumanEval, QA, CNN/DM) establish several claims:

• EVICT obtains up to 2.35× decoding speedup over autoregressive decoding and an average of 1.21× over EAGLE-3.
• Significant reduction in unnecessary expert activations during verification: For instance, adaptive verification yields up to 32.5% fewer unique expert activations per iteration.
• Pronounced benefits in mathematically or structurally predictable benchmarks (e.g., GSM8K, HumanEval), where acceptance is high and verification efficiency is critical.

  Figure 5: Adaptive verification significantly reduces MoE verification cost. The x-axis denotes the mean accepted tokens (Mat). By dynamically truncating low-value draft tokens across diverse contexts (a), EVICT substantially decreases the number of activated experts (b), which directly translates to lower target-model verification latency (c) compared to EAGLE-3's fixed-size trees.

These gains are achieved with a slight reduction in mean accepted tokens (Mat) per speculative step, highlighting that unconstrained acceptance pursuit is not optimal for MoE: EVICT intentionally trades a modest drop in Mat for substantial verification cost reduction, yielding superior overall throughput.

System Aspects and Ablations

EVICT's system-level integration is crucial for realizing its theoretical gains. In SGLang, EVICT implements graph-captured verification for possible prefix lengths, seamlessly fusing runtime operations with draft and target graphs. This eliminates CPU bottlenecks associated with dynamic truncation.

Figure 6: Decoding speed (token/s) of EVICT and EAGLE-3 for Qwen3-30B-A3B under different inference frameworks.

As detailed in Figure 6, relying solely on PyTorch or eager execution, the increased control-flow logic of adaptive truncation introduces CPU bottlenecks, negating GPU-side gains. Only with CUDA graph fusion does EVICT outperform EAGLE-3 consistently, establishing that methodological advances must be co-designed with high-efficiency serving stacks for deployment value.

Ablation studies demonstrate that cost-agnostic truncation (e.g., DDD, score-coverage thresholds) consistently underperform EVICT and are sensitive to thresholds, requiring non-trivial tuning, unlike the utility-maximizing, hyperparameter-free nature of EVICT's strategy.

Figure 7: Decoding speed (token/s) of EVICT, DDD, EAGLE-3, and score-coverage variants with $\rho \in \{0.7, 0.4\}$.

Implications and Future Directions

Practically, EVICT demonstrates that aggressive speculative drafting is suboptimal in sparse MoE inference; selective, cost-aware verification unlocks end-to-end acceleration without altering model outputs or requiring retraining or fine-tuning. Theoretical implications include an explicit decoupling of acceptance maximization from efficiency maximization in expert-sparse architectures, suggesting future inference acceleration must internalize cost heterogeneity.

For future developments, this methodology can be extended to other conditional computation models, multi-expert agent settings, or elastic expert architectures—where cost dynamics are even more pronounced. Further, the co-design principles (profiling, kernel fusion, offline adaptation) are fundamental for bridging methodological and systems advances in deployment-scale LLM inference.

Conclusion

EVICT offers a principled, adaptive, and fully lossless approach to speculative decoding in MoE LLMs, achieving tangible speedups by selectively verifying cost-effective prefixes and leveraging detailed cost profiles. By shifting the optimization target from pure acceptance to benefit-cost maximization and integrating with high-performance inference frameworks, EVICT establishes a new state of the art for efficient MoE inference acceleration while maintaining strict output fidelity (2605.00342).