EAGLE Speculative Decoding

• The paper presents EAGLE-based speculative decoding, demonstrating a 2.7×–3.5× speedup by using target-conditioned lightweight drafting with lossless batched verification.
• Tree drafting and dynamic draft depth techniques, such as in DDD, AdaEAGLE, and TALON, optimize resource allocation and improve acceptance lengths.
• Systems extensions like SpecBlock, HVD, and TriSpec address verification bottlenecks and integrate cost-effective mechanisms for practical deployment.

Searching arXiv for the core EAGLE speculative decoding papers and recent extensions. EAGLE-based speculative decoding denotes a family of lossless inference acceleration methods for LLMs in which a lightweight drafter proposes multiple candidate continuations and a larger target model verifies them in parallel, while preserving the target model’s output distribution. The line originates with EAGLE, which frames drafting as feature-level autoregression and addresses feature uncertainty by conditioning on a token sequence advanced by one time step (Li et al., 2024). Subsequent variants extend this core idea along several axes: dynamic tree construction, adaptive draft depth or length, alternative drafter architectures, parallel multi-token drafting, verification-cost reduction, vocabulary and systems optimization, and deployment on heterogeneous serving stacks. Across this literature, EAGLE-based methods are characterized by target-conditioned drafting, batched verification, and a recurring design tension between accepted length, drafter cost, and verification cost (Li et al., 2024, Zhang et al., 2024, Pan et al., 1 May 2026).

1. Origins and defining formulation

EAGLE, introduced in “Speculative Sampling Requires Rethinking Feature Uncertainty,” departs from token-level drafting by arguing that autoregression at the feature (second-to-top-layer) level is more straightforward than at the token level, while also identifying inherent uncertainty in feature-level autoregression as a limiting factor (Li et al., 2024). Its central mechanism is to predict the next feature using prior features together with a token sequence advanced by one time step, thereby resolving ambiguity induced by token sampling. In the paper’s notation, the draft model takes feature history and shifted tokens as input:

$f_{j+1} = \text{AutoReg}( F_{1:j}, T_{2:j+1} )$

and then produces next-token probabilities through the LM head (Li et al., 2024).

The original EAGLE training objective combines feature regression and next-token classification:

$$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$

with a SmoothL1 term on predicted features and a cross-entropy term on next-token prediction (Li et al., 2024). Verification follows the standard speculative sampling acceptance rule, with acceptance probability

$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$

so the final output distribution remains unchanged (Li et al., 2024).

Empirically, the original EAGLE reports a latency speedup ratio of $2.7\times$–$3.5\times$ for LLaMA2-Chat 70B, doubled throughput, and maintained distribution of the generated text (Li et al., 2024). It also reports that tree attention improves average acceptance lengths, and that EAGLE is compatible with other accelerations such as quantization and compilation (Li et al., 2024). These results established the two defining commitments of the EAGLE line: target-aligned lightweight drafting and lossless batched verification.

2. Tree drafting and the transition from static to adaptive structures

A major development in EAGLE-based speculative decoding is the move from chain drafts to tree-structured drafts. The provided literature describes EAGLE as using a tree draft structure with a draft model derived from the target model, where candidate tokens are preselected in a fixed fashion and verified using modified attention masks (Brown et al., 2024). EAGLE-2 then introduces a dynamic tree structure by applying beam search over draft model outputs, selecting the top $k$ token sequences based on a heuristic such as the sum of logprobs, while still keeping beam width and sequence depth fixed across inference sequences (Brown et al., 2024).

Dynamic Depth Decoding (DDD) is presented specifically as an optimization of EAGLE-2’s tree drafting method through dynamic depth control (Brown et al., 2024). Its heuristic is

$H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$

evaluated at selected steps; if $H < x$, drafting stops early (Brown et al., 2024). The paper states that DDD extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16\times$ (Brown et al., 2024). It also reports that on the hardware studied there is a $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$0 improvement over EAGLE-2, and a $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$1 improvement in a “strict” mode that breaks lazy evaluation at each draft call (Brown et al., 2024).

AdaEAGLE pushes adaptivity further by explicitly modeling adaptive draft structures with a Lightweight Draft Length Predictor (LDLP), described as the first speculative decoding framework that explicitly models adaptive draft structures (Zhang et al., 2024). Rather than relying on manual thresholds, LDLP predicts the optimal number of draft tokens during inference from the embedding and last hidden state of the last validated token:

$$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$2

with a penalized regression loss that penalizes under-prediction more heavily (Zhang et al., 2024). The paper reports that AdaEAGLE achieves comparable speedup results without manual thresholds, yields $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$3 speedup over vanilla autoregressive decoding on average, and, together with threshold-based strategies, achieves $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$4 speedup over vanilla autoregressive decoding while maintaining output quality (Zhang et al., 2024).

A related training-free line is TALON, a budget-driven adaptive tree expansion framework that can be plugged into existing tree-based methods (Liu et al., 12 Jan 2026). TALON constructs the draft tree iteratively under a fixed node budget, using Top-$$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$5 initialization at the root and confidence-gated expansion at deeper layers. For a current layer candidate set $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$6, it uses

$$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$7

to retain nodes before budget truncation (Liu et al., 12 Jan 2026). The paper states that TALON consistently outperforms state-of-the-art EAGLE-3, achieving up to $$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$8 end-to-end speedup over auto-regressive decoding (Liu et al., 12 Jan 2026).

3. Drafter architectures and training regimes

Although the original EAGLE uses a lightweight autoregression head over target features, later work broadens the drafter design space. One prominent direction is architectural simplification. “Cross-Attention Speculative Decoding” introduces Budget EAGLE (Beagle), described as the first cross-attention-based Transformer decoder speculative decoding model that achieves performance on par with leading self-attention speculative decoding models (EAGLE-v2) while eliminating the need for pooling or auxiliary components (Zhong et al., 30 May 2025). Its draft block is defined by

$$\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}$$9

with a Two-Stage Block-Attention Training procedure that combines multi-token early-stage learning with later autoregressive simulation (Zhong et al., 30 May 2025). The paper reports that Beagle matches or slightly exceeds EAGLE-v2 on major benchmarks, improves training efficiency, and uses less peak memory, while EAGLE-v2 uses $$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$0–$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$1 more GPU memory (Zhong et al., 30 May 2025).

Another direction is parallel drafting. P-EAGLE transforms EAGLE from autoregressive to parallel multi-token prediction via a learnable shared hidden state and a learnable mask token embedding, enabling $$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$2 tokens to be drafted in a single forward pass (Hui et al., 1 Feb 2026). It argues that positional information is recoverable via self-attention with RoPE, so explicit depth-specific hidden states are unnecessary, and formalizes the substitution for multi-token prediction positions as

$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$3

To make long-context training practical, P-EAGLE introduces attention mask pre-computation, sequence partitioning, and Conditional Drop-token sampling with total active positions

$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$4

The paper reports speedups of $$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$5–$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$6 over autoregressive EAGLE-3 across GPT-OSS 120B, 20B, and Qwen3-Coder 30B, and states that it is implemented in vLLM (Hui et al., 1 Feb 2026).

SpecBlock addresses a different weakness: the non-trivial share of per-iteration latency incurred by repeated drafter calls in autoregressive drafters such as EAGLE-3 (Shi et al., 8 May 2026). Each SpecBlock forward produces $$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$7 dependent positions, and the draft tree grows through repeated block expansions. Within-block dependence is carried by a layer-wise shift,

$$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$8

while a co-trained rank head allocates per-position branching and a valid-prefix mask drops loss on later positions once an earlier one is wrong (Shi et al., 8 May 2026). The paper reports that SpecBlock improves mean speedup by $$\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),$$9–$2.7\times$0 over EAGLE-3 at $2.7\times$1–$2.7\times$2 of its drafting cost, and that cost-aware adaptation extends this lead to $2.7\times$3–$2.7\times$4 (Shi et al., 8 May 2026).

The training infrastructure around EAGLE-3 has also become a research topic in its own right. SpecForge is an open-source framework with full support for EAGLE-3, including target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines (Li et al., 19 Mar 2026). It reports up to $2.7\times$5 faster EAGLE-3 training for Qwen3-235B-A22B and releases SpecBundle, a suite of production-grade EAGLE-3 draft models achieving up to $2.7\times$6 end-to-end inference speedup on SGLang (Li et al., 19 Mar 2026). The paper also states that EAGLE-3 has become the “de facto industrial standard” for speculative decoding (Li et al., 19 Mar 2026).

4. Verification bottlenecks, MoE-specific issues, and adaptive allocation

As EAGLE-based methods improved drafting, several papers shifted attention to verification cost. This shift is especially visible for sparse Mixture-of-Experts models. “Making Every Verified Token Count” studies tree-based speculative decoding for MoE models and identifies a specific failure mode: as the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost (Pan et al., 1 May 2026). The paper describes EAGLE-3 as always verifying a fixed-size set of draft nodes, usually $2.7\times$7, regardless of context difficulty or diminishing returns (Pan et al., 1 May 2026).

EVICT addresses this by selecting the cost-effective prefix of the draft tree instead of maximizing accepted draft tokens alone (Pan et al., 1 May 2026). For a node $2.7\times$8, it defines

$2.7\times$9

and the estimated accepted length

$3.5\times$0

With offline-profiled verification latency $3.5\times$1, EVICT chooses

$3.5\times$2

and truncates the tree to the top-$3.5\times$3 ancestor-closed prefix (Pan et al., 1 May 2026). The paper reports up to $3.5\times$4 speedup over autoregressive decoding and an average $3.5\times$5 speedup over EAGLE-3, while reducing the average number of unique experts loaded per target step by about $3.5\times$6 and reducing target verification latency by about $3.5\times$7 (Pan et al., 1 May 2026).

TriSpec generalizes the focus on verification cost by introducing a lightweight proxy verifier between drafter and target, yielding a ternary system with drafter $3.5\times$8, proxy verifier $3.5\times$9, and target $k$0 (Jiang et al., 30 Jan 2026). It defines proxy-side acceptance indicators

$k$1

and a margin-based routing predicate

$k$2

The latency decomposition is

$k$3

and TriSpec reduces $k$4 by shifting many verification decisions to the proxy (Jiang et al., 30 Jan 2026). The paper states that TriSpec can be integrated with EAGLE-3, achieves up to $k$5 speedup over standard speculative decoding, and reduces target model invocations by up to $k$6 while maintaining comparable accuracy (Jiang et al., 30 Jan 2026).

Hybrid Verified Decoding (HVD) addresses a different allocation problem: whether to verify a parameter-free cache draft or fall back to a model-based drafter such as EAGLE3 (Su et al., 31 May 2026). It trains a payoff predictor $k$7 to estimate accepted length for a cache draft from runtime features $k$8, then uses

$k$9

with $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$0 (Su et al., 31 May 2026). The paper states that across three LLMs and sixteen datasets, HVD outperforms EAGLE3 in every setting on agentic workflows with a $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$1 average speedup (Su et al., 31 May 2026). A plausible implication is that EAGLE-based decoding is increasingly being treated as one component in a broader verification-allocation policy rather than as a complete end-to-end solution.

5. Systems engineering, serving frameworks, and deployment constraints

A recurrent theme in EAGLE-based speculative decoding is that algorithmic gains depend heavily on systems realization. Several papers focus on tree attention, KV-cache management, graph capture, and serving integration.

EVICT emphasizes compatibility with the high-performance graph-based serving framework SGLang by pre-capturing all required verification graphs during initialization and fusing adaptive logic into draft graph capture (Pan et al., 1 May 2026). SpecForge likewise integrates training and inference with production engines such as SGLang and vLLM, using target-draft decoupling so that the draft can be trained with ZeRO-style sharding while the target runs with tensor or expert parallelism and optimized inference kernels (Li et al., 19 Mar 2026). It reports that optimized tree attention via BlockMasks reduces memory usage by more than $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$2 and speeds execution by more than $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$3 compared to a naive SDPA/softmax implementation, while in-place gradients save an additional $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$4–$H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$5 memory (Li et al., 19 Mar 2026).

Production-scale deployment for Llama models is examined in “Efficient Speculative Decoding for Llama at Scale,” which details training and inference optimization techniques for EAGLE-based speculative decoding on GPU clusters (Tang et al., 11 Aug 2025). The paper describes an inference workflow with prefill, tree dispatcher, drafting, validation, multi-round sampling, and bookkeeping, together with a prefix/suffix split for tree attention, PyTorch 2.x compilation for sampling, CUDA graphs, dynamic tree selection, persistent KV caches, and hardware-adaptive attention kernel selection (Tang et al., 11 Aug 2025). It reports that Llama4 Maverick decodes at about $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$6 ms per token on $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$7 NVIDIA H100 GPUs at batch size one, about $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$8 faster than the previously best known method, and that EAGLE-based speculative decoding achieves speed-up for large batch sizes between $H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),$9 and $H < x$0 at production scale (Tang et al., 11 Aug 2025).

EAGLE-Pangu examines portability to Ascend NPUs, arguing that attention masking, KV-cache layouts, and indexing semantics are not interchangeable across accelerator stacks (Han et al., 9 Mar 2026). It introduces an explicit branch/commit cache manager, accelerator-safe tree tensorization with a dummy-root row that removes undefined negative indices by construction, structural invariant checks, and a fused-kernel-compatible teacher verification path with a debuggable eager fallback (Han et al., 9 Mar 2026). On $H < x$1 turns from MT-Bench and HumanEval-style prompts, it reports $H < x$2 average throughput improvement and up to $H < x$3 at p99 over teacher-only greedy decoding in the fused-kernel path (Han et al., 9 Mar 2026). This line makes explicit that EAGLE-style tree verification is not only a modeling problem but also an indexing and kernel-contract problem.

These systems papers also sharpen a common misconception: tree-structured speculative decoding is not automatically fast once the draft tree is available. The data repeatedly indicate that tree attention, cache alignment, bookkeeping, and validation path implementation can dominate end-to-end behavior at realistic batch sizes or on non-GPU backends (Tang et al., 11 Aug 2025, Han et al., 9 Mar 2026).

6. Comparative extensions, limitations, and adjacent directions

Recent work increasingly treats EAGLE-based speculative decoding as a strong baseline rather than an endpoint. Some papers remain within the EAGLE design space, while others surpass it by changing the drafter, the verifier, or both.

SpecVocab argues that the primary computational bottleneck in state-of-the-art speculative decoding methods using a single decoder layer and output embedding matrix is the output projection or distribution step (Williams et al., 14 Feb 2026). It replaces a fixed reduced vocabulary with a per-step speculative vocabulary $H < x$4, selected through approximate logits and Top-$H < x$5 subset selection, then computes exact logits only on that subset:

$H < x$6

The paper reports that SpecVocab can achieve a higher acceptance length than EAGLE-3 and yields up to an $H < x$7 increase in average throughput over EAGLE-3 (Williams et al., 14 Feb 2026).

HeteroSpec frames EAGLE-3’s weakness as uniform resource allocation across contexts of different linguistic complexity (Liu et al., 19 May 2025). It introduces a cumulative meta-path Top-$H < x$8 entropy metric,

$H < x$9

and uses entropy partitioning to allocate deeper drafting and more aggressive pruning to low-entropy, predictable contexts (Liu et al., 19 May 2025). The paper reports an average speedup of $44\%$0 and states that it consistently outperforms EAGLE-3 across speedup rates, average acceptance length, and verification cost, without draft model retraining (Liu et al., 19 May 2025).

Long-context behavior is challenged directly by OWL, which argues that existing speculative decoding methods fail to generalize to real-world long-context settings and states that EAGLE3 can even slow down generation speed to $44\%$1 (Lee et al., 8 Oct 2025). OWL replaces the transformer drafter with an LSTM-based drafter conditioned only on the last-token hidden state, adds a special $44\%$2 token in the verifier, and combines tree and non-tree decoding in a hybrid algorithm (Lee et al., 8 Oct 2025). On LongSpecBench, it reports acceptance lengths of $44\%$3 and $44\%$4 for Llama-3.1-8B and Llama-3.3-70B respectively, versus $44\%$5 and $44\%$6 for EAGLE3, and HOWL reaches $44\%$7 and $44\%$8 (Lee et al., 8 Oct 2025). This suggests that transformer-based EAGLE drafters trained on short windows inherit a window length-dependence that becomes acute in long-context regimes.

Quantization introduces another limitation. “Speculative Decoding Meets Quantization” studies EAGLE-2 on quantized models and reports that for 4-bit weight quantized models, verifying a tree-style draft incurs significantly more time overhead than a single-token forward pass, reducing the expected synergy between speculative decoding and quantization (Zhang et al., 28 May 2025). The paper proposes a hierarchical framework in which a small intermediate model converts tree-style drafts into sequence drafts before the large quantized target verifies them, achieving $44\%$9 speedup across various tasks for a 4-bit Llama-3-70B on A100 and outperforming EAGLE-2 by $3.16\times$0 (Zhang et al., 28 May 2025). The concrete lesson is that EAGLE-style tree verification can become compute-dominated once memory bandwidth is sufficiently reduced.

Some newer approaches depart more radically from EAGLE-style autoregressive drafting. Double introduces synchronous retrieval speculative parallelism, is training-free and lossless, and reports state-of-the-art speedup of $3.16\times$1 on LLaMA3.3-70B and $3.16\times$2 on Qwen3-32B while significantly outperforming EAGLE-3 (Shen et al., 9 Jan 2026). DDTree extends a block diffusion drafter by constructing a tree from per-position marginals with a best-first heap algorithm, and the paper states that these gains place DDTree among the leading approaches to speculative decoding and that it outperforms strong autoregressive drafters such as EAGLE-3 (Ringel et al., 14 Apr 2026). These comparisons clarify that “EAGLE-based” now refers as much to a methodological lineage and benchmark standard as to the current performance frontier.

7. Conceptual synthesis and research outlook

Across the literature, EAGLE-based speculative decoding evolves from a specific feature-level autoregressive insight into a broad family of drafting-and-verification systems. Three stable principles recur. First, losslessness is preserved through explicit target-side verification (Li et al., 2024, Jiang et al., 30 Jan 2026). Second, speedup depends not only on acceptance length but on the ratio between accepted tokens, drafter cost, and verification cost; several papers formalize this trade-off directly, including TALON’s

$3.16\times$3

and SpecBlock’s

$3.16\times$4

(Liu et al., 12 Jan 2026, Shi et al., 8 May 2026). Third, the dominant bottleneck moves with the setting: drafting in autoregressive drafters, verification in MoE or quantized targets, vocabulary projection in large-vocabulary models, and tree attention or cache semantics in production systems (Pan et al., 1 May 2026, Williams et al., 14 Feb 2026, Tang et al., 11 Aug 2025).

The major controversies or misconceptions are correspondingly narrow rather than foundational. One misconception is that maximizing acceptance length is sufficient; EVICT shows that in MoE models accepted tokens can increase while end-to-end speed worsens because expert-union growth inflates verification cost (Pan et al., 1 May 2026). Another is that speculative decoding improvements transfer uniformly across contexts; OWL and HVD both emphasize that long contexts and agentic workloads change the utility landscape substantially (Lee et al., 8 Oct 2025, Su et al., 31 May 2026). A third is that a single fixed tree shape is adequate; DDD, AdaEAGLE, TALON, and HeteroSpec all argue, by different mechanisms, for adaptive depth, length, or resource allocation (Brown et al., 2024, Zhang et al., 2024, Liu et al., 12 Jan 2026, Liu et al., 19 May 2025).

A plausible synthesis is that the EAGLE line has bifurcated into two research programs. One program refines the original target-conditioned drafter through better architectures, training, and systems co-design, as seen in Beagle, P-EAGLE, SpecBlock, SpecForge, and large-scale production deployments (Zhong et al., 30 May 2025, Hui et al., 1 Feb 2026, Shi et al., 8 May 2026, Li et al., 19 Mar 2026, Tang et al., 11 Aug 2025). The other program treats EAGLE-3 as a strong baseline and reallocates computation around it through adaptive verification, proxy verification, cache-aware hybridization, dynamic vocabularies, or entirely different drafting mechanisms (Pan et al., 1 May 2026, Jiang et al., 30 Jan 2026, Su et al., 31 May 2026, Williams et al., 14 Feb 2026, Shen et al., 9 Jan 2026, Ringel et al., 14 Apr 2026). In both cases, EAGLE-based speculative decoding remains a central reference point for lossless acceleration research because it ties together drafting quality, verification structure, and deployment realism in a single framework.