---
title: EAGLE Speculative Decoding
url: https://www.emergentmind.com/topics/eagle-based-speculative-decoding
type: topic
---

# EAGLE Speculative Decoding

Searching arXiv for the core EAGLE speculative decoding papers and recent extensions.
EAGLE-based speculative decoding denotes a family of lossless inference acceleration methods for large language models in which a lightweight drafter proposes multiple candidate continuations and a larger target model verifies them in parallel, while preserving the target model’s output distribution. The line originates with EAGLE, which frames drafting as feature-level autoregression and addresses feature uncertainty by conditioning on a token sequence advanced by one time step [2401.15077]. Subsequent variants extend this core idea along several axes: dynamic tree construction, adaptive draft depth or length, alternative drafter architectures, parallel multi-token drafting, verification-cost reduction, vocabulary and systems optimization, and deployment on heterogeneous serving stacks. Across this literature, EAGLE-based methods are characterized by target-conditioned drafting, batched verification, and a recurring design tension between accepted length, drafter cost, and verification cost [2401.15077], [2412.18910], [2605.00342].

## 1. Origins and defining formulation

EAGLE, introduced in “Speculative Sampling Requires Rethinking Feature Uncertainty,” departs from token-level drafting by arguing that autoregression at the feature (second-to-top-layer) level is more straightforward than at the token level, while also identifying inherent uncertainty in feature-level autoregression as a limiting factor [2401.15077]. Its central mechanism is to predict the next feature using prior features together with a token sequence advanced by one time step, thereby resolving ambiguity induced by token sampling. In the paper’s notation, the draft model takes feature history and shifted tokens as input:
$$
f_{j+1} = \text{AutoReg}( F_{1:j}, T_{2:j+1} )
$$
and then produces next-token probabilities through the LM head [2401.15077].

The original EAGLE training objective combines feature regression and next-token classification:
$$
\mathcal{L} = \mathcal{L}_{\text{reg}} + w_{cls} \mathcal{L}_{\text{cls}}
$$
with a SmoothL1 term on predicted features and a cross-entropy term on next-token prediction [2401.15077]. Verification follows the standard speculative sampling acceptance rule, with acceptance probability
$$
\min\left(1, \frac{p^{\text{target}}_i(t_i)}{p^{\text{draft}}_i(t_i)}\right),
$$
so the final output distribution remains unchanged [2401.15077].

Empirically, the original EAGLE reports a latency speedup ratio of \(2.7\times\)–\(3.5\times\) for LLaMA2-Chat 70B, doubled throughput, and maintained distribution of the generated text [2401.15077]. It also reports that tree attention improves average acceptance lengths, and that EAGLE is compatible with other accelerations such as quantization and compilation [2401.15077]. These results established the two defining commitments of the EAGLE line: target-aligned lightweight drafting and lossless batched verification.

## 2. Tree drafting and the transition from static to adaptive structures

A major development in EAGLE-based speculative decoding is the move from chain drafts to tree-structured drafts. The provided literature describes EAGLE as using a tree draft structure with a draft model derived from the target model, where candidate tokens are preselected in a fixed fashion and verified using modified attention masks [2409.00142]. EAGLE-2 then introduces a dynamic tree structure by applying beam search over draft model outputs, selecting the top \(k\) token sequences based on a heuristic such as the sum of logprobs, while still keeping beam width and sequence depth fixed across inference sequences [2409.00142].

Dynamic Depth Decoding (DDD) is presented specifically as an optimization of EAGLE-2’s tree drafting method through dynamic depth control [2409.00142]. Its heuristic is
$$
H = \log \left( \sum_{i=0}^{w} \exp(\logprobsum[i]) \right),
$$
evaluated at selected steps; if \(H < x\), drafting stops early [2409.00142]. The paper states that DDD extends the average speedup that EAGLE-2 achieves over EAGLE by \(44\%\), giving DDD an average speedup of \(3.16\times\) [2409.00142]. It also reports that on the hardware studied there is a \(4\%\) improvement over EAGLE-2, and a \(5\%\) improvement in a “strict” mode that breaks lazy evaluation at each draft call [2409.00142].

AdaEAGLE pushes adaptivity further by explicitly modeling adaptive draft structures with a Lightweight Draft Length Predictor (LDLP), described as the first speculative decoding framework that explicitly models adaptive draft structures [2412.18910]. Rather than relying on manual thresholds, LDLP predicts the optimal number of draft tokens during inference from the embedding and last hidden state of the last validated token:
$$
\bar{k}_{r+1} = \text{Round}\left(\mathrm{MLP}(e_{j+k^\circ_r}, f_{j+k^\circ_r})\right), \quad
k_{r+1} = \min\left\{k_\text{max}, \max\{0, \bar{k}_{r+1}\}\right\}
$$
with a penalized regression loss that penalizes under-prediction more heavily [2412.18910]. The paper reports that AdaEAGLE achieves comparable speedup results without manual thresholds, yields \(1.61\times\) speedup over vanilla autoregressive decoding on average, and, together with threshold-based strategies, achieves \(1.62\times\) speedup over vanilla autoregressive decoding while maintaining output quality [2412.18910].

A related training-free line is TALON, a budget-driven adaptive tree expansion framework that can be plugged into existing tree-based methods [2601.07353]. TALON constructs the draft tree iteratively under a fixed node budget, using Top-\(K\) initialization at the root and confidence-gated expansion at deeper layers. For a current layer candidate set \(\mathcal{S}_d\), it uses
$$
m_d = \max_{u \in \mathcal{S}_d} p(u), \qquad
\mathcal{P}_{d+1} = \{u \in \mathcal{S}_d \mid p(u) \ge \mu \cdot m_d\}
$$
to retain nodes before budget truncation [2601.07353]. The paper states that TALON consistently outperforms state-of-the-art EAGLE-3, achieving up to \(5.16\times\) end-to-end speedup over auto-regressive decoding [2601.07353].

## 3. Drafter architectures and training regimes

Although the original EAGLE uses a lightweight autoregression head over target features, later work broadens the drafter design space. One prominent direction is architectural simplification. “Cross-Attention Speculative Decoding” introduces Budget EAGLE (Beagle), described as the first cross-attention-based Transformer decoder speculative decoding model that achieves performance on par with leading self-attention speculative decoding models (EAGLE-v2) while eliminating the need for pooling or auxiliary components [2505.24544]. Its draft block is defined by
$$
\mathbf{y}_n = \operatorname{CrossAttn}(\mathbf{h}_{1:n-1}, \mathbf{e}(t_n)) + \mathbf{e}(t_n), \qquad
\mathbf{h}_n = \operatorname{MLP}(\mathbf{y}_n) + \mathbf{y}_n
$$
with a Two-Stage Block-Attention Training procedure that combines multi-token early-stage learning with later autoregressive simulation [2505.24544]. The paper reports that Beagle matches or slightly exceeds EAGLE-v2 on major benchmarks, improves training efficiency, and uses less peak memory, while EAGLE-v2 uses \(10\)–\(15\%\) more GPU memory [2505.24544].

Another direction is parallel drafting. P-EAGLE transforms EAGLE from autoregressive to parallel multi-token prediction via a learnable shared hidden state and a learnable mask token embedding, enabling \(K\) tokens to be drafted in a single forward pass [2602.01469]. It argues that positional information is recoverable via self-attention with RoPE, so explicit depth-specific hidden states are unnecessary, and formalizes the substitution for multi-token prediction positions as
$$
h_{\mathrm{MTP}} = h_{\text{shared}}.
$$
To make long-context training practical, P-EAGLE introduces attention mask pre-computation, sequence partitioning, and Conditional Drop-token sampling with total active positions
$$
n \cdot \sum_{d=0}^{K-1} r^d = n \cdot \frac{1-r^K}{1-r}.
$$
The paper reports speedups of \(1.10\)–\(1.36\times\) over autoregressive EAGLE-3 across GPT-OSS 120B, 20B, and Qwen3-Coder 30B, and states that it is implemented in vLLM [2602.01469].

SpecBlock addresses a different weakness: the non-trivial share of per-iteration latency incurred by repeated drafter calls in autoregressive drafters such as EAGLE-3 [2605.07243]. Each SpecBlock forward produces \(K\) dependent positions, and the draft tree grows through repeated block expansions. Within-block dependence is carried by a layer-wise shift,
$$
\tilde{h}^{(\ell)}_{t,k} = W^{(\ell)}_{\text{shift}} [h^{(\ell)}_{t,k},\, h^{(\ell)}_{t,k{-}1}],
$$
while a co-trained rank head allocates per-position branching and a valid-prefix mask drops loss on later positions once an earlier one is wrong [2605.07243]. The paper reports that SpecBlock improves mean speedup by \(8\)–\(13\%\) over EAGLE-3 at \(44\)–\(52\%\) of its drafting cost, and that cost-aware adaptation extends this lead to \(11\)–\(19\%\) [2605.07243].

The training infrastructure around EAGLE-3 has also become a research topic in its own right. SpecForge is an open-source framework with full support for EAGLE-3, including target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines [2603.18567]. It reports up to \(9.9\times\) faster EAGLE-3 training for Qwen3-235B-A22B and releases SpecBundle, a suite of production-grade EAGLE-3 draft models achieving up to \(4.48\times\) end-to-end inference speedup on SGLang [2603.18567]. The paper also states that EAGLE-3 has become the “de facto industrial standard” for speculative decoding [2603.18567].

## 4. Verification bottlenecks, MoE-specific issues, and adaptive allocation

As EAGLE-based methods improved drafting, several papers shifted attention to verification cost. This shift is especially visible for sparse Mixture-of-Experts models. “Making Every Verified Token Count” studies tree-based speculative decoding for MoE models and identifies a specific failure mode: as the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost [2605.00342]. The paper describes EAGLE-3 as always verifying a fixed-size set of draft nodes, usually \(32\), regardless of context difficulty or diminishing returns [2605.00342].

EVICT addresses this by selecting the cost-effective prefix of the draft tree instead of maximizing accepted draft tokens alone [2605.00342]. For a node \(v\), it defines
$$
\operatorname{Score}(v) = \prod_{u \in \text{Path(root}, v)} q_{t+1}^{\mathcal{T}(u)}
$$
and the estimated accepted length
$$
\widehat{\mathbb{E}[A(\mathcal{T})]} = \sum_{v \in \mathcal{T}} \operatorname{Score}(v).
$$
With offline-profiled verification latency \(C(k)\), EVICT chooses
$$
k^* = \arg\max_{1 \leq k \leq \text{draft\_tokens}} \frac{\widehat{\mathbb{E}[A(\mathcal{T}_k)]}}{C(k)}
$$
and truncates the tree to the top-\(k^*\) ancestor-closed prefix [2605.00342]. The paper reports up to \(2.35\times\) speedup over autoregressive decoding and an average \(1.21\times\) speedup over EAGLE-3, while reducing the average number of unique experts loaded per target step by about \(32.5\%\) and reducing target verification latency by about \(26.6\%\) [2605.00342].

TriSpec generalizes the focus on verification cost by introducing a lightweight proxy verifier between drafter and target, yielding a ternary system with drafter \(\mathcal{M}_d\), proxy verifier \(\mathcal{M}_p\), and target \(\mathcal{M}_t\) [2601.23180]. It defines proxy-side acceptance indicators
$$
s_i \sim \mathrm{Bernoulli}\left(\min\Big\{1,\frac{\mathbf{p}^{(p)}_i(x^{(d)}_i)}{\mathbf{p}^{(d)}_i(x^{(d)}_i)}\Big\}\right)
$$
and a margin-based routing predicate
$$
g(\mathbf{p}^{(p)}_i) = \mathbf{1}\Big\{\operatorname{top}_1(\mathbf{p}^{(p)}_i) - \operatorname{top}_2(\mathbf{p}^{(p)}_i) \ge \lambda\Big\}.
$$
The latency decomposition is
$$
L = \frac{N}{\tau}(t_d + t_v + t_o),
$$
and TriSpec reduces \(t_v\) by shifting many verification decisions to the proxy [2601.23180]. The paper states that TriSpec can be integrated with EAGLE-3, achieves up to \(35\%\) speedup over standard speculative decoding, and reduces target model invocations by up to \(50\%\) while maintaining comparable accuracy [2601.23180].

Hybrid Verified Decoding (HVD) addresses a different allocation problem: whether to verify a parameter-free cache draft or fall back to a model-based drafter such as EAGLE3 [2606.01019]. It trains a payoff predictor \(g_\theta\) to estimate accepted length for a cache draft from runtime features \(\phi_t\), then uses
$$
d_t =
\begin{cases}
d_t^c, & \text{if } \hat{y}_t \ge \tau \\
D_m(x_{1:t}), & \text{otherwise}
\end{cases}
$$
with \(\hat{y}_t = g_\theta(\phi_t)\) [2606.01019]. The paper states that across three LLMs and sixteen datasets, HVD outperforms EAGLE3 in every setting on agentic workflows with a \(2.73\times\) average speedup [2606.01019]. A plausible implication is that EAGLE-based decoding is increasingly being treated as one component in a broader verification-allocation policy rather than as a complete end-to-end solution.

## 5. Systems engineering, serving frameworks, and deployment constraints

A recurrent theme in EAGLE-based speculative decoding is that algorithmic gains depend heavily on systems realization. Several papers focus on tree attention, KV-cache management, graph capture, and serving integration.

EVICT emphasizes compatibility with the high-performance graph-based serving framework SGLang by pre-capturing all required verification graphs during initialization and fusing adaptive logic into draft graph capture [2605.00342]. SpecForge likewise integrates training and inference with production engines such as SGLang and vLLM, using target-draft decoupling so that the draft can be trained with ZeRO-style sharding while the target runs with tensor or expert parallelism and optimized inference kernels [2603.18567]. It reports that optimized tree attention via BlockMasks reduces memory usage by more than \(93\%\) and speeds execution by more than \(60\%\) compared to a naive SDPA/softmax implementation, while in-place gradients save an additional \(30\)–\(40\%\) memory [2603.18567].

Production-scale deployment for Llama models is examined in “Efficient Speculative Decoding for Llama at Scale,” which details training and inference optimization techniques for EAGLE-based speculative decoding on GPU clusters [2508.08192]. The paper describes an inference workflow with prefill, tree dispatcher, drafting, validation, multi-round sampling, and bookkeeping, together with a prefix/suffix split for tree attention, PyTorch 2.x compilation for sampling, CUDA graphs, dynamic tree selection, persistent KV caches, and hardware-adaptive attention kernel selection [2508.08192]. It reports that Llama4 Maverick decodes at about \(4\) ms per token on \(8\) NVIDIA H100 GPUs at batch size one, about \(10\%\) faster than the previously best known method, and that EAGLE-based speculative decoding achieves speed-up for large batch sizes between \(1.4\times\) and \(2.0\times\) at production scale [2508.08192].

EAGLE-Pangu examines portability to Ascend NPUs, arguing that attention masking, KV-cache layouts, and indexing semantics are not interchangeable across accelerator stacks [2603.08088]. It introduces an explicit branch/commit cache manager, accelerator-safe tree tensorization with a dummy-root row that removes undefined negative indices by construction, structural invariant checks, and a fused-kernel-compatible teacher verification path with a debuggable eager fallback [2603.08088]. On \(240\) turns from MT-Bench and HumanEval-style prompts, it reports \(1.27\times\) average throughput improvement and up to \(2.46\times\) at p99 over teacher-only greedy decoding in the fused-kernel path [2603.08088]. This line makes explicit that EAGLE-style tree verification is not only a modeling problem but also an indexing and kernel-contract problem.

These systems papers also sharpen a common misconception: tree-structured speculative decoding is not automatically fast once the draft tree is available. The data repeatedly indicate that tree attention, cache alignment, bookkeeping, and validation path implementation can dominate end-to-end behavior at realistic batch sizes or on non-GPU backends [2508.08192], [2603.08088].

## 6. Comparative extensions, limitations, and adjacent directions

Recent work increasingly treats EAGLE-based speculative decoding as a strong baseline rather than an endpoint. Some papers remain within the EAGLE design space, while others surpass it by changing the drafter, the verifier, or both.

SpecVocab argues that the primary computational bottleneck in state-of-the-art speculative decoding methods using a single decoder layer and output embedding matrix is the output projection or distribution step [2602.13836]. It replaces a fixed reduced vocabulary with a per-step speculative vocabulary \(\mathcal{K}_t\), selected through approximate logits and Top-\(k\) subset selection, then computes exact logits only on that subset:
$$
\mathbf{k}_t = \text{top-k}(\mathbf{s}_t, k), \qquad
\mathbf{z}'_t = \mathbf{U}'_t \mathbf{h}_t.
$$
The paper reports that SpecVocab can achieve a higher acceptance length than EAGLE-3 and yields up to an \(8.1\%\) increase in average throughput over EAGLE-3 [2602.13836].

HeteroSpec frames EAGLE-3’s weakness as uniform resource allocation across contexts of different linguistic complexity [2505.13254]. It introduces a cumulative meta-path Top-\(K\) entropy metric,
$$
H_{\mathrm{path}^{(\mathrm{Top}-K)}(\mathcal{P})}
= -\sum_{t=1}^{T} \sum_{i=1}^{K} \tilde{p}_{t,i} \log \tilde{p}_{t,i},
$$
and uses entropy partitioning to allocate deeper drafting and more aggressive pruning to low-entropy, predictable contexts [2505.13254]. The paper reports an average speedup of \(4.26\times\) and states that it consistently outperforms EAGLE-3 across speedup rates, average acceptance length, and verification cost, without draft model retraining [2505.13254].

Long-context behavior is challenged directly by OWL, which argues that existing speculative decoding methods fail to generalize to real-world long-context settings and states that EAGLE3 can even slow down generation speed to \(0.81\times\) [2510.07535]. OWL replaces the transformer drafter with an LSTM-based drafter conditioned only on the last-token hidden state, adds a special \([\mathrm{SPEC}]\) token in the verifier, and combines tree and non-tree decoding in a hybrid algorithm [2510.07535]. On LongSpecBench, it reports acceptance lengths of \(4.00\) and \(4.27\) for Llama-3.1-8B and Llama-3.3-70B respectively, versus \(1.28\) and \(1.35\) for EAGLE3, and HOWL reaches \(6.14\) and \(5.31\) [2510.07535]. This suggests that transformer-based EAGLE drafters trained on short windows inherit a window length-dependence that becomes acute in long-context regimes.

Quantization introduces another limitation. “Speculative Decoding Meets Quantization” studies EAGLE-2 on quantized models and reports that for 4-bit weight quantized models, verifying a tree-style draft incurs significantly more time overhead than a single-token forward pass, reducing the expected synergy between speculative decoding and quantization [2505.22179]. The paper proposes a hierarchical framework in which a small intermediate model converts tree-style drafts into sequence drafts before the large quantized target verifies them, achieving \(2.78\times\) speedup across various tasks for a 4-bit Llama-3-70B on A100 and outperforming EAGLE-2 by \(1.31\times\) [2505.22179]. The concrete lesson is that EAGLE-style tree verification can become compute-dominated once memory bandwidth is sufficiently reduced.

Some newer approaches depart more radically from EAGLE-style autoregressive drafting. Double introduces synchronous retrieval speculative parallelism, is training-free and lossless, and reports state-of-the-art speedup of \(5.3\times\) on LLaMA3.3-70B and \(2.8\times\) on Qwen3-32B while significantly outperforming EAGLE-3 [2601.05524]. DDTree extends a block diffusion drafter by constructing a tree from per-position marginals with a best-first heap algorithm, and the paper states that these gains place DDTree among the leading approaches to speculative decoding and that it outperforms strong autoregressive drafters such as EAGLE-3 [2604.12989]. These comparisons clarify that “EAGLE-based” now refers as much to a methodological lineage and benchmark standard as to the current performance frontier.

## 7. Conceptual synthesis and research outlook

Across the literature, EAGLE-based speculative decoding evolves from a specific feature-level autoregressive insight into a broad family of drafting-and-verification systems. Three stable principles recur. First, losslessness is preserved through explicit target-side verification [2401.15077], [2601.23180]. Second, speedup depends not only on acceptance length but on the ratio between accepted tokens, drafter cost, and verification cost; several papers formalize this trade-off directly, including TALON’s
$$
R = \frac{\tau}{1 + c \cdot \delta}
$$
and SpecBlock’s
$$
\Phi = \frac{\tau}{T_\mathcal{M} + T_\mathcal{D}}
$$
[2601.07353], [2605.07243]. Third, the dominant bottleneck moves with the setting: drafting in autoregressive drafters, verification in MoE or quantized targets, vocabulary projection in large-vocabulary models, and tree attention or cache semantics in production systems [2605.00342], [2602.13836], [2508.08192].

The major controversies or misconceptions are correspondingly narrow rather than foundational. One misconception is that maximizing acceptance length is sufficient; EVICT shows that in MoE models accepted tokens can increase while end-to-end speed worsens because expert-union growth inflates verification cost [2605.00342]. Another is that speculative decoding improvements transfer uniformly across contexts; OWL and HVD both emphasize that long contexts and agentic workloads change the utility landscape substantially [2510.07535], [2606.01019]. A third is that a single fixed tree shape is adequate; DDD, AdaEAGLE, TALON, and HeteroSpec all argue, by different mechanisms, for adaptive depth, length, or resource allocation [2409.00142], [2412.18910], [2601.07353], [2505.13254].

A plausible synthesis is that the EAGLE line has bifurcated into two research programs. One program refines the original target-conditioned drafter through better architectures, training, and systems co-design, as seen in Beagle, P-EAGLE, SpecBlock, SpecForge, and large-scale production deployments [2505.24544], [2602.01469], [2605.07243], [2603.18567], [2508.08192]. The other program treats EAGLE-3 as a strong baseline and reallocates computation around it through adaptive verification, proxy verification, cache-aware hybridization, dynamic vocabularies, or entirely different drafting mechanisms [2605.00342], [2601.23180], [2606.01019], [2602.13836], [2601.05524], [2604.12989]. In both cases, EAGLE-based speculative decoding remains a central reference point for lossless acceleration research because it ties together drafting quality, verification structure, and deployment realism in a single framework.

Source: https://www.emergentmind.com/topics/eagle-based-speculative-decoding