--- title: 'EAGLE-3: Accelerating LLM Inference' url: https://www.emergentmind.com/topics/eagle-3 type: topic --- # EAGLE-3: Accelerating LLM Inference EAGLE-3 is an inference acceleration technique for large language models (LLMs) that employs speculative sampling with direct token-level drafting and multi-layer feature fusion facilitated by the "training-time test" procedure. Developed to address the persistent inference bottleneck in autoregressive LLMs, EAGLE-3 significantly improves throughput and speedup across diverse task domains and model families, advancing previous feature-level speculative sampling methods by eliminating feature prediction constraints and more effectively utilizing large training corpora [2503.01840]. ## 1. Inference Bottleneck and Speculative Sampling Autoregressive LLMs, particularly those with hundreds of billions of parameters, require a sequential full forward pass for each generated token, resulting in slow and costly inference. Speculative sampling mitigates this by splitting generation into two phases: (1) a lightweight "draft" model that proposes multiple tokens in parallel and (2) verification by the full-scale "target" LLM that scores and accepts or rejects the draft outputs. Vanilla speculative sampling employs a small draft model \( q \) to generate \( k \) candidate tokens in one pass, followed by parallel scoring in the target model \( p \). The acceptance probability for the \( i \)-th draft token \( \hat t_{j+i} \) is given by: \[ \alpha_i = \min \Bigl( 1, \frac{p(\hat t_{j+i} \mid T_{1:j+i-1})}{q(\hat t_{j+i} \mid T_{1:j+i-1})}\Bigr) \] A key limitation is the insufficient accuracy of the draft model, leading to low acceptance rates and restricted speedups. The EAGLE and EAGLE-2 methods introduced feature-prediction-based drafting: instead of predicting tokens, the draft model generates top-layer hidden features \( f_{t+1} \) for the target model, which are then passed to the LM head to obtain token distributions. The corresponding training loss is: \[ \mathcal{L}_{\rm EAGLE} = \underbrace{\|\hat f_{t+1} - f_{t+1}\|^2}_{\ell_{\rm fea}} + \underbrace{-\log\,p(t_{t+1} \mid \hat f_{t+1})}_{\ell_{\rm token}} \] These approaches, particularly EAGLE-2 with dynamic tree-pruning, improve speculative sampling but remain limited by expressivity constraints imposed by feature regression and are not able to fully leverage larger training sets. ## 2. Direct Token Prediction and Training-Time Test EAGLE-3 introduces two principal innovations: the replacement of feature-level drafting in favor of direct token prediction, and the fusion of multi-layer features using a novel "training-time test" simulation. The draft model in EAGLE-3 outputs a token distribution \( q \) at each step, removing the feature regression loss entirely. The training objective for a generation prefix of length \( t \) is: \[ \mathcal{L}_{\rm E3} = -\sum_{i=1}^k \log\,q(t_{t+i} \mid g_{1:t},\, a_{t+1:t+i-1}) \] where \( g_{1:t} \) are fused features and \( a_{t+1:t+i-1} \) are previous draft outputs. This formulation enables the draft model to capture richer representations and fully exploit increases in training corpus size. Multi-layer feature fusion avoids the narrow top-layer constraints of prior EAGLE versions by collecting features from multiple transformer layers (e.g., low-, mid-, and high-level). These are fused via concatenation and a learned linear projection: \[ g_t = W_{\rm fuse}[f^{(1)}_t; \dots; f^{(L)}_t] \in \mathbb{R}^k \] The *training-time test* enables the draft model to simulate multi-step autoregressive generation during training, attending to its own previous predictions through custom causal masks. At each simulated position, the model restricts attention to the correct causal prefix, thereby learning to handle its own outputs as inputs—a crucial requirement for robust drafting at inference. ## 3. Draft Model Architecture and Pipeline Design EAGLE-3 integrates multi-layer feature fusion and direct token prediction within a streamlined draft model architecture. The inference pipeline operates as follows: 1. The target model \( p \) pre-fills the prefix \( T_{1:t} \), exposing low-, mid-, and high-level features \((l_t, m_t, h_t)\). 2. These features are fused into \( g_t = W_{\rm fuse}[l_t; m_t; h_t] \). 3. The draft model receives as input the tuple \( \{g_{1:t},\, a_{t+1:t+j-1}\} \) and processes it with a single-layer Transformer decoder (consisting of self-attention and feed-forward modules). 4. The output vector \( a_{t+j} \) is projected by the target model’s LM head to yield the draft token distribution \( q(\cdot) \). The draft token \( \hat t_{t+j} \) is sampled from this distribution. 5. This draft/verify loop is repeated for up to \( k \) tokens, after which all are verified by the target LLM in parallel. Compared to prior methods, the multi-layer fusion approach enables the draft model to access more diverse semantic representations, improving draft quality and acceptance rates. ## 4. Mathematical Formulation and Speedup Metrics The target LLM’s next-token distribution is expressed as: \[ p(t_{t+1} \mid T_{1:t}) \] while the draft model’s approximation is: \[ q(\hat t_{t+1} \mid g_{1:t},\, a_{