Speculative Decoding with Quadratic Expansion

• Speculative decoding with quadratic expansion is an inference acceleration method that non-linearly expands candidate token paths in autoregressive models while ensuring exact output distributions.
• It employs a tree-structured, multi-candidate approach that boosts token acceptance rates by verifying multiple paths in parallel during generation.
• Context-aware adaptive expansion dynamically adjusts candidate breadth based on prediction entropy, optimizing computational resources and achieving significant throughput improvements.

Speculative decoding with quadratic expansion refers to a family of inference acceleration techniques for autoregressive models—most notably LLMs—in which the pool of candidate tokens or candidate token paths is expanded beyond the single-path, greedy speculation of classical methods. This expansion is often managed in a way that the number of candidates per step, or the breadth and depth of speculative search trees, grow non-linearly (often quadratically) in either time or resource allocation. The aim is to improve the likelihood of matching the predictions of the target model during verification, thereby increasing the acceptance rate and overall throughput without compromising the exactness of the output distribution. Recent research has demonstrated that mechanisms akin to quadratic expansion—multi-candidate batching, tree-structured speculation, and dynamic context-aware expansion—offer significant advantages in practical inference efficiency and scaling.

1. Foundations of Speculative Decoding and Quadratic Expansion

Speculative decoding was introduced as a principled method to accelerate generation from autoregressive models by leveraging lightweight "draft" models to generate multiple candidate tokens or token segments ahead ("drafting"), which are then verified in parallel by the expensive target model (Leviathan et al., 2022). Classical speculative decoding generates a single linear sequence of draft tokens and verifies their correctness sequentially, accepting as many tokens as possible until a mismatch occurs.

Quadratic expansion arises when the speculative pool is constructed with broader and/or deeper candidate sets: for example, by proposing $k$ candidate tokens at each of $y$ positions, resulting in a $k$-ary tree structure of candidates with up to $O(k y)$ tokens, or, in multi-step generalizations, an exponentially large set of possible draft paths. The term "quadratic expansion" is used to indicate that, under certain configurations, the volume of candidates and the acceptance probability scale in a nonlinear, often quadratic fashion relative to baseline approaches (Yang et al., 12 Jan 2024).

Critically, all accepted tokens must still be distributed as if sampled directly from the target model—ensuring lossless acceleration.

2. Multi-Candidate and Tree-Based Expansion

A central development in quadratic expansion is the transition from single-path speculation to multi-candidate or tree-based speculative decoding. Instead of proposing one token per position, the draft model generates $k$ candidates at each step. These candidates are arranged as branches of a $k$-ary tree, and specialized attention masking ensures that only valid ancestor-descendant relationships are attended during verification (Yang et al., 12 Jan 2024).

Efficiency gains result from this approach because:

• The chance that at least one candidate matches the target model’s preferred token increases with $k$ per step, yielding higher overall acceptance rates.
• Verification can be performed over the tree structure in a single batched pass, amortizing the cost of cache duplication and model forwarding.

Empirically, increasing $k$ from 1 (classical) to higher values (e.g., 4 or 8) leads to acceptance rate improvements from about 49% to over 67% on tasks involving LLaMA and Vicuna models (Yang et al., 12 Jan 2024). The tradeoff entails increased VRAM consumption and computational cost, which may plateau as further candidate expansion yields diminishing returns.

Tree attention (Yang et al., 12 Jan 2024) enables this expansion by allowing all candidate branches to be organized as a single sequential input with a masked attention pattern. Combined with capped budget strategies (e.g., limiting total candidate tokens $K$), quadratic expansion increases throughput while retaining output fidelity.

3. Context-Aware Adaptive Expansion

Quadratic expansion is most powerful when combined with context-driven mechanisms. HeteroSpec (Liu et al., 19 May 2025) leverages a data-driven entropy metric—cumulative meta-path top-$K$ entropy—to gauge the predictability of candidate paths. By partitioning candidate branches into entropy-based bins using a shallow decision tree, HeteroSpec applies:

• Dynamic Extended Drafting: Extending the draft depth proportionally to the simplicity (low entropy) of the context, allowing more aggressive expansion in easy regions.
• Top-N Pruning: Reducing the breadth of candidate branches in regions deemed "hard" by the entropy metric.

These mechanisms adapt the speculative expansion non-uniformly, giving rise to sections of the tree where the number of candidates grows more rapidly (i.e., exhibiting localized quadratic expansion). Experiments on language benchmarks reveal that this strategy yields up to 4.26$\times$ speedups over autoregressive baselines with higher acceptance lengths and lower verification cost than EAGLE-3 (Liu et al., 19 May 2025).

The expansion is dynamically optimized to avoid the explosive growth of candidates in complex regions and exploit high throughput in simple, repetitive contexts.

4. Algorithmic and Theoretical Perspectives

Quadratic expansion naturally interacts with information-theoretic limits. The acceptance probability for any multi-candidate configuration is given by combining the probabilities of all candidate branches through a formula such as

$$P(\text{accept at step } i) = 1 - \prod_{j=1}^k (1 - p_j),$$

where $p_j$ is the probability that the target model would accept candidate $j$ at step $i$ (Yang et al., 12 Jan 2024). This probability increases rapidly as $k$ becomes large, though the incremental benefit is sublinear beyond moderate $k$.

The use of tree pruning and fusion—as in RASD (Quan et al., 5 Mar 2025)—addresses the quadratic cost of verifying overly large candidate sets. By merging branches with common prefixes, the effective verification tree is compressed, and computational cost is kept manageable (notably, the forward pass of attention layers scales quadratically with input length).

Theoretical analyses (Kobus et al., 21 Apr 2025) draw a parallel between speculative decoding and channel simulation, establishing information-theoretic upper bounds on acceleration as a function of the entropy and Kullback-Leibler divergence between the draft and target distributions. Even with quadratic expansion, speed gains are fundamentally limited by the quality of the candidate distribution and the efficient encoding of acceptance information.

5. Practical Implementations and System-Level Considerations

Quadratic expansion yields impressive practical results when combined with memory- and compute-aware implementation strategies:

• Tree Attention and Buffer Management: As in SpecMemo (Yildirim et al., 16 May 2025) and LongSpec (Yang et al., 24 Feb 2025), tree-based candidate expansion is paired with custom attention kernels and dynamic buffer pruning to minimize VRAM usage, enabling deployment on both memory-constrained mobile GPUs and distributed multi-GPU systems.
• Dynamic Depth and Confidence Heuristics: Dynamic Depth Decoding (Brown et al., 30 Aug 2024) halts expansion of the draft tree early if the beam’s cumulative confidence drops below a threshold. This avoids speculative waste when prediction uncertainty is high and extends speculative depth aggressively when the model is confident, further optimizing the performance/speed tradeoff.
• Retrieval and Multi-Sample Fusion: Retrieval-augmented methods (RASD (Quan et al., 5 Mar 2025)) and multi-sample inference schemes (Li et al., 7 Mar 2025) utilize external knowledge bases or consensus among parallel samples to further diversify and expand the candidate tree, compressing quadratic candidate relationships into single-pass verification trees via prefix-trie fusion or DAG aggregation.

Multi-level and staged speculative decoding architectures (Spector et al., 2023, Georganas et al., 17 Mar 2025) recursively apply quadratic expansion at each model level, combining quantized or smaller drafts at the lower levels with aggressive tree expansion at the top, yielding compounded speedup effects.

6. Applications, Limitations, and Future Directions

Quadratic expansion speculative decoding is particularly effective in LLM applications demanding low-latency generation: dialogue, summarization, code completion, machine translation, and multi-turn chatbot systems. Notable empirical gains include improvements of up to 4.26$\times$ average speedup (HeteroSpec (Liu et al., 19 May 2025)), over 3.16$\times$ single-batch boosting (DDD (Brown et al., 30 Aug 2024)), and substantial memory savings with high throughput maintenance (SpecMemo (Yildirim et al., 16 May 2025)).

However, the method introduces new system-level complexities:

• GPU memory utilization increases due to the larger candidate tree structure, necessitating efficient caching and batch management.
• Computational returns may plateau as candidate expansion increases; most empirical studies recommend moderate $k$ (e.g., 2–8).
• For extremely long-context or hierarchical tasks, attention overhead can become significant; frameworks like LongSpec (Yang et al., 24 Feb 2025) and specialized tree fusion/pruning (Quan et al., 5 Mar 2025) are thus critical.

Future research is directed toward even more context-sensitive expansion and pruning, hardware/software co-design for custom attention kernels, and further integration with other acceleration paradigms (quantization, distillation, retrieval augmentation). The interplay between expansion rates, batch sizes, model latency, and quality control remains an area of active investigation, particularly as LLM deployments become more heterogeneous and resource-constrained.

7. Summary Table: Key Methods Featuring Quadratic Expansion

Method/Paper	Expansion Strategy	Notable Results / Remarks
Multi-Candidate SD (Yang et al., 12 Jan 2024)	$k$-ary tree, increase per-step candidates	Acceptance rate up to 67%; 3.7× speedup
HeteroSpec (Liu et al., 19 May 2025)	Adaptive, entropy-based tree	4.26× avg. speedup vs. AR decoding
Staged SD (Spector et al., 2023)	Tree plus multi-stage speculation	3.16× single-batch latency boost
SpecMemo (Yildirim et al., 16 May 2025)	Memory-efficient tree + batch	65% memory cut, up to 8× throughput
RASD (Quan et al., 5 Mar 2025)	Retrieval+generation tree fusion	Quadratic pruning/fusion; SOTA accel.

Quadratic expansion in speculative decoding represents a principled, adaptive approach for accelerating large autoregressive models. Modern designs carefully balance candidate diversity, computational efficiency, and memory constraints using adaptive tree-structured batching, dynamic resource allocation, and advanced buffer and attention management. These strategies deliver substantial empirical speedups and have become foundational in state-of-the-art LLM inference pipelines.