Quadratic Speculative Decoding

Updated 20 July 2025

Quadratic Speculative Decoding is a novel technique that accelerates LLM inference by speculatively generating multiple candidate tokens or trees in parallel.
It employs a two-phase process with a fast drafting stage followed by a parallel verification phase, leveraging structured candidate generation and rejection sampling.
Researchers utilize this method to improve throughput in applications like long-context reasoning and constrained generation while managing memory and computational tradeoffs.

Quadratic Speculative Decoding refers to a class of algorithms and system designs for accelerating autoregressive LLM inference by speculatively generating multiple candidate tokens or token sequences in parallel, then verifying these proposals against a high-fidelity (target) model using a parallel or tree-structured mechanism. The term “quadratic” reflects the aspiration or theoretical potential for efficiency gains that scale more than linearly—ideally proportional to the square of some batching or speculation parameter—thereby exceeding the speedups of strictly linear or batch-parallel approaches. This technique builds on the core principle of speculative decoding but incorporates more sophisticated parallelism, structured candidate generation, and advanced verification strategies to maximize inference throughput while preserving or tightly bounding output quality.

1. Fundamental Principles and Algorithms

Quadratic speculative decoding extends traditional speculative decoding, which divides inference into a fast “draft” and a slower “verification” phase. In the classic algorithm, a small approximation model ( $M_q$ ) rapidly proposes several token candidates, and the large target model ( $M_p$ ) evaluates these in parallel, accepting or rejecting each according to a principled sampling or correction rule. The primary innovation of quadratic approaches lies in proposing multiple tokens (or even entire candidate trees) in one speculative pass and verifying them in a way that leverages both parallel hardware and combinatorial structure.

The general mechanism involves the following workflow:

Draft Phase: A small model (or a parallel drafter, as in ParallelSpec (Xiao et al., 8 Oct 2024)) outputs $k$ candidate tokens (or a candidate tree), potentially in a single forward pass.
Verification Phase: The target model computes logits or likelihoods for all candidates (e.g., one for each length $1, \ldots, k$ speculative prefix), in parallel or batched form. Verification may proceed token-wise or as a tree traversal.
Acceptance Procedure: Each candidate or branch is accepted probabilistically or deterministically based on a comparison of $q(x)$ (draft probability) and $p(x)$ (target probability), ensuring the final output matches or closely approximates the target distribution.
Correction Step: For rejected tokens or sequences, further sampling from the residual (difference) distribution is performed to retain unbiasedness (Leviathan et al., 2022).

The process is mathematically underpinned by a form of rejection sampling:

$\text{Accept token } x \text{ if } q(x) \leq p(x); \quad \text{otherwise, accept with probability}\; \frac{p(x)}{q(x)}$

with correction for excess mass, so that the output remains exactly distributed as $p(x)$ .

When larger speculative blocks or candidate trees are considered, the effective speedup arises from parallel verification (potentially quadratic in the number of tokens handled per pass), but careful construction is needed to avoid prohibitive computational or memory costs typically induced by naively scaling up the speculative tree (Yin et al., 30 Oct 2024, Marzollo et al., 8 Nov 2024, Yang et al., 24 Feb 2025).

2. Architectural Strategies: Parallelism, Trees, and Attention

Quadratic speculative decoding leverages hardware and algorithmic parallelism at multiple levels:

Parallel Drafting: Recent systems like ParallelSpec replace sequential (auto-regressive) drafting with masked, group-wise parallel drafting—using [MASK] tokens or special configurations to output multiple future tokens in one pass (Xiao et al., 8 Oct 2024). Training strategies align the parallel drafter's outputs with the target distribution via knowledge distillation, ensuring acceptability of parallel outputs.
Token Trees: Instead of linear token blocks, a tree of speculative continuations may be constructed, with each path through the tree representing a candidate sequence. Traversal Verification adapts verification to a bottom-up, leaf-to-root paradigm, computing acceptance probabilities for entire paths rather than discarding all descendants on a parent rejection (Weng et al., 18 May 2025). This enables longer and more efficient accepted speculative sequences.
Hybrid Attention and Cache Management: Handling full attention over speculative trees is a major source of quadratic computational cost. Approaches such as LongSpec introduce "Hybrid Tree Attention," splitting attention computation between cached prefixes (computed with optimized kernels) and speculative tokens (handled with custom, often quadratic, masking only on a small subset) (Yang et al., 24 Feb 2025). SwiftSpec develops custom attention operators and tree-aware KV cache management to minimize memory overhead and latency associated with speculative trees (Zhang et al., 12 Jun 2025).

3. Theoretical Analysis and Speedup Limits

The limits of quadratic speculative decoding are formally analyzed using Markov chain abstractions and information-theoretic connections:

Quality-Preserving Guarantees: Properly configured, speculative execution with appropriate correction yields outputs drawn exactly from the target model $p(x)$ (Leviathan et al., 2022, Yin et al., 30 Oct 2024). Unbiasedness is guaranteed by the structure of acceptance and correction formulas, with proofs provided in Traversal Verification (Weng et al., 18 May 2025).
Expected Acceleration: The expected number of tokens accepted per pass is given by

$E[\text{tokens}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}$

where $\alpha$ is the average acceptance probability. Walltime gain is thus a function of $k$ (block or tree size), $\alpha$ , and the cost ratio between draft and target models.

Quadratic Constraints: The quadratic nature often surfaces as the need to verify all speculative branches, whose number can grow rapidly with tree size. Practical architectures limit speculative width or employ tree fusion and pruning (as in RASD (Quan et al., 5 Mar 2025)) to control computational growth.

The theoretical optimality of rejection-based speculative decoding, batch algorithms, and the impact of total variation distance between draft and target models on speedup are established (Theorem 2 and related results in (Yin et al., 30 Oct 2024)). Information-theoretic limits (see exponential race analysis (Kobus et al., 21 Apr 2025)) show that speedup is fundamentally linked to how well $q(x)$ approximates $p(x)$ : tighter alignment yields higher acceptance rates and more candidate tokens processed per expensive forward pass.

4. Practical Implementations and Limitations

Quadratic speculative decoding has matured from theoretical proposals to production-ready methods:

Deployment: Many variants are "plug-and-play," requiring no retraining or architectural modification—a lattice for replacement in serving stacks (Leviathan et al., 2022, Zhao et al., 15 Oct 2024).
Latency and Throughput: Empirical studies report speedups in the $2\times$ – $4\times$ range on LLMs. Notably, SSSD demonstrates up to $4\times$ throughput improvement for short contexts in batched data-center settings (Marzollo et al., 8 Nov 2024), while SwiftSpec achieves 348 tokens/s for Llama3-70B at multi-GPU scale (Zhang et al., 12 Jun 2025).
Quality vs. Speed Tradeoff: High speculative width or tree size can induce quadratic verification costs; mitigation strategies include efficient tree construction, KV cache reuse, and limiting speculative depth (Yang et al., 24 Feb 2025, Weng et al., 18 May 2025).
Memory and Attention Bottlenecks: The quadratic scaling of memory with context or tree length can be a bottleneck (e.g., for key-value cache size), addressed by sliding-window attention, prefix re-use, and hybrid attention strategies (Yang et al., 24 Feb 2025, Zhang et al., 12 Jun 2025).
Reliability and Generalization: In out-of-domain or high-temperature sampling regimes, misalignment between draft and target distributions may degrade acceptance length and speedup. Knowledge distillation at matching temperatures and temperature-aware data composition can improve robustness (Ouyang et al., 14 Oct 2024).

A diverse ecosystem of quadratic speculative decoding methods exists or is emerging:

Speculative Cascades: Combine cascaded deferral rules (e.g., Chow’s rule) with speculative execution to achieve superior cost–quality tradeoffs and leverage ensemble effects (Narasimhan et al., 29 May 2024).
Retrieval-Augmented Drafting: Candidate tokens may be drawn from retrieval systems (datastores, input caches) and merged with model-based drafts using tree fusion to increase acceptance rates and reduce redundant computation (Quan et al., 5 Mar 2025, Marzollo et al., 8 Nov 2024).
Quantization Integration: Employing low-precision draft phase and high-precision verification (QSpec) enables quadratic speculative decoding benefits even under resource constraints, such as on edge devices (Zhao et al., 15 Oct 2024).
Multi-Sample Consensus: Methods such as that proposed in (Li et al., 7 Mar 2025) derive speculative draft tokens by finding consensus among parallel reasoning paths in self-consistency or Best-of- $N$ sampling scenarios.
Constrained and Reward-Guided Decoding: CDSL (Constrained Decoding with Speculative Lookaheads) and related techniques use task-specific reward functions in the validation of speculative tokens, offering massive speedups (up to $12\times$ on some tasks) with robust constraints adherence (Nakshatri et al., 9 Dec 2024).

A table summarizes several key system-level innovations:

Work	Drafting Method	Verification	System Advantage
ParallelSpec	Parallel masks/ $[$ MASK $]$	Tree-based, parallel	$2\times$ – $3\times$ speedup
SwiftSpec	Asynchronous tree, GPU grid	Tree-aware caching, fused op	$1.75\times$ speedup at scale
LongSpec	Sliding-window, hybrid attn	Flash decoding + tree kernel	$3.26\times$ speedup (long context)
QSpec	Quantized model switching	Same model, diff. precision	$1.8\times$ speedup, plug-and-play
Traversal Verif.	Tree, leaf-to-root	Sequence-level accept	$2\%$ – $5\%$ more accepted length

6. Applications, Performance Metrics, and Scaling

Quadratic speculative decoding has been applied in a variety of domains:

General LLM Serving: Drop-in acceleration for GPT, T5, Llama, and similar autoregressive models.
Long-Context Reasoning: Code completion, document summarization, and mathematical reasoning tasks benefit from LongSpec and similar strategies engineered for constant memory and fast hybrid attention (Yang et al., 24 Feb 2025).
Constrained Generation: Integration with constrained decoding (CDSL) provides real-time, reward-guided generation with constraint satisfaction (Nakshatri et al., 9 Dec 2024).
Production Deployment: SSSD and SwiftSpec demonstrate real-world scalability, handling large batch sizes and complex candidate structures efficiently in production-relevant systems (Marzollo et al., 8 Nov 2024, Zhang et al., 12 Jun 2025).
Resource-Constrained Devices: QSpec highlights the compatibility of quadratic speculative decoding with efficient quantization, providing real-time inference in restricted environments (Zhao et al., 15 Oct 2024).

Performance is measured via:

Speedup Ratio (SR): Ratio of speculative to standard decoding walltime.
Acceptance Length ( $\tau$ ): Average number of speculative tokens accepted per verification.
Throughput: Tokens per second, particularly at system scale.

7. Open Challenges and Future Directions

Despite progress, several open directions and complications remain:

Optimal Tree/Block Size: The tradeoff between speculative width, acceptance rate, and computational cost is context-dependent; adaptive mechanisms or formal policies for block/tree selection are under investigation (Narasimhan et al., 29 May 2024, Xiao et al., 8 Oct 2024).
Efficient Attention: Further optimization of attention over speculative trees, including GPU kernel development and streaming methods, may enable larger speculative blocks without quadratic bottlenecks (Yang et al., 24 Feb 2025, Zhang et al., 12 Jun 2025).
Knowledge Distillation and Generalization: Improved alignment of draft and target models—possibly domain-adaptive or temperature-matched KD—remains a critical research area (Ouyang et al., 14 Oct 2024).
Tradeoffs and Theoretical Limits: The Pareto frontier between speed (number of accepted tokens per pass or acceleration factor) and output quality is now characterized with precise bounds. Achieving further gains often requires accepting a quantifiable loss in output fidelity or solving quadratic-optimization subproblems (Yin et al., 30 Oct 2024).
Integration with Retrieval and Augmentation: Tree pruning/fusion and retrieval integration (as in RASD) offer routes to further efficiency, especially for knowledge-intensive or long-context tasks (Quan et al., 5 Mar 2025).
Ultra-Low Latency Designs: Disaggregation and fused, latency-optimized kernels (SwiftSpec) demonstrate that quadratic speculative decoding is compatible with small-batch, low-latency, tensor-parallel serving (Zhang et al., 12 Jun 2025).

Quadratic speculative decoding thus represents a maturing ecosystem of both theoretical frameworks and practical systems, providing robust, scalable, and quality-preserving acceleration for large-scale LLM inference, with broad applicability and ongoing innovation.