---
title: Quadratic Speculative Decoding
url: https://www.emergentmind.com/topics/quadratic-speculative-decoding
type: topic
---

# Quadratic Speculative Decoding

Quadratic Speculative Decoding refers to a class of algorithms and system designs for accelerating autoregressive large language model (LLM) inference by speculatively generating multiple candidate tokens or token sequences in parallel, then verifying these proposals against a high-fidelity (target) model using a parallel or tree-structured mechanism. The term “quadratic” reflects the aspiration or theoretical potential for efficiency gains that scale more than linearly—ideally proportional to the square of some batching or speculation parameter—thereby exceeding the speedups of strictly linear or batch-parallel approaches. This technique builds on the core principle of speculative decoding but incorporates more sophisticated parallelism, structured candidate generation, and advanced verification strategies to maximize inference throughput while preserving or tightly bounding output quality.

## 1. Fundamental Principles and Algorithms

Quadratic speculative decoding extends traditional speculative decoding, which divides inference into a fast “draft” and a slower “verification” phase. In the classic algorithm, a small approximation model ($M_q$) rapidly proposes several token candidates, and the large target model ($M_p$) evaluates these in parallel, accepting or rejecting each according to a principled sampling or correction rule. The primary innovation of quadratic approaches lies in proposing multiple tokens (or even entire candidate trees) in one speculative pass and verifying them in a way that leverages both parallel hardware and combinatorial structure.

The general mechanism involves the following workflow:

1. **Draft Phase**: A small model (or a parallel drafter, as in ParallelSpec [2410.05589]) outputs $k$ candidate tokens (or a candidate tree), potentially in a single forward pass.
2. **Verification Phase**: The target model computes logits or likelihoods for all candidates (e.g., one for each length $1, \ldots, k$ speculative prefix), in parallel or batched form. Verification may proceed token-wise or as a tree traversal.
3. **Acceptance Procedure**: Each candidate or branch is accepted probabilistically or deterministically based on a comparison of $q(x)$ (draft probability) and $p(x)$ (target probability), ensuring the final output matches or closely approximates the target distribution.
4. **Correction Step**: For rejected tokens or sequences, further sampling from the residual (difference) distribution is performed to retain unbiasedness [2211.17192].

The process is mathematically underpinned by a form of rejection sampling:
$$
\text{Accept token } x \text{ if } q(x) \leq p(x); \quad \text{otherwise, accept with probability}\; \frac{p(x)}{q(x)}
$$
with correction for excess mass, so that the output remains exactly distributed as $p(x)$.

When larger speculative blocks or candidate trees are considered, the effective speedup arises from parallel verification (potentially quadratic in the number of tokens handled per pass), but careful construction is needed to avoid prohibitive computational or memory costs typically induced by naively scaling up the speculative tree [2411.00841, 2411.05894, 2502.17421].

## 2. Architectural Strategies: Parallelism, Trees, and Attention

Quadratic speculative decoding leverages hardware and algorithmic parallelism at multiple levels:

- **Parallel Drafting**: Recent systems like ParallelSpec replace sequential (auto-regressive) drafting with masked, group-wise parallel drafting—using [MASK] tokens or special configurations to output multiple future tokens in one pass [2410.05589]. Training strategies align the parallel drafter's outputs with the target distribution via knowledge distillation, ensuring acceptability of parallel outputs.
- **Token Trees**: Instead of linear token blocks, a tree of speculative continuations may be constructed, with each path through the tree representing a candidate sequence. Traversal Verification adapts verification to a bottom-up, leaf-to-root paradigm, computing acceptance probabilities for entire paths rather than discarding all descendants on a parent rejection [2505.12398]. This enables longer and more efficient accepted speculative sequences.
- **Hybrid Attention and Cache Management**: Handling full attention over speculative trees is a major source of quadratic computational cost. Approaches such as LongSpec introduce "Hybrid Tree Attention," splitting attention computation between cached prefixes (computed with optimized kernels) and speculative tokens (handled with custom, often quadratic, masking only on a small subset) [2502.17421]. SwiftSpec develops custom attention operators and tree-aware KV cache management to minimize memory overhead and latency associated with speculative trees [2506.11309].

## 3. Theoretical Analysis and Speedup Limits

The limits of quadratic speculative decoding are formally analyzed using Markov chain abstractions and information-theoretic connections:

- **Quality-Preserving Guarantees**: Properly configured, speculative execution with appropriate correction yields outputs drawn exactly from the target model $p(x)$ [2211.17192, 2411.00841]. Unbiasedness is guaranteed by the structure of acceptance and correction formulas, with proofs provided in Traversal Verification [2505.12398].
- **Expected Acceleration**: The expected number of tokens accepted per pass is given by
$$
E[\text{tokens}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}
$$
where $\alpha$ is the average acceptance probability. Walltime gain is thus a function of $k$ (block or tree size), $\alpha$, and the cost ratio between draft and target models.
- **Quadratic Constraints**: The quadratic nature often surfaces as the need to verify all speculative branches, whose number can grow rapidly with tree size. Practical architectures limit speculative width or employ tree fusion and pruning (as in RASD [2503.03434]) to control computational growth.

The theoretical optimality of rejection-based speculative decoding, batch algorithms, and the impact of total variation distance between draft and target models on speedup are established (Theorem 2 and related results in [2411.00841]). Information-theoretic limits (see exponential race analysis [2504.15475]) show that speedup is fundamentally linked to how well $q(x)$ approximates $p(x)$: tighter alignment yields higher acceptance rates and more candidate tokens processed per expensive forward pass.

## 4. Practical Implementations and Limitations

Quadratic speculative decoding has matured from theoretical proposals to production-ready methods:

- **Deployment**: Many variants are "plug-and-play," requiring no retraining or architectural modification—a lattice for replacement in serving stacks [2211.17192, 2410.11305].
- **Latency and Throughput**: Empirical studies report speedups in the $2\times$–$4\times$ range on large language models. Notably, SSSD demonstrates up to $4\times$ throughput improvement for short contexts in batched data-center settings [2411.05894], while SwiftSpec achieves 348 tokens/s for Llama3-70B at multi-GPU scale [2506.11309].
- **Quality vs. Speed Tradeoff**: High speculative width or tree size can induce quadratic verification costs; mitigation strategies include efficient tree construction, KV cache reuse, and limiting speculative depth [2502.17421, 2505.12398].
- **Memory and Attention Bottlenecks**: The quadratic scaling of memory with context or tree length can be a bottleneck (e.g., for key-value cache size), addressed by sliding-window attention, prefix re-use, and hybrid attention strategies [2502.17421, 2506.11309].
- **Reliability and Generalization**: In out-of-domain or high-temperature sampling regimes, misalignment between draft and target distributions may degrade acceptance length and speedup. Knowledge distillation at matching temperatures and temperature-aware data composition can improve robustness [2410.10141].

## 5. Extensions, Variants, and Related Approaches

A diverse ecosystem of quadratic speculative decoding methods exists or is emerging:

- **Speculative Cascades**: Combine cascaded deferral rules (e.g., Chow’s rule) with speculative execution to achieve superior cost–quality tradeoffs and leverage ensemble effects [2405.19261].
- **Retrieval-Augmented Drafting**: Candidate tokens may be drawn from retrieval systems (datastores, input caches) and merged with model-based drafts using tree fusion to increase acceptance rates and reduce redundant computation [2503.03434, 2411.05894].
- **Quantization Integration**: Employing low-precision draft phase and high-precision verification (QSpec) enables quadratic speculative decoding benefits even under resource constraints, such as on edge devices [2410.11305].
- **Multi-Sample Consensus**: Methods such as that proposed in [2503.05330] derive speculative draft tokens by finding consensus among parallel reasoning paths in self-consistency or Best-of-$N$ sampling scenarios.
- **Constrained and Reward-Guided Decoding**: CDSL (Constrained Decoding with Speculative Lookaheads) and related techniques use task-specific reward functions in the validation of speculative tokens, offering massive speedups (up to $12\times$ on some tasks) with robust constraints adherence [2412.10418].

A table summarizes several key system-level innovations:

| Work             | Drafting Method             | Verification                 | System Advantage                   |
|------------------|----------------------------|------------------------------|------------------------------------|
| ParallelSpec     | Parallel masks/$[$MASK$]$   | Tree-based, parallel         | $2\times$–$3\times$ speedup        |
| SwiftSpec        | Asynchronous tree, GPU grid | Tree-aware caching, fused op | $1.75\times$ speedup at scale      |
| LongSpec         | Sliding-window, hybrid attn | Flash decoding + tree kernel | $3.26\times$ speedup (long context)|
| QSpec            | Quantized model switching   | Same model, diff. precision  | $1.8\times$ speedup, plug-and-play |
| Traversal Verif. | Tree, leaf-to-root         | Sequence-level accept        | $2\%$–$5\%$ more accepted length   |

## 6. Applications, Performance Metrics, and Scaling

Quadratic speculative decoding has been applied in a variety of domains:

- **General LLM Serving**: Drop-in acceleration for GPT, T5, Llama, and similar autoregressive models.
- **Long-Context Reasoning**: Code completion, document summarization, and mathematical reasoning tasks benefit from LongSpec and similar strategies engineered for constant memory and fast hybrid attention [2502.17421].
- **Constrained Generation**: Integration with constrained decoding (CDSL) provides real-time, reward-guided generation with constraint satisfaction [2412.10418].
- **Production Deployment**: SSSD and SwiftSpec demonstrate real-world scalability, handling large batch sizes and complex candidate structures efficiently in production-relevant systems [2411.05894, 2506.11309].
- **Resource-Constrained Devices**: QSpec highlights the compatibility of quadratic speculative decoding with efficient quantization, providing real-time inference in restricted environments [2410.11305].

Performance is measured via:

- **Speedup Ratio (SR)**: Ratio of speculative to standard decoding walltime.
- **Acceptance Length ($\tau$)**: Average number of speculative tokens accepted per verification.
- **Throughput**: Tokens per second, particularly at system scale.

## 7. Open Challenges and Future Directions

Despite progress, several open directions and complications remain:

- **Optimal Tree/Block Size**: The tradeoff between speculative width, acceptance rate, and computational cost is context-dependent; adaptive mechanisms or formal policies for block/tree selection are under investigation [2405.19261, 2410.05589].
- **Efficient Attention**: Further optimization of attention over speculative trees, including GPU kernel development and streaming methods, may enable larger speculative blocks without quadratic bottlenecks [2502.17421, 2506.11309].
- **Knowledge Distillation and Generalization**: Improved alignment of draft and target models—possibly domain-adaptive or temperature-matched KD—remains a critical research area [2410.10141].
- **Tradeoffs and Theoretical Limits**: The Pareto frontier between speed (number of accepted tokens per pass or acceleration factor) and output quality is now characterized with precise bounds. Achieving further gains often requires accepting a quantifiable loss in output fidelity or solving quadratic-optimization subproblems [2411.00841].
- **Integration with Retrieval and Augmentation**: Tree pruning/fusion and retrieval integration (as in RASD) offer routes to further efficiency, especially for knowledge-intensive or long-context tasks [2503.03434].
- **Ultra-Low Latency Designs**: Disaggregation and fused, latency-optimized kernels (SwiftSpec) demonstrate that quadratic speculative decoding is compatible with small-batch, low-latency, tensor-parallel serving [2506.11309].

Quadratic speculative decoding thus represents a maturing ecosystem of both theoretical frameworks and practical systems, providing robust, scalable, and quality-preserving acceleration for large-scale language model inference, with broad applicability and ongoing innovation.

Source: https://www.emergentmind.com/topics/quadratic-speculative-decoding