Dynamic Speculative Decoding Engine

• DSDE is a framework that dynamically adapts speculative decoding by controlling draft length, candidate trees, and verification policies based on runtime signals.
• It leverages online metrics like KLD variance and batch size to optimize performance, achieving throughput gains and significant latency reductions in various deployments.
• By replacing static draft–verify configurations with adaptive, per-sequence policies, DSDE enhances efficiency in both single-node and distributed inference architectures.

Dynamic Speculative Decoding Engine (DSDE) denotes a class of serving-time control frameworks for speculative decoding that replace a fixed draft–verify configuration with runtime adaptation. In the narrow sense, the term is explicitly instantiated by a training-free framework that selects speculation length per sequence and per iteration using post-hoc KLD-variance signals plus a batch-level cap (Yang et al., 1 Sep 2025). In a broader systems sense, recent work uses the same design pattern under different names: Nightjar adapts speculative length to request load and can disable speculative decoding entirely, FASER manages speculative length, pruning, and overlap inside continuous batching, and PipeDec turns speculative decoding into a pipeline-parallel execution engine with a dynamic prediction tree (Li et al., 27 Dec 2025, Chen et al., 22 Apr 2026, Yin et al., 5 Apr 2025). This suggests that DSDE is best understood less as a particular proposer architecture than as the online control and scheduling layer around speculative inference.

1. Conceptual scope and lineage

Standard speculative decoding accelerates autoregressive generation by letting a lightweight draft mechanism propose future tokens and letting the target model verify them in parallel. DSDE emerges when that mechanism is no longer treated as a static decoding mode but as a runtime policy problem: how many tokens to speculate, whether to speculate at all, how to structure candidate trees, how strictly to verify, and how to coordinate draft and verification phases under changing serving conditions. Nightjar makes this explicit by casting speculative length selection as an online decision problem over batch-conditioned goodput, with $\gamma=0$ representing “speculative decoding disabled” (Li et al., 27 Dec 2025).

The term also spans several orthogonal control surfaces. Some systems adapt speculation length, as in DSDE proper, Nightjar, FASER, and edge–cloud AWC; some adapt tree structure, as in DySpec’s dynamic token tree and DDD’s dynamic depth; some adapt verification policy, as in DIVERSED’s learned mixture of draft and target distributions; and some adapt execution topology, as in PipeDec’s pipeline-native dynamic prediction tree and decentralized DSD’s communication-aware verification (Xiong et al., 2024, Brown et al., 2024, Wang et al., 8 Apr 2026, Song et al., 13 Nov 2025). The commonality is runtime control over the speculative process rather than a fixed draft horizon or a fixed verifier.

A recurrent misconception is to equate DSDE with any speculative decoder. The literature distinguishes the two more sharply: the novel part relative to standard speculative decoding is not draft/verify itself, but the dynamic controller that decides when, how, and under which constraints draft/verify should run (Li et al., 27 Dec 2025).

2. Control objectives and operating regimes

The central DSDE objective is not merely “maximize acceptance,” but optimize useful progress under a changing hardware and serving regime. Nightjar frames the per-batch decision as

$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$

where $g_{B,\gamma}$ is goodput for batch size $B$ and speculative length $\gamma$. Its motivating empirical pattern is regime-dependent: at low request rates, speculative decoding can improve throughput because the target model is memory-bound, while at high request rates continuous batching pushes the system into a compute-bound regime where verification overhead dominates and plain autoregressive decoding can be faster. In the paper’s example, fixed $\gamma=3$ improved throughput by 15.5% at 15 QPS under low load, but at sufficiently high load speculative decoding degraded performance by up to 30.25% relative to no speculative decoding (Li et al., 27 Dec 2025).

FASER makes the same point from a different angle. In its H100 measurements, verification accounts for up to 83% of total latency as batch size rises from 16 to 256, whereas at small batch sizes draft latency is nearly half of decode latency. The implication is that DSDE policies must be regime-aware twice over: low-load systems suffer from serialized draft/verify phases and underutilized GPU resources, while high-load systems suffer from wasted verification on rejected suffixes (Chen et al., 22 Apr 2026).

Distributed settings add a second operating axis: communication. “Speculation at a Distance” shows that if the server can host both draft and target models, co-located speculative decoding has lower single-request latency than synchronous edge–cloud DSD, and pipelined DSD becomes competitive only when the round trip is shorter than the edge drafting window. Against cloud autoregressive decoding, DSD reduces latency only within a bounded RTT window determined by target speed, acceptance rate, speculation length, and communication payload. By contrast, the main positive case for DSD appears under multi-tenant capacity, where offloading draft computation can let a saturated cloud sustain

$\left(1+\gamma\,\frac{t_d}{t_v}\right)$

more concurrent clients at the same per-client output rate (Lyu et al., 23 Jun 2026).

Production-engine evidence reinforces the same conclusion. A systematic vLLM study reports that verification by the target model dominates execution, that acceptance length varies markedly across output token positions, requests, and datasets, and that measured performance often remains far below theoretical upper bounds. DSDE therefore sits at the intersection of algorithmic acceptance, verification cost, and serving-state dynamics rather than at any single acceptance-rate optimum (Liu et al., 31 Dec 2025).

3. Online signals and decision policies

Recent DSDE systems differ primarily in which runtime signals they trust. Nightjar uses a narrow but explicitly online control state: current batch size $B_t$, realized reward $r_t(\gamma_t)$ measured as empirical goodput, the previous speculative length $\gamma_{t-1}$, and an offline-profiled draft reactivation cost $$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$0. It treats speculative-length choice as contextual multi-armed bandit optimization over $$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$1, with regret

$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$2

and maintains per-$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$3 goodput estimates online by cumulative moving average. A notable feature is that $$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$4 is a first-class action, so the policy can disable speculation rather than only shorten it (Li et al., 27 Dec 2025).

The explicit DSDE formulation in (Yang et al., 1 Sep 2025) instead uses post-hoc KLD diagnostics. Its key signal is the Weighted Variance Intensity Ratio,

$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$5

combined with a scale factor

$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$6

The next-step prediction is

$$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$7

with a fallback to $$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$8 when $$\gamma^*(B)=\mathop{\arg\max}_{\gamma \in \{0,1,\dots,\Gamma_{\max}\}} g_{B,\gamma},$$9, and the batch-level cap is the arithmetic mean

$g_{B,\gamma}$0

This design is explicitly per-sequence and per-iteration, while the cap regularizes stragglers created by heterogeneous per-sequence lengths (Yang et al., 1 Sep 2025).

FASER uses a more systems-facing policy. For request $g_{B,\gamma}$1, batch size $g_{B,\gamma}$2, and draft SM allocation $g_{B,\gamma}$3, it minimizes estimated latency per committed token,

$g_{B,\gamma}$4

and searches over candidate lengths with GP-LCB:

$g_{B,\gamma}$5

The same controller uses a net-benefit trigger for verifier-side pruning, enabling early exit only when expected saved verification time exceeds pruning overhead. Edge–cloud DSD’s AWC uses a different feature vector—queue depth utilization, recent acceptance rate, per-link RTT statistics, TPOT statistics, and prior window size—and switches to fused mode when the predicted window satisfies $g_{B,\gamma}$6 (Chen et al., 22 Apr 2026, Yu et al., 26 Nov 2025).

Other work treats verification strength itself as the dynamic control surface. DIVERSED predicts

$g_{B,\gamma}$7

then verifies against

$g_{B,\gamma}$8

instead of the exact target distribution. In this formulation, DSDE does not only decide draft length; it decides how strictly the target should enforce agreement with the draft, trading exactness for higher acceptance under a task-dependent and context-dependent policy (Wang et al., 8 Apr 2026).

4. Execution architectures and engine modules

A DSDE can be decomposed into a small number of recurring modules. Nightjar’s blueprint is especially explicit: a runtime monitor observes current batch size, previous speculative mode, skipped-prefix lag, and realized goodput; an offline profiler populates the reactivation-cost lookup table; a policy state store maintains empirical goodput and per-batch-size exploration state; decision logic chooses between exploration and exploitation; and scheduler integration applies the chosen $g_{B,\gamma}$9 at decoding-step boundaries inside vLLM continuous batching (Li et al., 27 Dec 2025).

FASER expands this into a more fine-grained serving engine. Its four main components are an Offline Profiler, an Adaptive Drafter, a Token-wise Early Exiter, and a Pipeline Overlapper. The execution granularity is request-level for speculative length, token-level for pruning, and chunk-level for frontier overlap, with CUDA Green Contexts used to partition draft and target SM resources spatially (Chen et al., 22 Apr 2026).

PipeDec generalizes the idea to pipeline-parallel inference. It assigns a dedicated draft node $B$0, large-model pipeline nodes $B$1, and a dynamic prediction tree whose layers are streamed into the pipeline as soon as the draft model produces them. The engine maintains a two-level KV cache—ordinary model KV plus prediction-tree-specific KV—and prunes to the accepted subtree when the last pipeline group verifies a token. This architecture turns speculative decoding into a distributed task graph with transport scheduling, synchronization, and cache transfer, rather than a single monolithic verifier call (Yin et al., 5 Apr 2025).

DSDEs can also vary in proposer design. SSSD replaces the draft model with a CPU-side retrieval proposer that fuses prompt/self-output matches and suffix-array datastore matches into a verification tree, while DySpec builds a dynamic token tree online by greedily expanding the frontier node with maximal estimated contribution to expected accepted tokens. These systems suggest that a DSDE need not be tied to a neural drafter at all; it can treat proposing, verification, and scheduling as separable modules (Marzollo et al., 2024, Xiong et al., 2024).

5. Representative empirical findings

Empirical results across the literature are heterogeneous but consistently support the value of runtime adaptation. Nightjar reports up to 14.8% higher throughput and 20.2% lower latency than standard speculative decoding, with 7B ShareGPT throughput of 3734.23 tokens/s versus 3593.36 for fixed-length SD and 3653.61 without SD, and 7B Alpaca latency of 8868.17 ms versus 11110.68 ms for standard SD. Under a dynamic Azure-derived request trace, it reaches about 3800 tokens/s during low-load periods while maintaining the lead during high-load phases (Li et al., 27 Dec 2025).

FASER reports up to 53% throughput improvement and up to $B$2 latency reduction versus state-of-the-art systems. In its Qwen3 ablation, Adaptive Drafter alone yields up to 19% latency reduction and up to $B$3 throughput improvement, Early Exiter yields 26% latency reduction and $B$4 throughput improvement, and the full system reaches 61% latency reduction and $B$5 throughput improvement (Chen et al., 22 Apr 2026).

Pipeline-native and distributed DSDEs show that the same design logic applies beyond single-node serving. PipeDec achieves $B$6–$B$7 decoding speedup over traditional pipeline parallelism and $B$8–$B$9 over baseline tree-based speculative decoding in a 14-stage deployment (Yin et al., 5 Apr 2025). Decentralized DSD reports up to $\gamma$0 speedup on HumanEval and $\gamma$1 on GSM8K while preserving accuracy, and adaptive verification adds an additional 15% to 20% end-to-end speedup over a nonadaptive speculative baseline (Song et al., 13 Nov 2025). In heterogeneous edge–cloud serving, distributed SD with AWC achieves up to 1.1x speedup and 9.7% higher throughput over existing SD baselines (Yu et al., 26 Nov 2025).

Dynamic structure control can be even more aggressive when the target is large and the proposer is cheap. DySpec reports throughput up to $\gamma$2 and latency reduction up to $\gamma$3 on Llama2-70B at low temperature (Xiong et al., 2024). Spec-LLaVA reports up to $\gamma$4 faster decoding on LLaVA-1.5 7B/13B with no loss in generation quality under greedy decoding (Huo et al., 15 Sep 2025). SSSD reports 1.37–1.80x speedups even at batch size 64, a 4x throughput increase with no latency impact on the operating frontier for short-context generation, and 1.7–2x improvement in both latency and throughput for longer contexts (Marzollo et al., 2024). These figures should not be read as directly comparable across engines; they illustrate that DSDE is a family of control strategies whose realized gain depends strongly on model scale, proposer cost, verification overhead, and workload.

6. Limitations, misconceptions, and open problems

The main limitation across DSDE designs is partial observability. Nightjar conditions primarily on batch size and realized goodput; the paper explicitly notes that prompt length, output length, temperature, acceptance probability, queueing delay, and GPU utilization are not used as direct control features. It also provides no universal table mapping batch size to optimal speculative length, implying that practical policies remain deployment-specific (Li et al., 27 Dec 2025).

Signal quality is another unresolved issue. The KLD-based DSDE paper reports weak token-level Pearson correlations between acceptance and common signals on CNN/DM: at temperature 0.0, entropy has $\gamma$5, mean KLD $\gamma$6, and WVIR $\gamma$7; at temperature 1.0, the corresponding values are $\gamma$8, $\gamma$9, and $\gamma=3$0. The paper therefore presents KLD variance as a macroscopic diagnostic rather than a strong per-token predictor, and acknowledges weaker behavior under stochastic sampling (Yang et al., 1 Sep 2025).

Several systems require significant supporting infrastructure. FASER depends on offline profiling refreshed every two hours and leaves key controller details, such as exact candidate set sizes and depth-dependent Top-$\gamma=3$1 schedules, unspecified in the provided text. PipeDec attains low single-request latency at the cost of substantial runtime complexity, extra speculative KV state, and lower throughput under memory-rich conditions where ordinary PP or static-tree PP can batch more effectively (Chen et al., 22 Apr 2026, Yin et al., 5 Apr 2025).

Not all DSDE directions preserve exact target-model semantics. DIVERSED explicitly relaxes verification by sampling from a learned ensemble distribution rather than the target model, and cross-task transfer experiments show that higher acceptance can coexist with degraded downstream quality. This suggests a fundamental divide between lossless DSDEs, which adapt scheduling while preserving the target distribution, and lossy DSDEs, which treat verification policy itself as a quality–latency operating point (Wang et al., 8 Apr 2026).

Finally, communication-aware work places hard external limits on what a DSDE can do. “Speculation at a Distance” argues that edge–cloud DSD is generally infeasible against closed-source APIs without a verifier-only interface and should be judged primarily by multi-tenant capacity rather than single-request latency (Lyu et al., 23 Jun 2026). The broader vLLM study likewise cautions that real systems still leave substantial gaps between measured SD speedups and theoretical upper bounds, with target verification remaining dominant (Liu et al., 31 Dec 2025). The open problem, therefore, is not whether dynamic control matters—it does—but which signals, objectives, and system decompositions let that control remain lightweight, stable, and beneficial under production serving constraints.