OnlineSpec: Adaptive Speculative Decoding

• OnlineSpec is a unified framework for speculative decoding that leverages a draft–verify loop to continuously adapt draft models during deployment.
• It employs optimistic online learning and ensemble methods to reduce dynamic regret, thereby increasing accepted token lengths and achieving up to 24% extra speedup.
• By using verification feedback as free supervision, OnlineSpec enhances draft model improvement without extra target evaluations, ensuring efficient deployment.

Searching arXiv for the main paper and key related speculative decoding works.

Search query: (Qian et al., 13 Mar 2026)

Search query: speculative decoding Leviathan 2023

Search query: Hydra speculative decoding

Search query: EAGLE speculative decoding

OnlineSpec is a unified framework for speculative decoding that systematically leverages interactive feedback to continuously evolve draft models during deployment. In speculative decoding, a small draft model rapidly proposes a block of $k$ tokens that are then verified in parallel by a larger target model; OnlineSpec treats the resulting verification signal as an online learning loop—draft commits, target verifies, feedback is observed, draft adapts—and formalizes how online learning performance, especially dynamic regret, controls acceptance length and acceleration. The framework develops algorithms based on optimistic online learning and online ensemble learning, and reports up to $24\%$ additional speedup across seven benchmarks and three foundation models (Qian et al., 13 Mar 2026).

1. Speculative decoding setting and the OnlineSpec problem

OnlineSpec is defined in the setting where the target predictive distribution at context $x$ is denoted by $p_v(\cdot \mid x)$ and the draft predictive distribution by $q_{\theta}(\cdot \mid x)$. At each step $t$, the draft proposes a block of $k$ tokens $x_1,\ldots,x_k$ via autoregression under $q_{\theta_t}$, while the target computes in parallel $p_v(x_i \mid x_{<i})$. The standard acceptance test accepts token $24\%$0 if a uniform draw $24\%$1, otherwise it rejects at position $24\%$2. If $24\%$3 is the number of consecutive draft tokens accepted at step $24\%$4, then $24\%$5 (Qian et al., 13 Mar 2026).

A central quantity is the per-token acceptance probability

$24\%$6

Using $24\%$7 and the fact that both distributions sum to $24\%$8, the framework uses the identity

$24\%$9

where $x$0 is total variation distance. Under mild i.i.d. assumptions across draft positions $x$1 at step $x$2, the expected acceptance length $x$3 is a monotone function of $x$4. A common closed form for truncated geometric acceptance is

$x$5

up to an index shift variant used in prior work; all such forms increase with $x$6 and $x$7.

The acceleration metric is expressed in wall-clock terms. Let $x$8 be the time per target forward pass and $x$9 the time per draft per token, with $p_v(\cdot \mid x)$0. Generating a block costs $p_v(\cdot \mid x)$1, while baseline autoregressive generation costs $p_v(\cdot \mid x)$2 per token. Over $p_v(\cdot \mid x)$3 steps, if the total number of accepted tokens is $p_v(\cdot \mid x)$4, then the speedup is

$p_v(\cdot \mid x)$5

with upper bound

$p_v(\cdot \mid x)$6

This formulation makes long acceptance runs structurally necessary: short runs collapse speedup, whereas long runs are required to approach the maximal factor $p_v(\cdot \mid x)$7.

2. Verification feedback as online learning

OnlineSpec’s key observation is that verification intrinsically produces, at no extra target calls, a rich feedback signal that pinpoints draft–target discrepancies. The framework casts each decoding step as an online round. At round $p_v(\cdot \mid x)$8, the learner picks $p_v(\cdot \mid x)$9 and induces $q_{\theta}(\cdot \mid x)$0; the environment reveals feedback via verification, which defines a loss $q_{\theta}(\cdot \mid x)$1 measuring $q_{\theta}(\cdot \mid x)$2’s deviation from $q_{\theta}(\cdot \mid x)$3 at the contexts encountered at round $q_{\theta}(\cdot \mid x)$4; and the learner updates $q_{\theta}(\cdot \mid x)$5 (Qian et al., 13 Mar 2026).

A natural full-information loss is the cross-entropy of $q_{\theta}(\cdot \mid x)$6 against the target,

$q_{\theta}(\cdot \mid x)$7

where $q_{\theta}(\cdot \mid x)$8 is the relevant context distribution induced by verification at round $q_{\theta}(\cdot \mid x)$9. If $t$0 is an oracle per-round comparator minimizing $t$1, then

$t$2

By Pinsker’s inequality, $t$3, so the framework obtains

$t$4

Lowering per-round $t$5 therefore directly raises $t$6 and $t$7.

The online objective is dynamic regret,

$t$8

Lower dynamic regret means that the learner tracks the best time-varying draft $t$9 for each round. OnlineSpec then connects this quantity to acceptance and speed: as dynamic regret decreases, aggregate acceptance length increases, and acceleration improves. This suggests that speculative decoding is not merely a verification procedure but a deployment-time adaptation process in which verification feedback is already the supervision signal needed for continual draft improvement.

3. Algorithms and operational loop

OnlineSpec develops two principal algorithmic families. The first is optimistic online learning, denoted Opt. Optimism injects predictive hints about upcoming gradients to adapt faster when gradients are predictable. If $k$0 is obtained from verification at round $k$1, then a simple hint is $k$2; more generally, one may use an exponential moving average of past gradients, or task-specific predictors. Two equivalent update forms are described. Optimistic Mirror Descent uses

$k$3

$k$4

where $k$5 is a Bregman divergence for a $k$6-strongly convex regularizer $k$7. The Euclidean projected optimistic step is

$k$8

If hints are accurate, with small $k$9, regret scales with the hint error rather than $x_1,\ldots,x_k$0, leading to better dependence on non-stationarity and longer acceptance runs (Qian et al., 13 Mar 2026).

The second family is online ensemble learning, denoted Ens. Ensembles hedge against unknown and time-varying environments. The framework maintains $x_1,\ldots,x_k$1 draft models $x_1,\ldots,x_k$2 with parameters $x_1,\ldots,x_k$3 and diverse step sizes or inductive biases. Each base updates via online gradient descent,

$x_1,\ldots,x_k$4

Mixture weights $x_1,\ldots,x_k$5 are updated by Hedge or Exponentiated Gradient,

$x_1,\ldots,x_k$6

followed by normalization. The ensemble can be used as a mixture draft,

$x_1,\ldots,x_k$7

or through multi-branch drafting, where a small number of candidates from top-weighted drafters are sampled, batch-verified with the target, and the longest accepted branch is selected. The latter is described as compatible with Medusa/Hydra/EAGLE-style parallel trees.

The practical loop is explicit. At each round, the current draft autoregressively samples up to $x_1,\ldots,x_k$8 tokens. A single target forward then yields $x_1,\ldots,x_k$9 for $q_{\theta_t}$0 in batched form. Consecutive tokens are accepted until the first failure; if rejection occurs at position $q_{\theta_t}$1, the next token is resampled from the target as in standard speculative decoding. From the verification pass, the target logits are already available, so the framework computes a per-round loss and gradients without extra target calls. The loss can be token-level cross-entropy, a KL divergence estimate via logit matching, or, for reasoning, a preference-based or DPO loss that replaces token-level supervision with pairwise step preference. The learner then performs OGD, optimistic, or ensemble updates and proceeds to the next block.

4. Acceptance–regret theory and acceleration bounds

The theoretical program of OnlineSpec is to make the acceptance–regret link explicit under standard online convex optimization conditions: a bounded domain, bounded gradients, full-information feedback from verification, and the i.i.d.-across-positions simplification used in prior speculative-decoding analyses (Qian et al., 13 Mar 2026).

The framework’s key identities are:

$q_{\theta_t}$2

$q_{\theta_t}$3

and

$q_{\theta_t}$4

Hence

$q_{\theta_t}$5

and $q_{\theta_t}$6 increases as $q_{\theta_t}$7 decreases. Substituting this lower bound into the truncated geometric formula and aggregating across rounds yields a lower bound of the form

$q_{\theta_t}$8

for explicit $q_{\theta_t}$9, $p_v(x_i \mid x_{<i})$0, and $p_v(x_i \mid x_{<i})$1. Intuitively, as $p_v(x_i \mid x_{<i})$2, the aggregate acceptance length approaches the ideal $p_v(x_i \mid x_{<i})$3.

The acceleration theorem states that

$p_v(x_i \mid x_{<i})$4

Moreover, as the dynamic regret $p_v(x_i \mid x_{<i})$5 decreases sublinearly, $p_v(x_i \mid x_{<i})$6, the acceptance length $p_v(x_i \mid x_{<i})$7 increases, and $p_v(x_i \mid x_{<i})$8 improves. The paper then provides algorithm-specific regret and speedup translations, with bounds holding up to logarithmic factors and constants suppressed.

For online gradient descent with $p_v(x_i \mid x_{<i})$9,

$24\%$00

where

$24\%$01

is the path-length of the moving comparator sequence. The resulting speedup lower bound is

$24\%$02

For optimistic online learning with hint error

$24\%$03

the regret bound becomes

$24\%$04

and the speedup lower bound becomes

$24\%$05

Accurate hints therefore tighten regret and yield higher $24\%$06.

For the online ensemble method, with $24\%$07 base drafters with geometrically spaced step sizes and a Hedge meta-learner,

$24\%$08

with a similar speedup lower bound and strong adaptivity when $24\%$09 is large. The qualitative message is precise: stability, good hints, or ensembles reduce dynamic regret; smaller dynamic regret raises acceptance probability, accepted length, and thus speedup.

5. Implementation, compute profile, and deployment behavior

OnlineSpec is designed so that online adaptation does not change the basic speculative decoding systems picture. Per round, the target cost is one forward pass with $24\%$10 parallel positions, with cost $24\%$11. Drafting costs $24\%$12, where $24\%$13 and $24\%$14. Gradients are computed from target logits already available from verification, so the online update requires no extra target calls. Draft updates are small backprop steps on the draft parameters, or on head-only adapters, and are typically amortized asynchronously on separate devices. The latency-critical path remains dominated by $24\%$15 (Qian et al., 13 Mar 2026).

The implementation details are correspondingly lightweight. Token-level cross-entropy or KL is computed from the target logits produced during verification, with no target modification or extra queries. For reasoning tasks, the paper uses a DPO-style loss

$24\%$16

where $24\%$17 is a preferred/dispreferred pair from semantic verification and $24\%$18 is log-likelihood ratio against a reference policy. For Online-LR, AdamW with small learning rate is used; for EAGLE-style heads, Adam; for Hydra heads, SGD with momentum. Gradient clipping and mixed precision, specifically bf16, are used, with chunked streaming evaluation such as $24\%$19–$24\%$20 and periodic updates.

The ensemble implementation maintains $24\%$21 base drafters with geometrically spaced $24\%$22, updates meta-weights from per-round losses, and combines them by mixture or branch selection. Base learners train in parallel. On the systems side, KV cache reuse is unchanged; a single target pass is retained per block; updates run asynchronously on spare GPUs; and periodic parameter synchronization preserves inference-time efficiency. The experiments use $24\%$23A800 80GB, large CPU memory, and FlashAttention for efficient attention.

A plausible implication is that OnlineSpec’s deployment profile is intentionally conservative: it does not require extra target evaluation, does not alter the target model, and places most adaptation overhead on small draft-side optimization. This is why the framework is presented as deployment-friendly rather than as a retraining-heavy alternative to speculative decoding.

6. Empirical results, scope, and limitations

The empirical study spans seven datasets covering math reasoning, code generation, and finance question answering: GSM8K, MATH, the math subset of MMLU, CodeSearch-Python, Spider, MBPP, and Alpaca-finance. The target models are Vicuna-7B, Llama-2-7B-Chat, and Qwen3-8B. Across these settings, OnlineSpec instantiations consistently beat static speculative decoding and naive online baselines such as OSD, with speedups up to $24\%$24 wall-clock over prior SOTA and clear gains in average accepted length (Qian et al., 13 Mar 2026).

The reported task-specific instantiations are meant to show compatibility with existing speculative systems. Opt-Hydra improves over Hydra and OSD-Hydra across tasks. Ens-EAGLE and Ens-EAGLE-3 improve over EAGLE and OSD-EAGLE or OSD-EAGLE-3. Online-LR, which uses DPO-style online updates for lookahead reasoning, outperforms offline LR and naive OSD-LR. These results are paired with ablations on learning rates, hint quality, and online window sizes. Fixed $24\%$25 exhibits the expected tradeoff between slow and unstable behavior and does not match Opt-Hydra or Ens-EAGLE, which is presented as validation of optimism and ensembles. Better gradient predictability lowers $24\%$26 and improves speed. Longer online adaptation raises speed monotonically, matching the theory that $24\%$27 improves as regret per round shrinks.

The framework also identifies several limitations. Aggressive online updates can overfit recent feedback and shorten acceptance later in the same sequence; learning-rate control and optimism or ensemble smoothing mitigate but do not eliminate this. Per-deployment adaptation may forget older domains under cross-sequence distribution shift, making meta-learning and replay buffers promising. Feeding target logits into online training raises privacy considerations; secure logging and on-device learning are described as options. Compatibility with beam search and top-$24\%$28/top-$24\%$29 sampling is good in principle, but acceptance formulas and per-token independence approximations should be revisited under heavy sampling truncation. Server batching may couple different users’ online loops, so per-tenant adapters or weight partitioning can help. The paper further points to combining optimism and ensembles as an open engineering problem, and to bandit extensions when only partial feedback is available, such as acceptance length but not full logits.

In this formulation, OnlineSpec is neither a new decoding criterion nor a replacement for speculative decoding’s lossless verification semantics. It is a theory-backed adaptive layer on top of the draft–verify architecture: verification provides free, informative feedback about draft–target discrepancy; online learning converts that feedback into draft evolution; and smaller dynamic regret is translated into higher acceptance probability, longer accepted prefixes, and improved acceleration.