Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path

Abstract: A stateless inference server (vLLM, SGLang, TensorRT-LLM) idles between requests while the accelerator waits; a stateful session reclaims that idle time. Speculative pre-positioning decodes the session forward to its next decision point with the target model's own forward pass and no draft model, moving the cross-request prefill and entry-decode off the critical path: the next request resumes from a pre-paid entry on its delta, or, when a confidence gate fires, is answered from a cached distribution in one near-constant vocabulary scan with no decode, at a cost only of energy and a rare, bounded false accept. The payoff is conditional on capability: a capable model fires the gate at near-full coverage and about 87% precision (a smaller one never clears it), returning the first token in about 1.0 ms versus the 39 ms decode a prefix cache still pays.

• The paper demonstrates a method to pre-decode the next token during idle windows by caching predicted key-value state and output distributions.
• It uses a confidence gating mechanism that, when triggered, reduces first-token latency from over 50 ms to approximately 1 ms.
• Empirical results on 70B models show that this approach balances energy costs and speed gains, ensuring reliable performance in stateful LLM sessions.

Speculative Pre-Positioning: Off-Critical-Path Decoding for Stateful Sessions

Introduction and Motivation

This work introduces speculative pre-positioning (SPP) as a method to exploit the idle windows inherent in stateful session-based LLM inference—intervals between requests when GPU resources are underutilized. Whereas stateless servers (e.g., vLLM, SGLang, TensorRT-LLM) cannot make use of this downtime, stateful sessions maintain persistent session state, enabling advance computation. SPP moves the decode work for the next entry point off the critical path by forwarding the session to the predicted next request position during idle time, caching both the key-value (KV) state and the output distribution (i.e., logits at the next token position).

The core intuition is to amortize the computational cost of decoding by executing foreseen work in advance, “pre-paying” for the next session entry's compute during naturally occurring idle windows such as (i) between streaming data updates and queries, and (ii) between agentic tool call emits and deliveries. The next query or agentic resume either consumes this pre-positioned state for a delta-only update or is, in high-confidence cases, directly answered via a cached (single-token) output, contingent upon a calibrated confidence gate.

Formal Model and Mechanisms

Sessions are maintained with persistent KV caches. When an update completes, the session is in state $n_s$. The foreseeable next action will append an entry of $L_{\text{entry}}$ tokens, after which the model will be ready to begin generation at decision point $p = n_s + L_{\text{entry}}$.

During the idle window ($I_\text{idle}$), SPP performs:

• Entry prefill: Runs a model forward pass on these $L_{\text{entry}}$ tokens, updating the KV state.
• Ready-distribution caching: Saves the logits (the full output distribution) at $p$.
• Confidence-gated answer: At inference time, if the realized request matches the pre-positioned entry, a fast path is offered: if the gap between top-1 and top-2 logits ($\gamma$) exceeds a threshold $\tau$ and the top answer lies within a defined fast-answer set, return the cached response immediately via an argmax scan; else, fall back to delta-only decode.

The mechanism is generic to both streaming and agentic (tool-calling) sessions. Cache invalidation occurs upon any mutation of the entry-relevant state, ensuring correctness and freshness. All mechanisms run at lowest priority, yielding immediately on new request arrival to guarantee non-interference with critical work.

Confidence Gating and Selective Prediction

The approach is structurally different from speculative decoding schemes requiring draft models and verification [leviathan2023fast]. No such verification is run here; safety is instead enforced probabilistically via a calibrated gate. The model serves a cached token only if $\gamma > \tau$ and the token is admissible.

This operationalizes selective prediction [elyaniv2010selective]: one can trade off between coverage (requests seeing the fast path) and risk (false-accepts, i.e., when the cached answer diverges from what a full decode on the realized request would produce). The system tunes $\tau$ to meet a fixed risk budget.

Analytical Model and Cost Amortization

The cost benefit is made possible if the pre-position work fits within the idle window ($L_{\text{entry}}$0). If so, wall-clock latency is not increased; the only real cost is energy expenditure and, rarely, a false-accept, both of which are tightly bounded.

Performance is characterized by expected first-token latency:

$L_{\text{entry}}$1

where $L_{\text{entry}}$2 is the hit rate (fraction of pre-positions consumed before invalidation), $L_{\text{entry}}$3 is the probability the gate fires, and $L_{\text{entry}}$4, $L_{\text{entry}}$5, $L_{\text{entry}}$6 are the respective latencies for the fast, fallthrough, and cold path.

Numerically, for a representative 70B-scale model and 18-token entry,

• Cold path: P50 first-token is 53.1 ms.
• Fast path: P50 is 1.01 ms.
• Speedup: Over 50-fold reduction versus the cold path; nearly 40-fold versus traditional prefix/KV cache that still must run a decode.

Energy cost per pre-position is measured at 63.8 J. For idle:active query/update ratios as in realistic deployments, most pre-positions are consumed, minimizing waste.

Empirical Results

Empirical evaluation is conducted on a single H100 with a 70B-class model at 4-bit quantization (see Table~\ref{tab:measured} in the original). The selective prediction calibration shows the confidence gate, at $L_{\text{entry}}$7, covers roughly 94.5% of cases, with a measured false-accept rate of 13.2%. The measured per-pre-position hit rate ranges from 13% to 78% across session regimes.

Key observations:

• The fast path brings first-token latency to $L_{\text{entry}}$8 ms.
• The gate fires only for capable models (here, the 70B model); a smaller 8B model’s lower decisiveness means SPP does not fire, and provides no speedup.
• The critical amortization, and thus the practical benefit, depends on session/query workload and the model’s confidence.

Limitations and Applicability

• Model ability: SPP confers benefit only when the model is decisively confident (as measured by logit gap); for under-confident or small-scale models, no speedup is realized.
• Risk control: The absence of fast-path verification means risk is managed only via $L_{\text{entry}}$9 calibration; misconfiguration could degrade response fidelity.
• Energy tradeoff: Unconsumed pre-positions increase idle energy use. For update-heavy sessions, wasted energy can be nontrivial.
• Applicability: SPP requires persistent session state and is not compatible with stateless or batch-oriented inference looping where session state is not preserved across requests.
• Domain constraints: The single-token fast path is applicable to domains where a fast-answer set is appropriate; arbitrary long-form answers fall through to standard paths.

Relationship to Prior Work

SPP is distinct from prior speculative decoding [leviathan2023fast, chen2023spec] and prefix/KV caching [kwon2023vllm, zheng2024sglang]. Unlike those, it speculates the decode entry point and output distribution, not token sequences, and does so entirely with the main (“target”) model, not with drafts. Verification is replaced with a probabilistic guarantee via confidence gating. SPP composes with existing stateful LLM serving systems (especially LayerScale-style persistent-session systems) (Norgren, 13 May 2026, Norgren, 25 May 2026).

Future Directions

Open research directions include (i) extending SPP-type pre-positioning to multi-token or open-ended completions, (ii) more granular or hierarchical invalidation and caching strategies, (iii) empirical assessment across broader domains and conversational settings, and (iv) integration with model scaling laws to predict SPP hit/risk profiles for models between 8B and 70B parameters.

Conclusion

Speculative pre-positioning provides a rigorous method for anticipating and moving decode work off the critical path in stateful LLM serving by opportunistically decoding to the next decision point during idle windows, caching both KV state and output distributions. Strong empirical speedups are demonstrated (50×+ reduction in first-token latency) for high-capability models, with risk bounded by a selective-prediction-calibrated gate. The mechanism introduces new opportunities and energy-right tradeoffs for stateful, persistent-session LLM deployment in streaming and agentic (tool-using) workloads, though its benefit is contingent upon session regimes with predictive structure, persistent state, and sufficiently confident models.

Reference: "Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path" (2606.29565).