DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Abstract: Speculative decoding accelerates LLM inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token dependencies. Furthermore, indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems. We introduce DSpark, a speculative decoding framework that unifies high-throughput parallel generation with adaptive, load-aware verification. To maintain draft quality, DSpark utilizes a semi-autoregressive architecture, coupling a parallel backbone with a lightweight sequential module, to introduce intra-block dependency modeling and mitigate suffix decay. To optimize system efficiency, DSpark employs confidence-scheduled verification, dynamically tailoring the verification length for each request based on estimated prefix survival probabilities and engine-specific throughput profiles. On offline benchmarks across diverse domains, DSpark substantially improves the accepted length over state-of-the-art autoregressive and parallel drafters. When deployed within the DeepSeek-V4 serving system under live user traffic, DSpark successfully mitigates verification waste. Compared to the established production baseline (MTP-1), DSpark accelerates per-user generation speeds by 60 to 85 percent at matched throughput levels. More importantly, by preventing severe throughput degradation under strict interactivity constraints, it enables performance tiers that were previously unattainable, shifting the Pareto frontier of our serving system.

• The paper introduces DSpark, a hybrid semi-autoregressive decoding framework that unifies parallel processing with sequential dependency modeling to enhance throughput and token acceptance rates.
• It combines a deep parallel backbone with a lightweight sequential head and calibrated confidence estimation, enabling adaptive verification scheduling for efficient LLM inference.
• Empirical results show DSpark delivers significant throughput gains (up to 52% increase) over traditional autoregressive and parallel models while maintaining coherent output.

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Motivation and Problem Statement

LLM inference is fundamentally bottlenecked by token-level autoregressive decoding, which yields high latency and poor GPU utilization, especially under high-concurrency workloads and strict interactivity constraints. Speculative decoding addresses this by employing lightweight draft models to propose token blocks, which are then verified and accepted (or rejected) in a parallelized manner via rejection sampling. Prior architectures fall into autoregressive (high-quality, high-latency) and parallel (low-latency, low inter-token dependency) categories, each presenting trade-offs in throughput and acceptance rates. Parallel drafters, despite architectural speedup, suffer from rapid suffix acceptance decay due to lack of intra-block modeling, while indiscriminate block verification wastes critical compute under heavy serving loads.

DSpark Architecture and Decoding Pipeline

DSpark introduces a hybrid semi-autoregressive framework, combining a deep parallel backbone with a lightweight sequential output head to inject local dependencies and mitigate suffix decay, accompanied by a calibrated confidence model used for adaptive, system-aware verification budgeting.

Figure 1: DSpark architecture showcasing parallel backbone for draft token generation, sequential head for intra-block dependency, and dynamic confidence-based prefix scheduling.

The parallel backbone leverages target-model context features (KV injection), enabling deep, blockwise computation with $O(1)$ latency. The sequential module—Markov or optionally RNN head—adds prefix-conditioned transition biases, enabling left-to-right sampling conditioned on prior tokens within the block. Confidence estimation, supervised by analytical prefix survival probabilities, supports adaptive scheduling, allowing dynamic truncation of verification length informed by both empirical survival rates and hardware throughput profiles.

Algorithmic Formulation

Draft token generation proceeds via two stages:

1. Parallel backbone: produces token-level logits $U_k$ using anchor token and masks.
2. Sequential head: computes prefix-dependent bias $B_k(x_{<k}, x_k)$, realizing causal block distribution:

$$P(X | x_0) = \prod_{k=1}^{\gamma} p_k(x_k | x_0, x_{<k})$$

with direct softmax evaluation per token.

For Markov heads, $B(x_{k-1}, x_k)$ is approximated via low-rank factorization. For RNN heads, stateful prefix embeddings enable memory of all prior tokens in the block.

Confidence head outputs per-token survival probabilities, calibrated using Sequential Temperature Scaling (STS) to match empirical rejection rates. Dynamic prefix scheduler (via a greedy, monotonic global sort) maximizes expected system throughput $\Theta = \tau \cdot SPS(B)$, where $\tau$ is expected accepted tokens and $SPS(B)$ is profiled engine throughput for batch size $B$, while ensuring causal admission decisions to preserve lossless distribution.

Empirical Evaluation

DSpark consistently outperforms autoregressive (Eagle3) and parallel (DFlash) drafters in average accepted length ($\tau$), across diverse domains (math, code, chat) and target model scales. The macro-average improvements are 30.9–30.0% over Eagle3 and 16.3–18.4% relative to DFlash for Qwen3 target models (4B, 8B, 14B), with consistent gains for Gemma4-12B.

In position-wise conditional acceptance analysis, DSpark preserves the high initial acceptance seen in deep parallel drafters and mitigates suffix decay characteristic of pure parallel models, matching or exceeding autoregressive conditional acceptance rates past the first token.

Figure 2: Conditional acceptance across positions; DSpark stably mitigates suffix decay compared to pure parallel drafter.

Increasing drafter depth in DSpark yields monotonic gains, with a shallow 2-layer DSpark outperforming deeper (5-layer) DFlash, confirming parameter efficiency as a function of local auto-regression.

Figure 3: DSpark's accepted length increases with drafter depth, showing parameter efficiency of sequential modeling.

Scaling proposal length further increases DSpark's relative advantage over pure parallel drafting. Latency overhead from sequential head remains negligible, adding only 0.2–1.3% engine roundtrip for block sizes up to 16.

Figure 4: DSpark outperforms DFlash across proposal lengths, with minimal latency overhead from sequential head.

Confidence-Driven Verification and Calibration

Static confidence threshold sweeps exhibit substantial improvements in acceptance rates when pruning low-confidence suffix tokens, especially on high-entropy chat domains. Acceptance rate rises from 45.7% to 95.7% on chat, and 76.9% to 92.5% on math, demonstrating the estimator’s ability to identify low-value draft positions.

Figure 5: Confidence threshold sweep shows acceptance rate improvements due to confidence-based pruning.

Reliability diagrams highlight initial overconfidence in raw estimator predictions, corrected via STS calibration to align predicted survival with empirical acceptance. Post-calibration, ECE drops below 1%, supporting throughput-maximizing scheduling.

Figure 6: Reliability diagram (Alpaca) shows improvement from calibration, aligning predicted and empirical acceptance.

Production Deployment and Throughput Analysis

DSpark deployed in DeepSeek-V4 production engines (Flash, Pro) demonstrates robust gains under live traffic. At moderate interactivity SLAs, DSpark increases throughput by 51–52% over single-token (MTP-1) baseline. Under strict SLAs, DSpark maintains stable capacity, shifting the Pareto frontier; per-user generation speeds accelerate by 60–85% (Flash) and 57–78% (Pro) at matched throughput.

Figure 7: Throughput versus interactivity frontier; DSpark expands attainable performance envelope over MTP-1 baseline.

Scheduler adaptation to load is evident as verification budget per request shrinks smoothly under rising concurrency, preventing resource contention while maximizing idle compute utility.

Figure 8: Throughput and verification budget dynamics; scheduler restricts verification length to optimize resource utilization under load.

System-level optimizations (hidden state communication, anchor-bounded sequence packing, asynchronous scheduling) ensure scalable training and inference with negligible overhead, enabling efficient variable-length query processing and dynamic routing.

Implications and Future Directions

DSpark's semi-autoregressive architecture resolves the fundamental trade-off between initial block accuracy and suffix coherence. The calibrated confidence-scheduled verification maximizes hardware throughput while minimizing wasted compute under load, supporting production-scale serving with robust SLAs. The lossless rejection sampling protocol and causality-preserving scheduler guarantee exact target distribution recovery.

Algorithmically, the Markov head provides favorable deployment characteristics, with minimal complexity and strong performance scaling with block size. RNN heads offer incremental gains but increase implementation friction.

Practically, DSpark enables higher user interactivity tiers and maintains aggregate throughput under concurrency, opening avenues for efficient agentic workflows and real-time LLM applications. The open-source release of DSpark artifacts and DeepSpec training repository facilitate reproducibility and further research.

Theoretically, semi-autoregressive speculative decoding motivates new directions in hybrid modeling for sequence generation, bridging parallelization and dependency modeling. Further work may explore early-exit mechanisms in the drafter for difficulty-aware processing and structured draft models leveraging advanced global sequence objectives.

Conclusion

DSpark establishes a semi-autoregressive speculative decoding methodology, unifying parallel generation speed with coherent intra-block dependency modeling and adaptive verification scheduling. It advances both algorithmic and system-level LLM inference, demonstrating superior draft quality, system-aware verification, and practical throughput gains in live deployments. These architectural and scheduling innovations pave the way for scalable, low-latency LLM serving under diverse production workloads.

(2607.05147)