Distributed Split Speculative Decoding

• Distributed Split Speculative Decoding is a device–edge inference architecture that splits the verification phase to lessen heavy full-vocabulary uplink transfers.
• It leverages a two-model system where a local small language model drafts tokens and an edge-hosted large language model performs accept/reject validation.
• Experiments show that DSSD significantly reduces communication overhead and improves decoding speed compared to prior distributed speculative decoding methods.

Searching arXiv for the primary DSSD paper and closely related distributed speculative decoding work. Searching for "Distributed Split Speculative Decoding" and related edge/cloud speculative decoding papers on arXiv. Distributed Split Speculative Decoding (DSSD) is a device–edge collaborative inference architecture in which a small LLM $M_q$ on the device drafts tokens for a LLM $M_p$ at the base station or edge server, while the verification phase itself is partitioned across the two sides. In this design, accept/reject is performed at the edge and resample is performed on the device. DSSD is introduced to preserve the SLM–LLM split of distributed speculative decoding while replacing the uplink transmission of multiple vocabulary distributions with a single downlink transmission, thereby reducing communication latency while maintaining inference quality (Ning et al., 16 Jul 2025).

1. System model and motivation

DSSD is formulated for a device–edge system in which the device hosts an SLM $M_q$ and the edge hosts an LLM $M_p$. The paper assumes that the two models share a common vocabulary $\mathcal V$. The motivation is that modern LLMs are too heavy to run efficiently on mobile or resource-constrained devices, while edge-only execution introduces network delay and reliability problems. The resulting design problem is therefore not only model execution, but the joint balance of local computation and network communication (Ning et al., 16 Jul 2025).

The immediate precursor in this line is distributed speculative decoding in which the device-side drafter and edge-side verifier preserve the classical draft–target division but require the device to send full draft-side distributions for verification. In that broader edge–cloud formulation, the draft role is split to the edge and the verification role to the cloud, but the protocol remains a standard draft-then-verify loop (Yu et al., 26 Nov 2025). DSSD targets the specific communication bottleneck of that arrangement: for draft length $\gamma$, the uplink payload associated with transmitting draft-model vocabulary distributions scales as $\gamma \cdot |\mathcal V|$ (Ning et al., 16 Jul 2025).

The architectural distinction between DSD and DSSD is concise.

Aspect	DSD	DSSD
Drafting location	Device-side SLM	Device-side SLM
Accept/reject location	Edge	Edge
Resampling location	Edge	Device
Heavy full-distribution transfer	$\gamma$ uplink distributions	At most one downlink distribution on rejection

This means DSSD is not a conventional layer-partitioned split-computing system. Rather, it preserves the two-model speculative-decoding structure and splits the verification phase itself. That split is the defining feature of the method (Ning et al., 16 Jul 2025).

2. Protocol and split verification procedure

DSSD keeps the standard speculative-decoding drafting stage on the device. Starting from the current prefix, the device initializes $\mathbf y=[]$ and, for $i=1,\dots,\gamma$, computes

$M_p$0

then appends $M_p$1 to $M_p$2. After this step, the device has the drafted tokens $M_p$3 and the local draft distributions $M_p$4 (Ning et al., 16 Jul 2025).

The compact uplink message does not contain the full $M_p$5 vectors. Instead, the device sends the drafted token identities together with the scalar probabilities of those selected tokens, written in the paper as the probability values $M_p$6. From context, these are the probabilities of the drafted tokens under the SLM, i.e. $M_p$7. At the edge, the LLM computes the verification distributions

$M_p$8

conditioned on the prefix and the drafted continuation. Verification then proceeds sequentially. For position $M_p$9, the edge samples

$M_q$0

and accepts the draft token if

$M_q$1

If all $M_q$2 draft tokens are accepted, the edge samples

$M_q$3

The crucial split occurs when rejection happens. If the first rejection is at position $M_q$4, the edge sends back $M_q$5 and the index $M_q$6. The device then performs the corrective resampling locally using its stored $M_q$7: $M_q$8 and replaces $M_q$9 by $M_p$0. The prefix is then extended by $M_p$1. If all draft tokens were accepted, the edge instead returns $M_p$2 and $M_p$3, and the prefix is extended accordingly (Ning et al., 16 Jul 2025).

The paper notes one scheduling detail in the appendix: when rejection occurs, the resampled token still needs to be uploaded to the edge, but this can be deferred to the next draft–verify round. This suggests a limited form of deferred synchronization, although DSSD is otherwise presented as a round-based protocol rather than an explicitly asynchronous pipeline (Ning et al., 16 Jul 2025).

A common misunderstanding is to treat DSSD as ordinary split inference across model layers. The method is not described that way. Its novelty is the relocation of resampling from the edge to the device, so that the edge only needs enough draft-side information to run accept/reject, while the device reuses its local $M_p$4 to construct the residual resampling distribution (Ning et al., 16 Jul 2025).

3. Communication and latency model

The paper models total communication time as

$M_p$5

where $M_p$6 denotes non-transmission time. For the earlier DSD baseline, the dominant term is the uplink of $M_p$7 full draft distributions: $M_p$8 which yields

$M_p$9

The corresponding inference latency per round is

$\mathcal V$0

This is the communication regime DSSD is designed to avoid (Ning et al., 16 Jul 2025).

For DSSD, the expensive vocabulary-sized transmission moves to the downlink and occurs only if at least one drafted token is rejected. Let $\mathcal V$1 if there is a rejection and $\mathcal V$2 otherwise. Then

$\mathcal V$3

If $\mathcal V$4 denotes the per-token acceptance probability and independence across the $\mathcal V$5 drafted positions is assumed in the analysis, then the probability that all $\mathcal V$6 tokens are accepted is

$\mathcal V$7

Hence the expected DSSD communication time becomes

$\mathcal V$8

and the inference latency is

$\mathcal V$9

The analytical consequence is straightforward. DSD scales communication roughly with $\gamma$0 on the uplink, whereas DSSD scales it with $\gamma$1 on the downlink. The paper also states

$\gamma$2

As $\gamma$3, the probability of at least one rejection tends to $\gamma$4, so $\gamma$5. As $\gamma$6, the probability of needing the downlink distribution vanishes, so $\gamma$7. This is why DSSD is strongest when acceptance is high (Ning et al., 16 Jul 2025).

The paper claims that DSSD maintains inference quality because it uses the same speculative-decoding acceptance and residual-resampling mechanism as standard speculative decoding, merely relocating the resampling computation to the device. This suggests exact target-distribution preservation by construction. However, the paper does not provide a standalone theorem or formal proof of exact distributional equivalence, so the strongest faithful statement is that exactness is implied by construction rather than formally established (Ning et al., 16 Jul 2025).

4. Experimental characterization

The reported experiments use OPT-125M as the device-side draft model and either OPT-6.7B or OPT-13B as the edge-side target model. Each run starts from a common 128-token narrative prompt, generates 128 additional tokens, and uses top-$\gamma$8 sampling with

$\gamma$9

The homogeneous setup uses two NVIDIA H800 GPUs, one for the draft model and one for the target. The heterogeneous setup uses an NVIDIA A6000 on the device side and an NVIDIA H800 at the edge. Network conditions include non-transmission delays of $\gamma \cdot |\mathcal V|$0, $\gamma \cdot |\mathcal V|$1, and $\gamma \cdot |\mathcal V|$2 ms with a bandwidth cap of $\gamma \cdot |\mathcal V|$3 Mbps, and the tables additionally include $\gamma \cdot |\mathcal V|$4 Mbps and $\gamma \cdot |\mathcal V|$5 Mbps settings (Ning et al., 16 Jul 2025).

The communication reduction is extreme in the reported measurements. The paper states that DSD uploads approximately 61,269 bytes per round, whereas DSSD uploads less than 50 bytes each time. This matches the protocol change from full-vocabulary uplink distributions to drafted token indices plus scalar probabilities, with at most one vocabulary distribution returned on rejection (Ning et al., 16 Jul 2025).

The measured acceptance rates are approximately

$\gamma \cdot |\mathcal V|$6

for OPT-125M $\gamma \cdot |\mathcal V|$7 OPT-6.7B and

$\gamma \cdot |\mathcal V|$8

for OPT-125M $\gamma \cdot |\mathcal V|$9 OPT-13B. In the homogeneous experiments discussed in the text, DSSD speedup is roughly $\gamma$0 to $\gamma$1, whereas DSD often remains near $\gamma$2 or below it once communication cost is included. Representative points include, for $\gamma$3, $\gamma$4 ms, $\gamma$5 Mbps:

• DSD (125M $\gamma$6 6.7B): 1.41
• DSSD (125M $\gamma$7 6.7B): 2.31

and for $\gamma$8, $\gamma$9 ms, $\mathbf y=[]$0 Mbps:

• DSD: 0.43
• DSSD: 2.19 (Ning et al., 16 Jul 2025)

In the heterogeneous experiments, the paper defines

$\mathbf y=[]$1

with reported values $\mathbf y=[]$2 for 125M $\mathbf y=[]$3 6.7B and $\mathbf y=[]$4 for 125M $\mathbf y=[]$5 13B. Under these more realistic conditions, speedups are more sensitive to $\mathbf y=[]$6 and target size. The paper reports speedups up to roughly $\mathbf y=[]$7 for OPT-6.7B and $\mathbf y=[]$8 for OPT-13B. It also reports an interior optimum in the speculative length: under $\mathbf y=[]$9 ms and $i=1,\dots,\gamma$0 Mbps, $i=1,\dots,\gamma$1 performs best, whereas $i=1,\dots,\gamma$2 underutilizes speculation and $i=1,\dots,\gamma$3 increases mismatch and overhead. This directly contradicts the misconception that speculative length should simply be made as large as possible (Ning et al., 16 Jul 2025).

5. Relation to adjacent distributed and speculative decoding systems

Subsequent work broadened the distributed speculative-decoding design space in several directions, but not by replacing DSSD’s core communication split. The later edge–cloud framework “DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving” places a lightweight draft model on edge devices and a heavyweight target model in the cloud, then studies routing, batching, and an Adaptive Window Control policy through a discrete-event simulator. Its novelty is a systems framework for multi-device deployments rather than DSSD’s relocation of resampling from edge to device (Yu et al., 26 Nov 2025).

Other systems extend the split-execution idea to multi-user or one-to-many settings. “DiP-SD: Distributed Pipelined Speculative Decoding for Efficient LLM Inference at the Edge” studies a multi-user device–edge regime in which many devices draft locally and a centralized edge server performs batched verification; its emphasis is joint optimization of batch count, user-to-batch assignment, and draft lengths under a throughput-maximization objective. “StarSD: One-for-Many Speculative Decoding” instead uses a single draft model to serve multiple target models via a star topology, with a global request buffer and per-session state isolation, so its central question is draft sharing and utilization rather than DSSD’s split-verification communication rule (Xu et al., 22 Apr 2026, He et al., 29 Jan 2026).

At the algorithmic level, several papers contribute abstractions that are relevant to DSSD without being distributed DSSD methods themselves. “FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving” shows that dynamic online workloads benefit from per-request speculative length adaptation, token-wise early pruning, and chunked frontier verification that overlaps with drafting; these ideas map naturally to split or disaggregated pipelines, although the paper is primarily a local-runtime system rather than an edge–cloud design (Chen et al., 22 Apr 2026). “3-Model Speculative Decoding” introduces Pyramid Speculative Decoding, a hierarchical draft $i=1,\dots,\gamma$4 qualifier $i=1,\dots,\gamma$5 target cascade with fuzzy acceptance thresholds. It does not study distributed placement, but it offers a transferable hierarchical verification abstraction that could serve as a multi-stage verifier in a split deployment (Byun et al., 14 Oct 2025).

This broader literature also clarifies what DSSD is not. It is not a tree-structured adaptive topology method like DySpec, not a quantized multi-level verifier hierarchy like ML-SpecQD, and not a purely memory-management system like SpecMemo. DSSD’s distinctive contribution remains a communication-aware repartitioning of speculative verification itself, specialized to device–edge collaboration (Xiong et al., 2024, Georganas et al., 17 Mar 2025, Yildirim et al., 16 May 2025).

6. Limitations, misconceptions, and open directions

The original DSSD paper is explicit or implicit about several limitations. First, although the method is intended to preserve inference quality by using the standard speculative-decoding residual resampling rule on the device, it does not provide a formal correctness theorem. Second, the implementation is round-based; it does not develop a full asynchronous pipeline or overlap schedule. Third, benefits depend on draft–target alignment: if acceptance is poor, the probability of needing the downlink distribution rises and the advantage shrinks. Fourth, experiments are limited to OPT-family models and throughput-style evaluation rather than broader downstream quality benchmarks (Ning et al., 16 Jul 2025).

A broader systems caveat emerges from later analysis of edge–cloud speculative decoding. “Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off” argues that distributed speculative decoding is usually not a single-request WAN latency win when compared with co-located speculative decoding, and that its stronger case is multi-tenant capacity rather than per-request latency. This suggests that DSSD, too, should be evaluated not only on one-request speedup but also on concurrency, server utilization, and the conditions under which communication savings translate into actual serving gains (Lyu et al., 23 Jun 2026).

The most immediate open direction is communication-efficient proposal encoding. Two later papers, “Communication-Efficient Collaborative LLM Inference via Distributed Speculative Decoding” and “Fast Collaborative Inference via Distributed Speculative Decoding,” propose sparse proposal transmission—Top-$i=1,\dots,\gamma$6 sparse logits transmission and TSLT, respectively—to reduce the uplink burden of sending draft distributions. Their results show that acceptance-rate deviation can be bounded by discarded tail mass. This suggests that split verification and sparse proposal transmission are complementary rather than mutually exclusive: one reduces the need to send full distributions every round, and the other reduces the size of whatever proposal-side distribution information still must cross the network (Zheng et al., 4 Sep 2025, Zheng et al., 18 Dec 2025).

Another misconception is that DSSD exhausts the design space of distributed split speculative decoding. It does not. Open questions remain on adaptive draft length, asynchronous overlap, heterogeneous model/tokenizer support, and multi-stage or multi-tenant verification. Later work on batching, one-for-many draft sharing, hierarchical verification, and fine-grained phase overlap indicates that DSSD is best understood as one point in a larger design space whose central variables are where drafting occurs, where verification occurs, what information crosses the network, and how speculative errors are corrected (Xu et al., 22 Apr 2026, He et al., 29 Jan 2026, Chen et al., 22 Apr 2026, Byun et al., 14 Oct 2025).