---
title: Distributed Split Speculative Decoding
url: https://www.emergentmind.com/topics/distributed-split-speculative-decoding-dssd
type: topic
---

# Distributed Split Speculative Decoding

Searching arXiv for the primary DSSD paper and closely related distributed speculative decoding work.
Searching for "Distributed Split Speculative Decoding" and related edge/cloud speculative decoding papers on arXiv.
Distributed Split Speculative Decoding (DSSD) is a device–edge collaborative inference architecture in which a small language model \(M_q\) on the device drafts tokens for a large language model \(M_p\) at the base station or edge server, while the verification phase itself is partitioned across the two sides. In this design, **accept/reject** is performed at the edge and **resample** is performed on the device. DSSD is introduced to preserve the SLM–LLM split of distributed speculative decoding while replacing the uplink transmission of multiple vocabulary distributions with a single downlink transmission, thereby reducing communication latency while maintaining inference quality [2507.12000].

## 1. System model and motivation

DSSD is formulated for a **device–edge system** in which the device hosts an SLM \(M_q\) and the edge hosts an LLM \(M_p\). The paper assumes that the two models share a common vocabulary \(\mathcal V\). The motivation is that modern LLMs are too heavy to run efficiently on mobile or resource-constrained devices, while edge-only execution introduces network delay and reliability problems. The resulting design problem is therefore not only model execution, but the joint balance of local computation and network communication [2507.12000].

The immediate precursor in this line is distributed speculative decoding in which the device-side drafter and edge-side verifier preserve the classical draft–target division but require the device to send full draft-side distributions for verification. In that broader edge–cloud formulation, the draft role is split to the edge and the verification role to the cloud, but the protocol remains a standard draft-then-verify loop [2511.21669]. DSSD targets the specific communication bottleneck of that arrangement: for draft length \(\gamma\), the uplink payload associated with transmitting draft-model vocabulary distributions scales as \(\gamma \cdot |\mathcal V|\) [2507.12000].

The architectural distinction between DSD and DSSD is concise.

| Aspect | DSD | DSSD |
|---|---|---|
| Drafting location | Device-side SLM | Device-side SLM |
| Accept/reject location | Edge | Edge |
| Resampling location | Edge | Device |
| Heavy full-distribution transfer | \(\gamma\) uplink distributions | At most one downlink distribution on rejection |

This means DSSD is not a conventional layer-partitioned split-computing system. Rather, it preserves the two-model speculative-decoding structure and **splits the verification phase itself**. That split is the defining feature of the method [2507.12000].

## 2. Protocol and split verification procedure

DSSD keeps the standard speculative-decoding drafting stage on the device. Starting from the current prefix, the device initializes \(\mathbf y=[]\) and, for \(i=1,\dots,\gamma\), computes
\[
Q_i(x) \leftarrow M_q(\text{prefix}+\mathbf y), \qquad x_i \sim Q_i(x),
\]
then appends \(x_i\) to \(\mathbf y\). After this step, the device has the drafted tokens \([x_1,\dots,x_\gamma]\) and the local draft distributions \([Q_1(x),\dots,Q_\gamma(x)]\) [2507.12000].

The compact uplink message does **not** contain the full \(Q_i(x)\) vectors. Instead, the device sends the drafted token identities together with the scalar probabilities of those selected tokens, written in the paper as the probability values \([q_1(x),\dots,q_\gamma(x)]\). From context, these are the probabilities of the drafted tokens under the SLM, i.e. \(q_j(x_j)\). At the edge, the LLM computes the verification distributions
\[
P_1(x),\dots,P_{\gamma+1}(x)
\]
conditioned on the prefix and the drafted continuation. Verification then proceeds sequentially. For position \(j\), the edge samples
\[
r_j \sim U[0,1]
\]
and accepts the draft token if
\[
r_j < \min\left\{1,\frac{q_j(x_j)}{p_j(x_j)}\right\}.
\]
If all \(\gamma\) draft tokens are accepted, the edge samples
\[
x_{\gamma+1}\sim P_{\gamma+1}(x).
\]

The crucial split occurs when rejection happens. If the first rejection is at position \(j\), the edge sends back \(P_j(x)\) and the index \(j\). The device then performs the corrective resampling locally using its stored \(Q_j(x)\):
\[
x'_j \sim \mathrm{norm}\left(\max(0, P_j(x)-Q_j(x))\right),
\]
and replaces \(x_j\) by \(x'_j\). The prefix is then extended by \([x_1,\dots,x_j]\). If all draft tokens were accepted, the edge instead returns \(x_{\gamma+1}\) and \(j=\gamma+1\), and the prefix is extended accordingly [2507.12000].

The paper notes one scheduling detail in the appendix: when rejection occurs, the resampled token still needs to be uploaded to the edge, but this can be deferred to the next draft–verify round. This suggests a limited form of deferred synchronization, although DSSD is otherwise presented as a round-based protocol rather than an explicitly asynchronous pipeline [2507.12000].

A common misunderstanding is to treat DSSD as ordinary split inference across model layers. The method is not described that way. Its novelty is the relocation of **resampling** from the edge to the device, so that the edge only needs enough draft-side information to run accept/reject, while the device reuses its local \(Q_j(x)\) to construct the residual resampling distribution [2507.12000].

## 3. Communication and latency model

The paper models total communication time as
\[
T_{comm} = T_{up} + T_{down} + T_{NTT},
\]
where \(T_{NTT}\) denotes non-transmission time. For the earlier DSD baseline, the dominant term is the uplink of \(\gamma\) full draft distributions:
\[
D_{up} = \gamma \cdot |\mathcal V| \cdot b_{prob},
\]
which yields
\[
T_{comm} = \gamma \cdot \frac{|\mathcal V| b_{prob}}{R_{up}} + T_{NTT}.
\]
The corresponding inference latency per round is
\[
T_{inf} = \gamma \cdot T_{SLM} + T_{LLM} + \gamma \cdot \frac{|\mathcal V|b_{prob}}{R_{up}} + T_{NTT}.
\]
This is the communication regime DSSD is designed to avoid [2507.12000].

For DSSD, the expensive vocabulary-sized transmission moves to the downlink and occurs only if at least one drafted token is rejected. Let \(a=1\) if there is a rejection and \(a=0\) otherwise. Then
\[
T_{down} = a \cdot \frac{|\mathcal V|b_{prob}}{R_{down}}.
\]
If \(\alpha\) denotes the per-token acceptance probability and independence across the \(\gamma\) drafted positions is assumed in the analysis, then the probability that all \(\gamma\) tokens are accepted is
\[
\alpha^\gamma.
\]
Hence the expected DSSD communication time becomes
\[
T_{comm} = (1-\alpha^\gamma)\cdot \frac{|\mathcal V|b_{prob}}{R_{down}} + T_{NTT},
\]
and the inference latency is
\[
T_{inf} = \gamma \cdot T_{SLM} + T_{LLM} + (1-\alpha^\gamma)\cdot \frac{|\mathcal V|b_{prob}}{R_{down}} + T_{NTT}.
\]

The analytical consequence is straightforward. DSD scales communication roughly with \(\gamma |\mathcal V|\) on the uplink, whereas DSSD scales it with \((1-\alpha^\gamma)|\mathcal V|\) on the downlink. The paper also states
\[
T_{NTT} \le T_{comm} \le T_{down}+T_{NTT}.
\]
As \(\gamma \to \infty\), the probability of at least one rejection tends to \(1\), so \(T_{comm}\approx T_{down}+T_{NTT}\). As \(\alpha \to 1\), the probability of needing the downlink distribution vanishes, so \(T_{comm}\approx T_{NTT}\). This is why DSSD is strongest when acceptance is high [2507.12000].

The paper claims that DSSD maintains inference quality because it uses the same speculative-decoding acceptance and residual-resampling mechanism as standard speculative decoding, merely relocating the resampling computation to the device. This suggests exact target-distribution preservation by construction. However, the paper does **not** provide a standalone theorem or formal proof of exact distributional equivalence, so the strongest faithful statement is that exactness is **implied by construction rather than formally established** [2507.12000].

## 4. Experimental characterization

The reported experiments use **OPT-125M** as the device-side draft model and either **OPT-6.7B** or **OPT-13B** as the edge-side target model. Each run starts from a common **128-token narrative prompt**, generates **128 additional tokens**, and uses **top-\(k\) sampling** with
\[
k=10,\qquad \text{temperature}=1.0.
\]
The homogeneous setup uses two **NVIDIA H800** GPUs, one for the draft model and one for the target. The heterogeneous setup uses an **NVIDIA A6000** on the device side and an **NVIDIA H800** at the edge. Network conditions include non-transmission delays of \(0\), \(20\), and \(50\) ms with a bandwidth cap of \(100\) Mbps, and the tables additionally include \(50\) Mbps and \(10\) Mbps settings [2507.12000].

The communication reduction is extreme in the reported measurements. The paper states that DSD uploads approximately **61,269 bytes per round**, whereas DSSD uploads **less than 50 bytes each time**. This matches the protocol change from full-vocabulary uplink distributions to drafted token indices plus scalar probabilities, with at most one vocabulary distribution returned on rejection [2507.12000].

The measured acceptance rates are approximately
\[
\alpha = 0.61 \pm 0.05
\]
for OPT-125M \(\rightarrow\) OPT-6.7B and
\[
\alpha = 0.53 \pm 0.05
\]
for OPT-125M \(\rightarrow\) OPT-13B. In the homogeneous experiments discussed in the text, DSSD speedup is roughly **\(1.5\times\) to \(2.4\times\)**, whereas DSD often remains near **\(1\times\)** or below it once communication cost is included. Representative points include, for \(\gamma=8\), \(20\) ms, \(100\) Mbps:
- DSD (125M \(\rightarrow\) 6.7B): **1.41**
- DSSD (125M \(\rightarrow\) 6.7B): **2.31**

and for \(\gamma=8\), \(50\) ms, \(10\) Mbps:
- DSD: **0.43**
- DSSD: **2.19** [2507.12000]

In the heterogeneous experiments, the paper defines
\[
c = \frac{T_{SLM}}{T_{LLM}},
\]
with reported values \(c\approx 0.1\) for 125M \(\rightarrow\) 6.7B and \(c\approx 0.05\) for 125M \(\rightarrow\) 13B. Under these more realistic conditions, speedups are more sensitive to \(\gamma\) and target size. The paper reports speedups up to roughly **\(1.4\times\)** for OPT-6.7B and **\(1.8\times\)** for OPT-13B. It also reports an interior optimum in the speculative length: under \(20\) ms and \(100\) Mbps, \(\gamma=6\) performs best, whereas \(\gamma=4\) underutilizes speculation and \(\gamma=8\) increases mismatch and overhead. This directly contradicts the misconception that speculative length should simply be made as large as possible [2507.12000].

## 5. Relation to adjacent distributed and speculative decoding systems

Subsequent work broadened the distributed speculative-decoding design space in several directions, but not by replacing DSSD’s core communication split. The later edge–cloud framework “DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving” places a lightweight draft model on edge devices and a heavyweight target model in the cloud, then studies routing, batching, and an Adaptive Window Control policy through a discrete-event simulator. Its novelty is a systems framework for multi-device deployments rather than DSSD’s relocation of resampling from edge to device [2511.21669].

Other systems extend the split-execution idea to multi-user or one-to-many settings. “DiP-SD: Distributed Pipelined Speculative Decoding for Efficient LLM Inference at the Edge” studies a multi-user device–edge regime in which many devices draft locally and a centralized edge server performs batched verification; its emphasis is joint optimization of batch count, user-to-batch assignment, and draft lengths under a throughput-maximization objective. “StarSD: One-for-Many Speculative Decoding” instead uses a single draft model to serve multiple target models via a star topology, with a global request buffer and per-session state isolation, so its central question is draft sharing and utilization rather than DSSD’s split-verification communication rule [2604.20919] [2601.21622].

At the algorithmic level, several papers contribute abstractions that are relevant to DSSD without being distributed DSSD methods themselves. “FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving” shows that dynamic online workloads benefit from per-request speculative length adaptation, token-wise early pruning, and chunked frontier verification that overlaps with drafting; these ideas map naturally to split or disaggregated pipelines, although the paper is primarily a local-runtime system rather than an edge–cloud design [2604.20503]. “3-Model Speculative Decoding” introduces Pyramid Speculative Decoding, a hierarchical draft \(\rightarrow\) qualifier \(\rightarrow\) target cascade with fuzzy acceptance thresholds. It does not study distributed placement, but it offers a transferable **hierarchical verification abstraction** that could serve as a multi-stage verifier in a split deployment [2510.12966].

This broader literature also clarifies what DSSD is **not**. It is not a tree-structured adaptive topology method like DySpec, not a quantized multi-level verifier hierarchy like ML-SpecQD, and not a purely memory-management system like SpecMemo. DSSD’s distinctive contribution remains a communication-aware repartitioning of speculative verification itself, specialized to device–edge collaboration [2410.11744] [2503.13565] [2506.01986].

## 6. Limitations, misconceptions, and open directions

The original DSSD paper is explicit or implicit about several limitations. First, although the method is intended to preserve inference quality by using the standard speculative-decoding residual resampling rule on the device, it does **not** provide a formal correctness theorem. Second, the implementation is round-based; it does not develop a full asynchronous pipeline or overlap schedule. Third, benefits depend on draft–target alignment: if acceptance is poor, the probability of needing the downlink distribution rises and the advantage shrinks. Fourth, experiments are limited to OPT-family models and throughput-style evaluation rather than broader downstream quality benchmarks [2507.12000].

A broader systems caveat emerges from later analysis of edge–cloud speculative decoding. “Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off” argues that distributed speculative decoding is usually **not** a single-request WAN latency win when compared with co-located speculative decoding, and that its stronger case is multi-tenant capacity rather than per-request latency. This suggests that DSSD, too, should be evaluated not only on one-request speedup but also on concurrency, server utilization, and the conditions under which communication savings translate into actual serving gains [2606.25091].

The most immediate open direction is communication-efficient proposal encoding. Two later papers, “Communication-Efficient Collaborative LLM Inference via Distributed Speculative Decoding” and “Fast Collaborative Inference via Distributed Speculative Decoding,” propose sparse proposal transmission—Top-\(K\) sparse logits transmission and TSLT, respectively—to reduce the uplink burden of sending draft distributions. Their results show that acceptance-rate deviation can be bounded by discarded tail mass. This suggests that **split verification** and **sparse proposal transmission** are complementary rather than mutually exclusive: one reduces the need to send full distributions every round, and the other reduces the size of whatever proposal-side distribution information still must cross the network [2509.04576] [2512.16273].

Another misconception is that DSSD exhausts the design space of distributed split speculative decoding. It does not. Open questions remain on adaptive draft length, asynchronous overlap, heterogeneous model/tokenizer support, and multi-stage or multi-tenant verification. Later work on batching, one-for-many draft sharing, hierarchical verification, and fine-grained phase overlap indicates that DSSD is best understood as one point in a larger design space whose central variables are **where drafting occurs, where verification occurs, what information crosses the network, and how speculative errors are corrected** [2604.20919] [2601.21622] [2604.20503] [2510.12966].

Source: https://www.emergentmind.com/topics/distributed-split-speculative-decoding-dssd