SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

Abstract: LLMs have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.

• The paper introduces SPADE, a plug-and-play edge–cloud speculative decoding framework that uses a local draft model and parallel cloud verification to preserve the verifier’s sampling distribution without retraining.
• SPADE reduces cloud verifier calls by 77.4% on Spec-Bench and 76% on CNN/DailyMail, cuts cloud runtime to roughly 23–24% of the target-only baseline, and improves throughput.
• SPADE maintains near-target quality, scoring 4.38 versus 4.45 on Spec-Bench and achieving 23.92 versus 24.32 ROUGE-L on CNN/DailyMail, but savings depend on draft–verifier alignment and network costs.

Motivation and problem statement

Deploying LLMs at scale forces a trade-off between output quality and inference cost. Compressed variants—pruned, quantized, or distilled models—run on edge hardware but consistently underperform their full-scale counterparts, while cloud-hosted full models preserve quality but incur per-token autoregressive computation that scales with generation length. The paper frames the central question as how to reduce reliance on the cloud while retaining the accuracy of the full model, using only edge resources.

The proposed answer, SPADE, is a distributed inference framework that maps speculative decoding (SD) onto an edge–cloud topology: a compact draft model runs locally on the edge device, and a large verifier model hosted in the cloud validates candidate tokens in parallel. Because SD's verification step processes a block of $d$ candidate tokens in a single forward pass, each cloud invocation can confirm many tokens at once, directly reducing the number of billable, latency-inducing verifier calls.

Method

SPADE instantiates the standard two-model SD pipeline of Leviathan et al. [2307.XXXX, cited as leviathan2023fast] without modification to its acceptance rule. A draft model $\mathcal{M}_q$ on the edge autoregressively generates a block of $d$ candidate tokens conditioned on the current prefix. These are transmitted to the cloud, where the verifier $\mathcal{M}_p$ scores all candidates in one forward pass. Each candidate $x \sim q(x)$ is accepted with probability

$\alpha(x) = \min\left(1, \frac{p(x)}{q(x)}\right),$

and upon the first rejection, a replacement token is drawn from the residual distribution $p'(x) = \text{norm}(\max(0, p(x) - q(x)))$. This guarantees the emitted sequence is statistically identical to verifier-only decoding, so SPADE inherits the losslessness property of SD by construction rather than by empirical approximation. At least one token is appended per iteration, ensuring forward progress; generation terminates when the verifier accepts an end-of-sequence token.

A key design point is the treatment of the draft length $d$ as a systems hyperparameter governing the compute–communication trade-off: small $d$ makes the pipeline communication-bound due to frequent synchronization, while large $d$ increases rejection likelihood and wasted edge computation. The authors select $\mathcal{M}_q$0 empirically from acceptance rates on roughly ten validation samples. The framework is fully plug-and-play: no retraining, distillation, or architectural modification of either model is required, distinguishing it from self-speculative approaches such as LayerSkip (Elhoushi et al., 2024).

Experimental setup

The evaluation uses LLaMA-3.2-1B as the edge draft model and LLaMA-3.1-8B as the cloud verifier [2407.XXXX], with an RTX 3080 (12 GB) representing the edge and an RTX A6000 (48 GB) representing industrial-scale cloud infrastructure. Two benchmarks are used: Spec-Bench (Xia et al., 2024), covering six subtasks (multi-turn conversation, translation, summarization, QA, mathematical reasoning, RAG), scored 1–5 by Gemini-2.5-Flash-Lite as an LLM judge across six qualitative dimensions; and CNN/DailyMail summarization, evaluated with BLEU, ROUGE, and CIDEr-D. Baselines are the target-only cloud model (quality upper bound) and the draft-only edge model (cost lower bound).

Results

On Spec-Bench, SPADE reduces mean target model calls from 133.25 to 30.16—a 77.4% reduction—with cloud runtime dropping to 0.23× of the target-only baseline, while throughput improves from 2.43 to 3.25 tokens/s. The overall judged score is 4.38 versus 4.45 for the full model, with per-task gaps of at most 0.20 (mathematical reasoning: 4.68 vs. 4.88). On CNN/DailyMail, cloud calls fall from 127.30 to 30.79 (76% reduction, runtime 0.24×), throughput rises from 1.21 to 1.95 tokens/s, and ROUGE-L reaches 23.92 versus 24.32 for the target model. Notably, CIDEr-D exceeds the target model (3.19 vs. 2.50), though this likely reflects metric sensitivity to n-gram overlap rather than genuine quality gains, and the paper does not investigate it.

Metric	Target (cloud)	Draft (edge)	SPADE
Spec-Bench overall score	4.45	3.39	4.38
Mean target model calls	133.25	0	30.16
Cloud runtime	1.00×	—	0.23×
Throughput (tokens/s)	2.43	3.91	3.25
CNN/DM ROUGE-L F1	24.32	22.49	23.92
CNN/DM mean target calls	127.30	0	30.79

An ablation on $\mathcal{M}_q$1 shows that increasing the draft length monotonically reduces verifier invocations, amplified by the strong alignment between LLaMA-3.2-1B and LLaMA-3.1-8B. The authors acknowledge the converse: poor draft–verifier alignment would depress acceptance rates and erode the savings, meaning the reported reductions are contingent on model-family compatibility between draft and verifier.

Limitations and open questions

Several caveats qualify the results. First, the evaluation uses a single draft–verifier pair from the same model family; the framework's cost savings under heterogeneous or weaker-aligned pairs remain unquantified, despite the authors' own observation that misalignment increases target calls. Second, the claimed "zero loss in accuracy" holds exactly for the sampling distribution but not empirically—the judged scores show small consistent deficits (e.g., 4.38 vs. 4.45 overall)—so the claim should be read as distributional equivalence, not identical outputs in practice. Third, the analysis does not report network latency or bandwidth costs of transmitting draft blocks, which would be material in realistic edge deployments with constrained uplink. Fourth, the hyperparameter $\mathcal{M}_q$2 is tuned on a small validation subset without a principled adaptive scheme; whether $\mathcal{M}_q$3 should vary dynamically with observed acceptance rates is left open. Finally, the LLM-as-a-judge protocol introduces evaluator-dependent variance that the paper does not quantify against human judgments.

Conclusion

SPADE demonstrates that speculative decoding transfers naturally to a distributed edge–cloud setting, cutting cloud verifier invocations by roughly 76–77% across two benchmarks while keeping output quality within a small margin of the full model, with no retraining. Its practical value rests on the plug-and-play property and on draft–verifier alignment; characterizing robustness across model pairs, incorporating communication costs into the optimization, and adaptively scheduling the draft length are the concrete questions the paper leaves unresolved.