Jump-Share Speculative Decoding

• The paper presents a novel distributed inference strategy that separates token drafting from verification to maximize throughput and resource utilization.
• It employs a star-topology design to coordinate a centralized draft model with multiple verifiers, optimizing latency and scalability.
• Experimental results demonstrate significant performance gains and offer analytical models for effective infrastructure planning and deployment.

Jump-Share Speculative Decoding is a distributed inference strategy that extends speculative decoding techniques for autoregressive generation by leveraging a star topology to maximize resource utilization and throughput in large-scale LLM serving. The approach separates token proposal (drafting) from verification, enabling a centralized lightweight draft model to serve many target (verifier) models concurrently across heterogenous multi-accelerator clusters. This framework, most notably instantiated as StarSD, addresses scalability and deployment challenges in modern cloud and edge LLM inference, providing analytical models for utilization, response latency, and bottleneck prediction (He et al., 29 Jan 2026).

1. Core Principles of Speculative Decoding

Speculative decoding decomposes the traditional sequential generation process of autoregressive LLMs into parallelizable proposal and verification phases. The procedure involves:

• A draft model $M_q$ proposes up to $d$ tokens per generation iteration.
• A target model $M_p$ verifies the proposals in a batched forward pass, accepting the longest prefix matching $M_p$'s true conditional next-token distribution.

Given a desired output of $N_\mathrm{tokens}$, let $T_\mathrm{base}$ denote time with vanilla (non-speculative) target inference, and $T_\mathrm{spec}$ denote the speculative regime. The expected speedup is:

$S = \frac{T_\mathrm{base}}{T_\mathrm{spec}}$

Within each speculative step, the drafting cost is $S(d)=d\,t_s$ where $t_s$ is the draft model's per-token time, and the verification "return" cost is $d$0 (communication and verification). The overlap and synchronization requirements between these two phases fundamentally shape system utilization and scalability.

2. Star Topology and Pipeline Architecture

Jump-Share Speculative Decoding, as realized in the StarSD framework, adopts a star topology: a single draft model ($d$1) centrally located, serving $d$2 independent target models ($d$3) at peripheral nodes. Each $d$4 communicates over a dedicated channel to request, receive, and verify draft token proposals.

$d$5 manages:

• A global incoming request buffer $d$6 for session-tagged verified prefixes.
• A global outgoing buffer $d$7 for drafted tokens.
• Per-session key-value caches $d$8.

A multi-threaded, nonblocking pipeline orchestrates:

• A receiver for inbound verifications.
• A draft-inference thread generating tree-structured proposals to depth $d$9.
• A sender thread dispatching draft tokens to the corresponding targets.

Drafting and communication are decoupled, ensuring that the draft server is never idle while work is available, and each target proceeds only after a verification step.

3. System-Level Utilization Analysis

Server-side resource utilization is modeled analytically, predicting full-utilization regimes and optimal design points:

• The per-iteration wall-clock time:

$M_p$0

• Draft-server utilization:

$M_p$1

Utilization is maximized ($M_p$2) when the return cost $M_p$3 does not exceed the drafting work of all other verifiers,

$M_p$4

• Full-utilization threshold:

$M_p$5

A practical implication is that scaling the number of verifiers beyond $M_p$6 induces queueing delays rather than greater throughput, as the centralized draft service becomes the system bottleneck. The framework supports analysis of resource allocation trade-offs and serves as a predictive tool for infrastructure planning (He et al., 29 Jan 2026).

4. Experimental Methodology and Performance Results

Evaluation employed a cluster with one RTX 4090 GPU draft server ($M_p$7), and up to six target servers (three RTX 4090 + three A100 GPUs) running Vicuna-7B models. The draft model used was EAGLE-2 (∼1.3B parameters). The MT-Bench suite provided standard workloads, using a tree depth $M_p$8 and top-$M_p$9 proposals per layer.

Key findings include:

• Increasing $M_p$0 from 1 to 6 target instances raised total system throughput from 79.9 to 288.7 tokens/s, with per-target throughput dropping at high $M_p$1 due to increased queueing and load on the draft server.
• Draft service time per proposal iteration remained low (1.36–2.24 ms), and draft-server utilization increased with $M_p$2 (from 33% to ∼99.6%).
• Idle gaps at $M_p$3 shrank as $M_p$4 increased, vanishing for $M_p$5.
• The baseline distributed EAGLE (N=1, separate GPUs) reached 79.9 tok/s, about 15% lower than co-located performance due to cross-device overheads.

A summary table:

N (targets)	System Throughput (tok/s)	Draft Load (%)	Idle Gap (ms)
1	79.9	33.3	30.6
3	246.3	75.4	3.4
6	288.7	99.6	0.0

This demonstrates that appropriate scaling and configuration of the draft/verify ratio directly optimize system performance (He et al., 29 Jan 2026).

5. Runtime Design and Best Practices

Guidelines for deploying Jump-Share Speculative Decoding systems:

• Centralize the draft model ($M_p$6) as a shared cluster service; do not co-locate with any one target to conserve VRAM for more large model deployments.
• Implement work-conserving loops and global request buffers, strictly isolating key-value caches by session tag.
• Use the analytical model for admission control, ensuring $M_p$7 but capping at $M_p$8 to avoid diminishing per-target throughput.
• Maintain continuous request streams to keep accelerators active and avoid latency spikes, inserting warm-up bursts after idling when necessary.
• Bound worst-case latency by controlling speculation depth $M_p$9 and closely monitoring communication and verification time parameters.

6. Significance and Future Directions

Jump-Share Speculative Decoding, through frameworks like StarSD, makes it feasible to efficiently share lightweight draft computation across distributed LLM infrastructure, absorbing bursty traffic, and supporting heterogeneous hardware. The analytical model provides guarantees on throughput and utilization, simplifying deployment and tuning in both cloud and edge contexts.

A plausible implication is that further research may explore adaptive draft selection, dynamic scheduling across variable workloads, and integration with advanced speculative pipeline techniques. Extending these principles to serve wider classes of generative models and inference paradigms is a cited opportunity (He et al., 29 Jan 2026).