DSD-Sim: Distributed Speculative Decoding Simulator

• DSD-Sim is a simulator specifically designed for distributed speculative decoding, where draft models operate on edge devices and target models run in the cloud.
• It incorporates a SimPy-based core scheduler that models routing, queueing, batching, and an iterative draft–verify loop to capture realistic request-level timing.
• The simulator supports both trace-driven and synthetic Poisson workloads, enabling adaptive control policy training for improved throughput and reduced latency.

DSD-Sim is a request-level discrete-event simulator for distributed speculative decoding in heterogeneous edge-cloud LLM serving. It is introduced in “DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving” as the simulator used to study how speculative decoding behaves when draft and target models are placed on different devices, with explicit modeling of request routing, queueing, batching, network round-trip delays and jitter, heterogeneous device performance, and the iterative draft–verify loop that distinguishes distributed speculative decoding from single-node inference (Yu et al., 26 Nov 2025).

1. Definition and scope

Within the paper, DSD-Sim is not a generic simulator for LLM inference or a general distributed systems simulator. It is defined specifically for distributed speculative decoding, where smaller draft models run on edge devices and larger target models run on cloud servers. A request arrives at an edge node, the draft model proposes a block of speculative tokens, those tokens are sent to a remote target model for verification, and accepted tokens are committed while mismatches trigger correction and continuation from the verified prefix (Yu et al., 26 Nov 2025).

The paper distinguishes two execution modes. In fused mode, draft and target are co-located on the same server and the full speculation loop is local. In distributed mode, the draft device generates a speculative block with window size $y>1$, sends it to a remote target device, and pays explicit communication cost on every speculative iteration. DSD-Sim is designed to compare these modes under realistic system load and network conditions rather than under isolated single-request assumptions (Yu et al., 26 Nov 2025).

A common source of confusion is acronym overlap. The exact name DSD-Sim appears in the distributed speculative decoding paper, whereas other recent arXiv works use related acronyms for different topics, including DSD-GS for Dynamic-Static Decomposition of Gaussian Splatting (Han et al., 29 May 2026), DSD for Dual Self-Distillation in volumetric segmentation (Banerjee et al., 2023), DSD for densest subgraph discovery (Luo et al., 2023), and DSD for drop-size distribution in THz rain attenuation studies (Song et al., 17 Apr 2026). The VLT optical bench paper refers instead to DMS, DSM Simulator, and DSMSim, not DSD-Sim (Briguglio et al., 2022).

2. Architectural decomposition

The simulator is organized into four major components: a configuration parser, a core DSD scheduler, a hardware performance modeling engine, and a performance analyzer (Yu et al., 26 Nov 2025).

Component	Role
Configuration parser	Reads YAML system configuration and expands topology
Core DSD scheduler	Simulates routing, queueing, batching, drafting, verification, and mode control
Hardware performance engine	Predicts operation latency through a VIDUR-based API
Performance analyzer	Records SLO and system-level metrics and emits structured JSON

The configuration parser ingests device types, model assignments, hardware, network links, and policy settings. It also performs an auto_topology expansion from a high-level topology to explicit draft pools, target pools, and connectivity. This is important because the paper evaluates large heterogeneous deployments rather than a single fixed cluster shape (Yu et al., 26 Nov 2025).

The core scheduler leverages SimPy and represents draft servers and target servers as concurrent processes with explicit queues for batch formation and scheduling. The paper therefore models DSD-Sim as a discrete-event system whose main entities are requests, batches, draft workers, target workers, queues, and network links. Its role is to produce request-level timing under interacting scheduling and communication effects rather than to emulate GPU kernels directly (Yu et al., 26 Nov 2025).

The hardware performance engine uses a predictor exposed as predict(op, shape, hardware), derived from VIDUR and extended with profiling data for additional devices and models, including NVIDIA A40 and edge-oriented LLMs such as Qwen-7B and Llama2-7B. This means node-local service time is predicted rather than physically simulated, while network and queueing behavior remain explicit in the event loop (Yu et al., 26 Nov 2025).

3. Simulation semantics and workload model

DSD-Sim supports both trace-driven arrivals and synthetic arrivals generated by a Poisson process. In the synthetic setting, arrivals are generated globally and uniformly distributed across drafter devices. In the trace-driven setting, the workload record contains at least prompt_length, output_length, acceptance_seq, arrival_time_ms, and drafter_id (Yu et al., 26 Nov 2025).

Trace field	Meaning
prompt_length	Prompt length
output_length	Output length
acceptance_seq	Acceptance sequence captured from profiling
arrival_time_ms	Request arrival timestamp
drafter_id	Assigned drafter identity

A distinctive modeling choice is that speculative acceptance is not synthesized from a generic probability law during simulation. Instead, the trace embeds a ground-truth acceptance sequence captured from hardware profiling runs for a given draft–target pair. This gives DSD-Sim a replay-style treatment of accept/reject behavior while still simulating queueing, batching, and network delay at system scale (Yu et al., 26 Nov 2025).

The simulated request lifecycle is explicit. A request is routed to a target cluster, enqueued for batching, assigned a speculation window $y$ and an execution mode, processed by the draft model, transmitted if distributed, verified by the target, and then either advanced or corrected before the next speculative round. Completion occurs when the requested output length is reached or EOS is generated. The simulator therefore models the fundamental synchronization dependency of speculative decoding: the next draft step depends on target verification of the current speculative prefix (Yu et al., 26 Nov 2025).

Network links are modeled as delay elements with RTT and jitter. Queueing and batching are configurable through FIFO, Length-Aware Batching (LAB), optional continuous batching, and optional chunked prefills. Routing policies include Random, Round Robin, and Join-the-Shortest-Queue (JSQ). The main request-level metrics are TTFT, TPOT, and end-to-end latency; the main system-level metrics are throughput, target utilization, and aggregate queueing or network delay (Yu et al., 26 Nov 2025).

4. Role in adaptive control and DSD design

DSD-Sim is not used only as an evaluation backend. It is the mechanism through which the paper derives and trains Adaptive Window Control (AWC) for the distributed speculative decoding system. The paper states that DSD-Sim is used to generate labels by exhaustive simulation sweeps over workload traces, hardware deployments, network settings, all window sizes $y=2,\ldots,12$, and fused mode, covering more than 2,000 scenarios (Yu et al., 26 Nov 2025).

The learned controller, denoted WC-DNN, uses a five-dimensional feature vector: recent target queue utilization, recent acceptance rate, recent RTT statistics, recent TPOT statistics, and prior window size. Its architecture is a residual MLP with two blocks and SiLU activation, trained with L1 loss and AdamW for 100 epochs. The output is a scalar prediction of the speculation window. Post-processing includes clamping to a configured interval such as $[1,12]$, EMA smoothing with $\beta=0.4$, and hysteresis-based fused-mode switching when the smoothed prediction remains near $y=1$ for at least $k=2$ consecutive steps (Yu et al., 26 Nov 2025).

This suggests that the simulator’s central contribution is not only descriptive but also policy-generative. By exposing the interaction among acceptance rate, network delay, queue depth, and target utilization, DSD-Sim becomes the training environment for a systems controller rather than merely a passive measurement tool.

5. Validation and empirical findings

The paper reports component-level validation for the hardware modeling engine by comparing predicted and measured latencies for Qwen-7B, Qwen-72B, Llama2-7B, and Llama2-70B on A40, A100, and H100 hardware. The reported errors are 7.4% mean absolute error for prefill latency and 5.2% mean absolute error for decode latency. The paper also states that predicted latencies are consistently lower than measured real times because VIDUR models MLP and Attention kernel times but omits NCCL communication overhead and non-kernel overheads (Yu et al., 26 Nov 2025).

The workloads used in the simulator are GSM8K, CNN/DailyMail, and HumanEval, representing reasoning-heavy, summarization, and code-generation traffic. Calibration and evaluation center on representative RTT settings of 10 ms and 30 ms, motivated by public cloud measurements, while a broader sweep shows that distributed mode loses to fused or cloud-only operation at around 50–60 ms RTT (Yu et al., 26 Nov 2025).

The main system-level claim is that the full DSD stack, informed by DSD-Sim and including AWC, achieves up to 1.1x speedup and 9.7% higher throughput over existing speculative decoding baselines. In the AWC-specific comparison, the paper reports that AWC achieves the best throughput in all 12 tested configurations, outperforming static control by 3–10% and outperforming the heuristic dynamic baseline in 11/12 scenarios. In one policy-stack example on GSM8K, throughput improves from 25.1 to 28.1 req/s, TTFT decreases from 351 ms to 345 ms, and TPOT decreases from 45 ms to 37 ms (Yu et al., 26 Nov 2025).

The simulator also yields several system-design conclusions. JSQ is strongest under moderate load but can worsen TPOT near saturation; LAB improves TPOT but not the throughput ceiling; and fixed speculation windows are fragile because large $y$ increases rollback risk while small $y$ increases verification frequency and communication overhead. These are presented as simulation-derived insights that motivate adaptive distributed speculative decoding rather than as generic properties of all LLM serving systems (Yu et al., 26 Nov 2025).

6. Limitations and interpretation

The paper does not present DSD-Sim as a cycle-accurate simulator. It is a request-level abstraction whose network model is parameterized mainly by RTT and jitter, without an explicit bandwidth model in the reported description. Likewise, the hardware model depends on VIDUR-derived latency prediction and does not model GPU microarchitecture, detailed kernel overlap, or full-system communication overheads such as NCCL in a first-principles way (Yu et al., 26 Nov 2025).

Another limitation is that acceptance behavior is replayed from profiled traces rather than generated from an explicit probabilistic acceptance model. This improves realism for the evaluated draft–target pairs but may constrain extrapolation to unseen pairs or policies. The paper also validates components rather than reporting a full end-to-end correspondence between DSD-Sim and a deployed distributed speculative decoding system (Yu et al., 26 Nov 2025).

Even with those constraints, DSD-Sim occupies a specific methodological niche. It is a simulator for the interaction among speculative verification semantics, queueing, batching, heterogeneous deployment, and network delay in edge-cloud LLM serving. Its significance lies in showing that distributed speculative decoding is not reducible to a single-node speedup formula: it is a coupled scheduling and communication problem in which optimal behavior depends jointly on workload, acceptance dynamics, network conditions, and control policy (Yu et al., 26 Nov 2025).