---
title: 'DSD-Sim: Distributed Speculative Decoding Simulator'
url: https://www.emergentmind.com/topics/dsd-sim
type: topic
---

# DSD-Sim: Distributed Speculative Decoding Simulator

DSD-Sim is a request-level discrete-event simulator for distributed speculative decoding in heterogeneous edge-cloud large language model serving. It is introduced in “DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving” as the simulator used to study how speculative decoding behaves when draft and target models are placed on different devices, with explicit modeling of request routing, queueing, batching, network round-trip delays and jitter, heterogeneous device performance, and the iterative draft–verify loop that distinguishes distributed speculative decoding from single-node inference [2511.21669].

## 1. Definition and scope

Within the paper, DSD-Sim is not a generic simulator for LLM inference or a general distributed systems simulator. It is defined specifically for distributed speculative decoding, where smaller draft models run on edge devices and larger target models run on cloud servers. A request arrives at an edge node, the draft model proposes a block of speculative tokens, those tokens are sent to a remote target model for verification, and accepted tokens are committed while mismatches trigger correction and continuation from the verified prefix [2511.21669].

The paper distinguishes two execution modes. In **fused mode**, draft and target are co-located on the same server and the full speculation loop is local. In **distributed mode**, the draft device generates a speculative block with window size \(y>1\), sends it to a remote target device, and pays explicit communication cost on every speculative iteration. DSD-Sim is designed to compare these modes under realistic system load and network conditions rather than under isolated single-request assumptions [2511.21669].

A common source of confusion is acronym overlap. The exact name **DSD-Sim** appears in the distributed speculative decoding paper, whereas other recent arXiv works use related acronyms for different topics, including **DSD-GS** for Dynamic-Static Decomposition of Gaussian Splatting [2605.30863], **DSD** for Dual Self-Distillation in volumetric segmentation [2306.03271], **DSD** for densest subgraph discovery [2306.07927], and **DSD** for drop-size distribution in THz rain attenuation studies [2604.15835]. The VLT optical bench paper refers instead to **DMS**, **DSM Simulator**, and **DSMSim**, not DSD-Sim [2207.14099].

## 2. Architectural decomposition

The simulator is organized into four major components: a **configuration parser**, a **core DSD scheduler**, a **hardware performance modeling engine**, and a **performance analyzer** [2511.21669].

| Component | Role |
|---|---|
| Configuration parser | Reads YAML system configuration and expands topology |
| Core DSD scheduler | Simulates routing, queueing, batching, drafting, verification, and mode control |
| Hardware performance engine | Predicts operation latency through a VIDUR-based API |
| Performance analyzer | Records SLO and system-level metrics and emits structured JSON |

The configuration parser ingests device types, model assignments, hardware, network links, and policy settings. It also performs an **auto_topology** expansion from a high-level topology to explicit draft pools, target pools, and connectivity. This is important because the paper evaluates large heterogeneous deployments rather than a single fixed cluster shape [2511.21669].

The core scheduler leverages **SimPy** and represents draft servers and target servers as concurrent processes with explicit queues for batch formation and scheduling. The paper therefore models DSD-Sim as a discrete-event system whose main entities are requests, batches, draft workers, target workers, queues, and network links. Its role is to produce request-level timing under interacting scheduling and communication effects rather than to emulate GPU kernels directly [2511.21669].

The hardware performance engine uses a predictor exposed as `predict(op, shape, hardware)`, derived from **VIDUR** and extended with profiling data for additional devices and models, including NVIDIA A40 and edge-oriented LLMs such as Qwen-7B and Llama2-7B. This means node-local service time is predicted rather than physically simulated, while network and queueing behavior remain explicit in the event loop [2511.21669].

## 3. Simulation semantics and workload model

DSD-Sim supports both **trace-driven arrivals** and **synthetic arrivals** generated by a **Poisson process**. In the synthetic setting, arrivals are generated globally and uniformly distributed across drafter devices. In the trace-driven setting, the workload record contains at least `prompt_length`, `output_length`, `acceptance_seq`, `arrival_time_ms`, and `drafter_id` [2511.21669].

| Trace field | Meaning |
|---|---|
| `prompt_length` | Prompt length |
| `output_length` | Output length |
| `acceptance_seq` | Acceptance sequence captured from profiling |
| `arrival_time_ms` | Request arrival timestamp |
| `drafter_id` | Assigned drafter identity |

A distinctive modeling choice is that speculative acceptance is not synthesized from a generic probability law during simulation. Instead, the trace embeds a ground-truth **acceptance sequence** captured from hardware profiling runs for a given draft–target pair. This gives DSD-Sim a replay-style treatment of accept/reject behavior while still simulating queueing, batching, and network delay at system scale [2511.21669].

The simulated request lifecycle is explicit. A request is routed to a target cluster, enqueued for batching, assigned a speculation window \(y\) and an execution mode, processed by the draft model, transmitted if distributed, verified by the target, and then either advanced or corrected before the next speculative round. Completion occurs when the requested output length is reached or EOS is generated. The simulator therefore models the fundamental synchronization dependency of speculative decoding: the next draft step depends on target verification of the current speculative prefix [2511.21669].

Network links are modeled as delay elements with **RTT** and **jitter**. Queueing and batching are configurable through **FIFO**, **Length-Aware Batching (LAB)**, optional **continuous batching**, and optional **chunked prefills**. Routing policies include **Random**, **Round Robin**, and **Join-the-Shortest-Queue (JSQ)**. The main request-level metrics are **TTFT**, **TPOT**, and end-to-end latency; the main system-level metrics are throughput, target utilization, and aggregate queueing or network delay [2511.21669].

## 4. Role in adaptive control and DSD design

DSD-Sim is not used only as an evaluation backend. It is the mechanism through which the paper derives and trains **Adaptive Window Control (AWC)** for the distributed speculative decoding system. The paper states that DSD-Sim is used to generate labels by exhaustive simulation sweeps over workload traces, hardware deployments, network settings, all window sizes \(y=2,\ldots,12\), and fused mode, covering more than 2,000 scenarios [2511.21669].

The learned controller, denoted **WC-DNN**, uses a five-dimensional feature vector: recent target queue utilization, recent acceptance rate, recent RTT statistics, recent TPOT statistics, and prior window size. Its architecture is a residual MLP with two blocks and SiLU activation, trained with **L1 loss** and **AdamW** for **100 epochs**. The output is a scalar prediction of the speculation window. Post-processing includes clamping to a configured interval such as \([1,12]\), **EMA smoothing** with \(\beta=0.4\), and hysteresis-based fused-mode switching when the smoothed prediction remains near \(y=1\) for at least \(k=2\) consecutive steps [2511.21669].

This suggests that the simulator’s central contribution is not only descriptive but also policy-generative. By exposing the interaction among acceptance rate, network delay, queue depth, and target utilization, DSD-Sim becomes the training environment for a systems controller rather than merely a passive measurement tool.

## 5. Validation and empirical findings

The paper reports component-level validation for the hardware modeling engine by comparing predicted and measured latencies for Qwen-7B, Qwen-72B, Llama2-7B, and Llama2-70B on A40, A100, and H100 hardware. The reported errors are **7.4% mean absolute error** for prefill latency and **5.2% mean absolute error** for decode latency. The paper also states that predicted latencies are consistently lower than measured real times because VIDUR models MLP and Attention kernel times but omits NCCL communication overhead and non-kernel overheads [2511.21669].

The workloads used in the simulator are **GSM8K**, **CNN/DailyMail**, and **HumanEval**, representing reasoning-heavy, summarization, and code-generation traffic. Calibration and evaluation center on representative RTT settings of **10 ms** and **30 ms**, motivated by public cloud measurements, while a broader sweep shows that distributed mode loses to fused or cloud-only operation at around **50–60 ms RTT** [2511.21669].

The main system-level claim is that the full DSD stack, informed by DSD-Sim and including AWC, achieves up to **1.1x speedup** and **9.7% higher throughput** over existing speculative decoding baselines. In the AWC-specific comparison, the paper reports that AWC achieves the best throughput in **all 12** tested configurations, outperforming static control by **3–10%** and outperforming the heuristic dynamic baseline in **11/12** scenarios. In one policy-stack example on GSM8K, throughput improves from **25.1** to **28.1 req/s**, TTFT decreases from **351 ms** to **345 ms**, and TPOT decreases from **45 ms** to **37 ms** [2511.21669].

The simulator also yields several system-design conclusions. JSQ is strongest under moderate load but can worsen TPOT near saturation; LAB improves TPOT but not the throughput ceiling; and fixed speculation windows are fragile because large \(y\) increases rollback risk while small \(y\) increases verification frequency and communication overhead. These are presented as simulation-derived insights that motivate adaptive distributed speculative decoding rather than as generic properties of all LLM serving systems [2511.21669].

## 6. Limitations and interpretation

The paper does not present DSD-Sim as a cycle-accurate simulator. It is a request-level abstraction whose network model is parameterized mainly by RTT and jitter, without an explicit bandwidth model in the reported description. Likewise, the hardware model depends on VIDUR-derived latency prediction and does not model GPU microarchitecture, detailed kernel overlap, or full-system communication overheads such as NCCL in a first-principles way [2511.21669].

Another limitation is that acceptance behavior is replayed from profiled traces rather than generated from an explicit probabilistic acceptance model. This improves realism for the evaluated draft–target pairs but may constrain extrapolation to unseen pairs or policies. The paper also validates components rather than reporting a full end-to-end correspondence between DSD-Sim and a deployed distributed speculative decoding system [2511.21669].

Even with those constraints, DSD-Sim occupies a specific methodological niche. It is a simulator for the interaction among speculative verification semantics, queueing, batching, heterogeneous deployment, and network delay in edge-cloud LLM serving. Its significance lies in showing that distributed speculative decoding is not reducible to a single-node speedup formula: it is a coupled scheduling and communication problem in which optimal behavior depends jointly on workload, acceptance dynamics, network conditions, and control policy [2511.21669].

Source: https://www.emergentmind.com/topics/dsd-sim