---
title: 'SubSpec: Accelerated Speculative Decoding for LLMs'
url: https://www.emergentmind.com/topics/subspec
type: topic
---

# SubSpec: Accelerated Speculative Decoding for LLMs

Searching arXiv for recent papers on speculative decoding and offloaded LLM inference to ground the article.
arxiv_search(query="speculative decoding offloaded LLM inference speculative decoding acceleration draft model", max_results=10)
arxiv_search(query="LLM speculative decoding offloading quantized draft model KV-cache sharing", max_results=10)
arxiv_search(query="speculative decoding large language models", max_results=10)
SubSpec, short for **Substitute Speculative Decoding**, is a method for accelerating parameter-offloaded large language models by constructing a highly aligned GPU-resident draft model from the target model itself. It is formulated for deployments in which model weights exceed available GPU memory and are therefore offloaded to CPU memory, a regime in which repeated PCIe transfers dominate latency. In the reported formulation, SubSpec is **lossless** because the target LLM’s FP16 weights and outputs remain unchanged, and **training-free** because it requires no extra distillation or fine-tuning. Its central objective is to increase the average token acceptance length $\tau$ under speculative decoding, ideally to approximately $30$ under greedy decoding, so that the number of expensive target forward passes is sharply reduced [2509.18344].

## 1. Computational setting and problem formulation

Modern open-source LLMs such as Llama3 and Qwen2.5 often exceed **8–24 GB** of GPU memory. Parameter offloading to CPU memory mitigates VRAM pressure, but the latency cost is substantial because each forward pass requires repeated transfer of offloaded weights over PCIe. In the deployment regime described for SubSpec, offloaded inference on a single RTX 4090 is often only **1–2 tokens/s** [2509.18344].

SubSpec is situated within the speculative decoding paradigm, in which a fast draft model proposes $D$ candidate tokens and the full target model verifies them in parallel with one forward pass [2211.17192]. In the notation used for SubSpec, if $\tau$ tokens are accepted on average, the number of target forward passes for a sequence of length $N$ drops from $N$ to $N/\tau$, which directly targets the transfer bottleneck of offloaded inference [2509.18344].

The method is explicitly posed against two limitations of prior offloading-oriented speculative decoding. First, many methods require a smaller pretrained model from the same family, which may be unavailable for custom-trained or fine-tuned LLMs. Second, methods that train a draft model typically produce only modest speedups because insufficient alignment with the target prevents large acceptance lengths; the summary reports that existing offloading-oriented approaches typically achieve $\tau < 7$ [2509.18344]. SubSpec addresses this by deriving the draft model from the target itself rather than from a separately trained small model.

## 2. Substitute-model construction and alignment mechanisms

The draft model in SubSpec resides entirely on GPU and is assembled from three components: quantized substitute layers for offloaded portions of the target model, verbatim sharing of GPU-resident layers, and a shared KV-Cache [2509.18344].

For the offloaded portion of the target LLM, SubSpec replaces each target layer in the draft model with a **4-bit quantized substitute** using **group size = 64**, via **HQQ or HIGGS**. The quantization rule is given per weight group $w$ by
$$
w_q = \operatorname{round}\!\left((w/\operatorname{scale})\cdot(2^{b-1}-1)\right), \qquad
\operatorname{scale} = \max |w| \text{ over group}.
$$
This design is intended to preserve structural correspondence with the target while enabling high-throughput low-bit GEMM kernels [2509.18344].

For the layers that remain resident in GPU memory, such as the first $k$ decoder layers, the draft and target models **share the same full-precision layers verbatim**. This is not merely a memory optimization. The reported rationale is that sharing maximizes alignment wherever full-precision execution is already available and simultaneously avoids duplicating GPU-resident parameters [2509.18344].

The third component is a **single shared KV-Cache**. During speculation, the draft model appends to the shared cache; during verification, the target model overwrites any divergent entries. The reported advantages are threefold: **halved KV-Cache footprint**, **perfect contextual alignment**, and **no separate prefill for the draft model** [2509.18344]. A plausible implication is that SubSpec treats alignment as a systems property, not only as a modeling property: draft-target agreement is enforced simultaneously at the level of weights, active layers, and autoregressive state.

## 3. Decoding algorithm and analytical model

The core decoding loop is described as **greedy, tree-based speculative decoding**. Starting from context $C$ and shared KV-Cache $K$, the method builds a speculation tree of depth $D$. At each depth, for every current leaf node $\ell$, it computes draft probabilities $p = \text{draft\_forward}(\ell, K)$, then applies **probability sharpening**
$$
p' \sim p^{1/\tau_s} \Big/ \sum p^{1/\tau_s}, \qquad \tau_s = 0.2.
$$
It then collects the **top-$k$ children per leaf** according to
$$
\text{score} = p'_{\text{child}} \times \text{path\_score}(\ell),
$$
forms the next leaf set, and finally flattens the resulting tree into a sequence of at most $k\!\cdot\!D$ candidate tokens $S$ [2509.18344].

Verification is then performed by a single target-model forward pass on $S$ using the shared cache. The algorithm accepts the **longest prefix** of $S$ matching the target outputs, denoted $t_{\text{accept}}$, appends it to the context, and advances the KV-Cache by the accepted tokens. The average number of accepted tokens per iteration is
$$
\tau = \mathbb{E}[\# \text{ tokens accepted per iteration}].
$$

The paper also gives a simple timing model. For autoregressive decoding of $N$ tokens,
$$
T_{\mathrm{AR}} = N \cdot t_{\text{target}}.
$$
For speculative decoding,
$$
T_{\mathrm{SD}} = N \cdot (D \cdot t_{\text{draft}} + \gamma \cdot t_{\text{target}})/\tau, \qquad 1 \le \tau \le D+1.
$$
The corresponding speedup is
$$
\text{speedup} = T_{\mathrm{AR}}/T_{\mathrm{SD}}
= \tau \Big/ \left(D \cdot (t_{\text{draft}}/t_{\text{target}}) + \gamma\right).
$$
Within this model, high $\tau$ is the dominant lever, and SubSpec is explicitly optimized for that objective [2509.18344].

## 4. Reported performance

The reported evaluation uses **tokens/s**, **VRAM use**, and **$\tau$** as primary metrics, with wall-clock speedup defined as
$$
(\text{tokens/s})_{\text{SubSpec}} / (\text{tokens/s})_{\text{baseline offload}}.
$$
The principal results are summarized below [2509.18344].

| Setup | Throughput and $\tau$ | Reported speedup |
|---|---|---|
| Qwen2.5-7B, 8 GB VRAM, baseline offload, MT-Bench | 2.77 tok/s, $\tau=1.0$ | baseline |
| Qwen2.5-7B, 8 GB VRAM, Qwen2.5-1.5B draft, MT-Bench | 17.7 tok/s, $\tau=13.8$ | 6.6$\times$ |
| Qwen2.5-7B, 8 GB VRAM, SubSpec, MT-Bench | 25.4 tok/s, $\tau=27.1$ | 10.1$\times$ |
| Qwen2.5-32B, 24 GB VRAM, baseline offload, popular benchmarks | 0.52 tok/s | baseline |
| Qwen2.5-32B, 24 GB VRAM, SubSpec, popular benchmarks | 6.50 tok/s, $\tau=28.97$ | 12.5$\times$ |

The source material reports multiple closely related headline numbers for the **Qwen2.5-7B** case. The abstract states **9.1x speedup** on MT-Bench under an **8 GB VRAM limit**, while the detailed results summary gives **25.4 tok/s** and **10.1x**, and the ablation endpoint reports **25.35 tok/s** with **9.15x** [2509.18344]. These figures are best read as distinct reported measurements within the same experimental configuration family rather than as a single canonical value.

The ablation summary for **Qwen2.5-7B, 8 GB VRAM, MT-Bench** decomposes the contribution of major components. The configuration labeled **base substitute+sharing** reaches **19.5 tok/s** with **$\tau=23.1$** and **7.1x** speedup. Adding **shared KV-Cache** increases this to **22.0 tok/s**, **$\tau=25.1$**, and **7.9x**. Adding **probability sharpening** yields **23.7 tok/s**, **$\tau=27.1$**, and **8.5x**. Adding **async data transfer** yields **25.35 tok/s**, **$\tau=27.1$**, and **9.15x** [2509.18344]. This progression suggests that the largest gains arise from alignment-oriented components first, with systems-level overlap then extracting additional throughput.

## 5. Memory behavior, alignment, and trade-offs

SubSpec’s core engineering thesis is that speculative decoding for offloaded LLMs is bottlenecked less by draft-model raw speed than by **draft-target alignment**. The paper attributes high acceptance length to two mechanisms in particular: sharing full-precision GPU-resident layers and sharing the KV-Cache. The result is that the draft mirrors the target “wherever possible,” while quantized substitute layers approximate only the offloaded portion [2509.18344].

The quantization bit-width is presented as a direct trade-off between VRAM and fidelity. The reported summary states that **4-bit** substitute layers yield **$\tau \approx 27$–$29$**, whereas more aggressive **2-bit/3-bit** quantization might reduce VRAM further at the cost of some acceptance length [2509.18344]. This suggests that SubSpec’s performance envelope is governed by a three-way coupling among substitute precision, GPU residency of shared layers, and acceptance dynamics.

The method is not free of memory cost. The summary states that SubSpec requires approximately **7 GB minimum for Qwen2.5-7B**, including all substitute layers. It also notes that **fewer full-precision layers remain GPU-resident than in other speculative decoding methods**, although the overall throughput still leads [2509.18344]. A common misunderstanding would be to view SubSpec as target-model compression; the paper’s own framing is narrower. The target model remains unchanged, and only the draft path is quantized. This suggests that the method is better characterized as a speculative-control mechanism for offloaded inference than as a compression scheme for deployment.

## 6. Applicability, limitations, and prospective directions

The paper presents SubSpec as best suited to **dense LLMs**. It explicitly notes that **Mixture-of-Experts (MoE)** architectures would require further adaptation [2509.18344]. That limitation is structurally consistent with the method’s reliance on layer substitution and shared execution state, both of which are simpler to define for dense decoder stacks than for routed expert topologies.

Three research directions are identified in the source summary. The first is to explore **hybrid layer-skipping and substitute quantization** to reduce the VRAM floor. The second is to test **more aggressive quantization schemes** for substitute layers while balancing $\tau$ against memory consumption. The third is to exploit improvements in **PCIe 5.0/6.0** and kernel optimizations; the summary states that SubSpec’s relative speedup is expected to hold or even grow as interconnects and kernels improve [2509.18344].

In the paper’s reported results, SubSpec is therefore positioned as a **plug-and-play**, **lossless**, and **training-free** method for accelerating offloaded LLM inference on consumer GPUs, with headline speedups above **10x** in the cited configurations and without changing target-model outputs [2509.18344]. Tables and figures in the paper are reported to include full cross-benchmark throughputs, VRAM footprints, and $\tau$ values under both greedy and stochastic sampling, indicating that the method is meant as a systems technique for practical constrained-memory deployment rather than as a decoding heuristic tied to a single benchmark regime.

Source: https://www.emergentmind.com/topics/subspec