SubSpec: Accelerated Speculative Decoding for LLMs

• SubSpec is a method for accelerating parameter-offloaded LLMs by constructing a highly aligned GPU-resident draft model derived from the target model.
• It replaces offloaded layers with 4-bit quantized substitutes while sharing GPU-resident layers and a KV-cache to maximize draft-target alignment.
• This training-free approach achieves up to 10× speedup in throughput by significantly increasing token acceptance lengths during speculative decoding.

Searching arXiv for papers on speculative decoding and offloaded LLM inference to ground the article. arxiv_search(query="speculative decoding offloaded LLM inference speculative decoding acceleration draft model", max_results=10) arxiv_search(query="LLM speculative decoding offloading quantized draft model KV-cache sharing", max_results=10) arxiv_search(query="speculative decoding LLMs", max_results=10) SubSpec, short for Substitute Speculative Decoding, is a method for accelerating parameter-offloaded LLMs by constructing a highly aligned GPU-resident draft model from the target model itself. It is formulated for deployments in which model weights exceed available GPU memory and are therefore offloaded to CPU memory, a regime in which repeated PCIe transfers dominate latency. In the reported formulation, SubSpec is lossless because the target LLM’s FP16 weights and outputs remain unchanged, and training-free because it requires no extra distillation or fine-tuning. Its central objective is to increase the average token acceptance length $\tau$ under speculative decoding, ideally to approximately $30$ under greedy decoding, so that the number of expensive target forward passes is sharply reduced (Wang et al., 22 Sep 2025).

1. Computational setting and problem formulation

Modern open-source LLMs such as Llama3 and Qwen2.5 often exceed 8–24 GB of GPU memory. Parameter offloading to CPU memory mitigates VRAM pressure, but the latency cost is substantial because each forward pass requires repeated transfer of offloaded weights over PCIe. In the deployment regime described for SubSpec, offloaded inference on a single RTX 4090 is often only 1–2 tokens/s (Wang et al., 22 Sep 2025).

SubSpec is situated within the speculative decoding paradigm, in which a fast draft model proposes $D$ candidate tokens and the full target model verifies them in parallel with one forward pass (Leviathan et al., 2022). In the notation used for SubSpec, if $\tau$ tokens are accepted on average, the number of target forward passes for a sequence of length $N$ drops from $N$ to $N/\tau$, which directly targets the transfer bottleneck of offloaded inference (Wang et al., 22 Sep 2025).

The method is explicitly posed against two limitations of prior offloading-oriented speculative decoding. First, many methods require a smaller pretrained model from the same family, which may be unavailable for custom-trained or fine-tuned LLMs. Second, methods that train a draft model typically produce only modest speedups because insufficient alignment with the target prevents large acceptance lengths; the summary reports that existing offloading-oriented approaches typically achieve $\tau < 7$ (Wang et al., 22 Sep 2025). SubSpec addresses this by deriving the draft model from the target itself rather than from a separately trained small model.

2. Substitute-model construction and alignment mechanisms

The draft model in SubSpec resides entirely on GPU and is assembled from three components: quantized substitute layers for offloaded portions of the target model, verbatim sharing of GPU-resident layers, and a shared KV-Cache (Wang et al., 22 Sep 2025).

For the offloaded portion of the target LLM, SubSpec replaces each target layer in the draft model with a 4-bit quantized substitute using group size = 64, via HQQ or HIGGS. The quantization rule is given per weight group $w$ by

$$w_q = \operatorname{round}\!\left((w/\operatorname{scale})\cdot(2^{b-1}-1)\right), \qquad \operatorname{scale} = \max |w| \text{ over group}.$$

This design is intended to preserve structural correspondence with the target while enabling high-throughput low-bit GEMM kernels (Wang et al., 22 Sep 2025).

For the layers that remain resident in GPU memory, such as the first $30$0 decoder layers, the draft and target models share the same full-precision layers verbatim. This is not merely a memory optimization. The reported rationale is that sharing maximizes alignment wherever full-precision execution is already available and simultaneously avoids duplicating GPU-resident parameters (Wang et al., 22 Sep 2025).

The third component is a single shared KV-Cache. During speculation, the draft model appends to the shared cache; during verification, the target model overwrites any divergent entries. The reported advantages are threefold: halved KV-Cache footprint, perfect contextual alignment, and no separate prefill for the draft model (Wang et al., 22 Sep 2025). A plausible implication is that SubSpec treats alignment as a systems property, not only as a modeling property: draft-target agreement is enforced simultaneously at the level of weights, active layers, and autoregressive state.

3. Decoding algorithm and analytical model

The core decoding loop is described as greedy, tree-based speculative decoding. Starting from context $30$1 and shared KV-Cache $30$2, the method builds a speculation tree of depth $30$3. At each depth, for every current leaf node $30$4, it computes draft probabilities $30$5, then applies probability sharpening

$30$6

It then collects the top-$30$7 children per leaf according to

$30$8

forms the next leaf set, and finally flattens the resulting tree into a sequence of at most $30$9 candidate tokens $D$0 (Wang et al., 22 Sep 2025).

Verification is then performed by a single target-model forward pass on $D$1 using the shared cache. The algorithm accepts the longest prefix of $D$2 matching the target outputs, denoted $D$3, appends it to the context, and advances the KV-Cache by the accepted tokens. The average number of accepted tokens per iteration is

$D$4

The paper also gives a simple timing model. For autoregressive decoding of $D$5 tokens,

$D$6

For speculative decoding,

$D$7

The corresponding speedup is

$D$8

Within this model, high $D$9 is the dominant lever, and SubSpec is explicitly optimized for that objective (Wang et al., 22 Sep 2025).

4. Reported performance

The reported evaluation uses tokens/s, VRAM use, and $\tau$0 as primary metrics, with wall-clock speedup defined as

$\tau$1

The principal results are summarized below (Wang et al., 22 Sep 2025).

Setup	Throughput and $\tau$2	Reported speedup
Qwen2.5-7B, 8 GB VRAM, baseline offload, MT-Bench	2.77 tok/s, $\tau$3	baseline
Qwen2.5-7B, 8 GB VRAM, Qwen2.5-1.5B draft, MT-Bench	17.7 tok/s, $\tau$4	6.6$\tau$5
Qwen2.5-7B, 8 GB VRAM, SubSpec, MT-Bench	25.4 tok/s, $\tau$6	10.1$\tau$7
Qwen2.5-32B, 24 GB VRAM, baseline offload, popular benchmarks	0.52 tok/s	baseline
Qwen2.5-32B, 24 GB VRAM, SubSpec, popular benchmarks	6.50 tok/s, $\tau$8	12.5$\tau$9

The source material reports multiple closely related headline numbers for the Qwen2.5-7B case. The abstract states 9.1x speedup on MT-Bench under an 8 GB VRAM limit, while the detailed results summary gives 25.4 tok/s and 10.1x, and the ablation endpoint reports 25.35 tok/s with 9.15x (Wang et al., 22 Sep 2025). These figures are best read as distinct reported measurements within the same experimental configuration family rather than as a single canonical value.

The ablation summary for Qwen2.5-7B, 8 GB VRAM, MT-Bench decomposes the contribution of major components. The configuration labeled base substitute+sharing reaches 19.5 tok/s with $N$0 and 7.1x speedup. Adding shared KV-Cache increases this to 22.0 tok/s, $N$1, and 7.9x. Adding probability sharpening yields 23.7 tok/s, $N$2, and 8.5x. Adding async data transfer yields 25.35 tok/s, $N$3, and 9.15x (Wang et al., 22 Sep 2025). This progression suggests that the largest gains arise from alignment-oriented components first, with systems-level overlap then extracting additional throughput.

5. Memory behavior, alignment, and trade-offs

SubSpec’s core engineering thesis is that speculative decoding for offloaded LLMs is bottlenecked less by draft-model raw speed than by draft-target alignment. The paper attributes high acceptance length to two mechanisms in particular: sharing full-precision GPU-resident layers and sharing the KV-Cache. The result is that the draft mirrors the target “wherever possible,” while quantized substitute layers approximate only the offloaded portion (Wang et al., 22 Sep 2025).

The quantization bit-width is presented as a direct trade-off between VRAM and fidelity. The reported summary states that 4-bit substitute layers yield $N$4–$N$5, whereas more aggressive 2-bit/3-bit quantization might reduce VRAM further at the cost of some acceptance length (Wang et al., 22 Sep 2025). This suggests that SubSpec’s performance envelope is governed by a three-way coupling among substitute precision, GPU residency of shared layers, and acceptance dynamics.

The method is not free of memory cost. The summary states that SubSpec requires approximately 7 GB minimum for Qwen2.5-7B, including all substitute layers. It also notes that fewer full-precision layers remain GPU-resident than in other speculative decoding methods, although the overall throughput still leads (Wang et al., 22 Sep 2025). A common misunderstanding would be to view SubSpec as target-model compression; the paper’s own framing is narrower. The target model remains unchanged, and only the draft path is quantized. This suggests that the method is better characterized as a speculative-control mechanism for offloaded inference than as a compression scheme for deployment.

6. Applicability, limitations, and prospective directions

The paper presents SubSpec as best suited to dense LLMs. It explicitly notes that Mixture-of-Experts (MoE) architectures would require further adaptation (Wang et al., 22 Sep 2025). That limitation is structurally consistent with the method’s reliance on layer substitution and shared execution state, both of which are simpler to define for dense decoder stacks than for routed expert topologies.

Three research directions are identified in the source summary. The first is to explore hybrid layer-skipping and substitute quantization to reduce the VRAM floor. The second is to test more aggressive quantization schemes for substitute layers while balancing $N$6 against memory consumption. The third is to exploit improvements in PCIe 5.0/6.0 and kernel optimizations; the summary states that SubSpec’s relative speedup is expected to hold or even grow as interconnects and kernels improve (Wang et al., 22 Sep 2025).

In the paper’s reported results, SubSpec is therefore positioned as a plug-and-play, lossless, and training-free method for accelerating offloaded LLM inference on consumer GPUs, with headline speedups above 10x in the cited configurations and without changing target-model outputs (Wang et al., 22 Sep 2025). Tables and figures in the paper are reported to include full cross-benchmark throughputs, VRAM footprints, and $N$7 values under both greedy and stochastic sampling, indicating that the method is meant as a systems technique for practical constrained-memory deployment rather than as a decoding heuristic tied to a single benchmark regime.