Grouped Speculative Decoding (GSD)

• The paper introduces a method that replaces token-level verification with dynamic, context-aware group verification, building on Speculative Jacobi Decoding.
• It exploits redundancy in VQ-style visual codebooks to reduce inference latency by up to 4.88× while preserving metrics like CLIP and FID.
• The approach serves as a strong training-free relaxation baseline for autoregressive image models, though its single-sequence design limits multi-path optimization.

Grouped Speculative Decoding (GSD) is a speculative decoding method for autoregressive image generation that replaces token-level verification with group-level verification over clusters of visually valid tokens, thereby exploiting redundancy and diversity in VQ-style visual codebooks to increase acceptance rates and reduce inference latency without additional training (So et al., 11 Aug 2025). In the visual-autoregressive literature, GSD is most specifically the grouped relaxed verification variant built on Speculative Jacobi Decoding (SJD); in later taxonomies it is positioned as “SJD + grouped relaxed sampling in codebook space,” and as a single-sequence, self-speculative method rather than a multi-path one (Lei et al., 9 Jun 2026).

1. Definition and taxonomic position

Autoregressive image models such as Lumina-mGPT, LLaMA-Gen, Chameleon, Emu3, and related systems generate images as long sequences of discrete visual tokens. In the formulation used for modern AR image models, the model factorizes

$$p_\theta(x_{1:T}) = \prod_{t=1}^T p_\theta(x_t \mid x_{<t}, \text{prompt}),$$

with each $x_t$ a codebook index produced by a VQ-style tokenizer (So et al., 11 Aug 2025). Because generation is strictly sequential, inference latency grows with image resolution and token count.

Within this setting, GSD is not a separate draft-model framework. It extends SJD, which is self-speculative: previous target-model predictions are reused as the draft distribution, and verification is normally performed at the exact-token level. GSD keeps SJD’s self-speculative structure but changes verification so that semantically similar codebook entries are treated as groups rather than isolated tokens (Lei et al., 9 Jun 2026). In PathRelax’s taxonomy, the progression is: vanilla autoregressive decoding, Jacobi decoding, SJD, GSD, then multi-path variants such as PathExplore and PathRelax; this makes explicit that GSD remains single-sequence and inherits SJD’s chain structure (Lei et al., 9 Jun 2026).

2. Motivation from visual-token statistics

The central motivation for GSD is that visual tokens do not behave like text tokens. The image-generation paper reports that image-token distributions are often high-entropy and near-uniform over many vocabulary entries, whereas language-model next-token distributions are typically much more peaked (So et al., 11 Aug 2025). For Lumina-mGPT under standard decoding settings, many token positions have top-1 probability below $5\%$, indicating that the model assigns comparable probability mass to many distinct codebook entries.

This statistical flatness matters because standard speculative decoding accepts or rejects one sampled token at a time. For image AR, many alternative tokens can decode to nearly indistinguishable local patches, so token-level verification produces excessive false-negative rejections: the draft and target may disagree on the exact codebook index while still agreeing on the local visual semantics (So et al., 11 Aug 2025). The paper supports this interpretation with a random-replacement experiment in which $50\%$ of tokens are replaced by random Top-100 alternatives while the resulting image retains a CLIP score of $32.089$, and with visualizations showing that codebook embeddings are almost uniformly distributed and do not form clear semantic clusters (So et al., 11 Aug 2025).

A further complication is contextual dependence. The same token grid can decode into visibly different RGB structure depending on how much surrounding context is present, so static similarity in embedding space is not a reliable proxy for semantic interchangeability during autoregressive decoding (So et al., 11 Aug 2025). This is the immediate reason the method moves from static clustering toward dynamic, context-aware grouping.

3. Group-level verification mechanism

GSD partitions the token vocabulary $\mathcal{X}$ into disjoint clusters

$\mathcal{C} = \{C_1, C_2, \dots, C_K\},$

with $C_i \cap C_j = \varnothing$ for $i \neq j$ and $\bigcup_i C_i = \mathcal{X}$ (So et al., 11 Aug 2025). Given expert and draft distributions $x_t$0 and $x_t$1, it defines grouped masses

$x_t$2

If $x_t$3 denotes the cluster containing token $x_t$4, GSD replaces the token-level acceptance rule with the group-level criterion

$x_t$5

When the group is rejected, the procedure falls back to the standard correction step based on $x_t$6 (So et al., 11 Aug 2025).

The practical method is dynamic rather than static. At each verification step, the expert distribution $x_t$7 is sorted, the draft token’s position in that sorted list is located, and a local neighborhood of size $x_t$8 around that position is used as an initial cluster candidate. That neighborhood is then filtered by two criteria: probability proximity and embedding-distance proximity, using thresholds $x_t$9 and $5\%$0 together with a precomputed codebook-distance matrix $5\%$1 (So et al., 11 Aug 2025). The resulting cluster is therefore conditioned on the current context, not only on codebook geometry.

This design directly addresses the failure mode of static embedding clustering. The static alternative produces only modest reductions in the number of function evaluations, whereas dynamic grouping based on the expert distribution gives much larger gains and preserves CLIP more effectively (So et al., 11 Aug 2025). The paper also reports that dynamic clustering based on the draft distribution $5\%$2 is inferior to clustering based on the expert distribution $5\%$3, which is consistent with GSD’s emphasis on verifier-side semantics.

4. Acceptance-rate theory, exactness, and structural limits

The key theoretical result in the image-generation paper is that grouping cannot reduce the average acceptance rate. Standard speculative decoding has acceptance

$5\%$4

and GSD induces

$5\%$5

Because grouping applies the triangle inequality to sums over clusters, the paper proves

$5\%$6

hence

$5\%$7

for any clustering $5\%$8 (So et al., 11 Aug 2025). The theoretical gain is therefore distributional rather than heuristic: cluster-level aggregation necessarily smooths token-level discrepancies.

This does not mean that all clusterings are equally useful. The same paper shows that static clustering based on embedding distance is ineffective in practice, because the codebook embedding space is nearly uniform and semantic similarity is strongly context-dependent (So et al., 11 Aug 2025). A plausible implication is that the theorem guarantees acceptance improvement, but not optimal semantic fidelity.

A second important point is exactness. PathRelax characterizes GSD as a relaxed speculative method: it increases acceptance length over lossless SJD, but it introduces a controlled approximation, so generation is no longer strictly lossless with respect to the original $5\%$9 (Lei et al., 9 Jun 2026). That observation is central in later comparisons. PathRelax therefore treats GSD as a useful baseline relaxation mechanism, but also as one whose acceleration remains structurally limited by SJD’s single-chain design (Lei et al., 9 Jun 2026).

5. Empirical performance in autoregressive image generation

On Lumina-mGPT-7B at $50\%$0, with Top-$50\%$1 and temperature $50\%$2, the original GSD paper reports that GSD accelerates AR image models by an average of $50\%$3 while preserving image quality, and reaches up to $50\%$4 latency speedup on Parti-Prompt with $50\%$5 (So et al., 11 Aug 2025). The same study compares vanilla autoregressive decoding, lossless SJD, naive lossy baselines such as Amplify and Addition, and GSD with different group sizes. The dominant empirical pattern is that GSD improves the speed–quality Pareto frontier: moderate group sizes give large reductions in latency and number of function evaluations while keeping CLIP and FID close to the vanilla baseline.

Setting	Latency / NFE	Quality
Parti, Vanilla	112.29 s / 2392	CLIP 32.091
Parti, SJD	52.34 s / 1035.3	CLIP 32.090
Parti, GSD $50\%$6	24.13 s / 636.75	CLIP 32.075
COCO, Vanilla	122.45 s / 2379	FID 30.79, CLIP 31.308
COCO, SJD	55.26 s / 1058.6	FID 30.78, CLIP 31.308
COCO, GSD $50\%$7	32.52 s / 674.04	FID 33.55, CLIP 31.24

The ablation on clustering strategy is especially diagnostic. On Parti-Prompt, baseline SJD gives NFE $50\%$8 and CLIP $50\%$9; static embedding-distance grouping reduces NFE only to $32.089$0, whereas dynamic grouping based on the expert distribution reduces NFE to about $32.089$1 while keeping CLIP near baseline (So et al., 11 Aug 2025). This is the empirical basis for treating “dynamic GSD” as the actual method, rather than merely any grouped verifier.

Later evaluations use GSD as a baseline relaxation method. In PathRelax’s MSCOCO 2017 results for Lumina-GPT-7B-768, GSD reaches $32.089$2, $32.089$3, and 41.2 s, while PathSpec reaches $32.089$4 and $32.089$5 (Lei et al., 9 Jun 2026). Similar gaps appear on T2ICompBench and Parti-Prompts, and the same paper explicitly notes that even lossless multi-path PathExplore can outperform relaxed single-chain methods such as GSD in speedup ratio (Lei et al., 9 Jun 2026). In that comparative literature, GSD therefore appears as an important intermediate method: it exploits codebook-level redundancy effectively, but remains constrained by single-sequence drafting.

6. Terminological overload and related grouped-decoding lines

The acronym “GSD” is overloaded in speculative-decoding research, and the image-generation method is only one of several meanings. In LLM inference, “Graph-Structured Speculative Decoding” uses a DAG of hypotheses rather than codebook-group verification: recurring $32.089$6-grams across multiple branches are merged into shared subgraphs, and the method reports $32.089$7 to $32.089$8 speedup on LLaMA-2-70B, substantially surpassing standard speculative decoding in that setting (Gong et al., 2024). This is a graph-compression method for multi-hypothesis drafting, not a codebook-clustering method.

A different use of grouped decoding appears in batch serving. “Batch Speculative Decoding Done Right” instantiates grouped speculative decoding through EXSpec, which keeps a pool of ragged sequences and dynamically forms temporary same-length groups so that speculative decoding can proceed without breaking position IDs, attention masks, or KV-cache alignment; on SpecBench it reports up to $32.089$9 throughput improvement at batch size 8 while maintaining $\mathcal{X}$0 output equivalence (Zhang et al., 26 Oct 2025). Here “grouped” refers to length-based grouping of requests, not token similarity.

Sparse long-context verification introduces yet another grouping axis. SSV groups nearby verifier queries so that dynamic sparse attention can reuse overlapping KV blocks across speculative branches; on H100 GPUs it reports up to $\mathcal{X}$1 end-to-end throughput over autoregressive NSA decoding and up to $\mathcal{X}$2 kernel speedups (Wang et al., 19 May 2026). In this usage, grouping is over sparse verifier queries and block schedules.

For the image-generation literature, the most precise interpretation remains the 2025 formulation: GSD as grouped relaxed verification in codebook space for autoregressive image models (So et al., 11 Aug 2025). Its lasting significance is twofold. First, it established that visual-token redundancy can be operationalized directly at verification time through grouped acceptance. Second, subsequent work clarified both its value and its limits: GSD is a strong training-free relaxation method, but one whose single-sequence structure leaves room for further gains from multi-path drafting and cross-path relaxation (Lei et al., 9 Jun 2026).