ViSpec: Vision-Aware Speculative Decoding

• ViSpec is a framework that accelerates multimodal autoregressive generation by using a vision-aware draft model for bulk token proposal and verification.
• It integrates a lightweight vision adaptor and global feature injection to compress visual tokens and enhance multimodal coherence throughout the decoding process.
• Empirical evaluations demonstrate that ViSpec achieves up to 2-3× speedups on VLM benchmarks by effectively mitigating redundant image-token sequences.

Vision-Aware Speculative Decoding (ViSpec) is a framework for accelerating vision-LLMs (VLMs) by making the speculative draft model explicitly multimodal rather than text-only. Introduced by Kang et al. in "ViSpec: Accelerating Vision-LLMs with Vision-Aware Speculative Decoding" (Kang et al., 17 Sep 2025), it is motivated by the observation that speculative decoding, although widely adopted in LLMs, had remained underexplored in VLMs, with existing methods achieving only modest speedups of less than $1.5\times$. ViSpec attributes this gap to a modality-specific asymmetry: large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so (Kang et al., 17 Sep 2025).

1. Problem setting and motivation

Speculative decoding accelerates autoregressive generation by allowing a small draft model to propose several tokens and a large target model to verify them in bulk. In text-only LLMs, this procedure can yield substantial acceleration, but ViSpec argues that direct transfer to VLMs is ineffective because image-token sequences are long, redundant, and difficult for shallow drafters to use productively (Kang et al., 17 Sep 2025).

The central premise of ViSpec is that vision awareness is not optional in multimodal speculative decoding. The framework is designed around two claims stated in the original work: first, shallow draft models are overwhelmed by redundant image tokens; second, multimodal coherence degrades when visual information is not persistently available throughout long responses (Kang et al., 17 Sep 2025). A later benchmark study, MMSpec, generalized this point beyond a single implementation, reporting that methods designed for text-only LLMs degrade in multimodal scenarios and that vision awareness becomes increasingly important at larger batch sizes (Shen et al., 16 Mar 2026).

A common misconception is that speculative decoding for VLMs fails primarily because verification is multimodal. ViSpec instead locates the main bottleneck in drafting: the draft must generate text that is both fluent and visually grounded, yet it is the component least capable of handling long visual token streams. This motivates compressing the visual input before it reaches the draft model, rather than removing visual conditioning altogether (Kang et al., 17 Sep 2025).

2. Architectural design of the vision-aware drafter

ViSpec equips the draft model with two architectural mechanisms: a lightweight vision adaptor module and global visual feature injection. The vision adaptor compresses image tokens into a compact representation that is integrated into the draft model’s attention mechanism while preserving original image positional information; the global feature vector augments all subsequent text tokens to enhance multimodal coherence (Kang et al., 17 Sep 2025).

The adaptor is described as Q-Former inspired. A fixed set of learnable query vectors attends over the original image-token sequence and produces a small set of compressed image embeddings. In the notation given in the paper, with stacked learnable queries $Q$ and visual embeddings $V$, compression is written as

$$\mathrm{Attention}(Q, V, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$

where $K = V$ and the result is the compressed visual representation (Kang et al., 17 Sep 2025).

ViSpec justifies this compression with a redundancy argument for shallow attention. Given $R$ identical embeddings $s$ and one unique embedding $t$, the attention weight on the unique embedding is

$$\alpha_{unique} = \frac{\exp(B)}{R\exp(A) + \exp(B)} \xrightarrow{R \gg 1} 0.$$

In the paper’s interpretation, as redundant visual tokens accumulate, a shallow model increasingly “forgets” unique image content. The vision adaptor is therefore not merely a systems optimization; it is a mechanism for preserving salient visual information under limited draft capacity (Kang et al., 17 Sep 2025).

The second mechanism is persistent global conditioning. ViSpec extracts a global feature vector $g$ from the image and injects it into every generated text token:

$Q$0

Here $Q$1 is the draft hidden state at position $Q$2, $Q$3 is a learnable projection, and $Q$4 is the global visual feature vector (Kang et al., 17 Sep 2025). This is intended to mitigate the lost-in-the-middle problem that arises when image tokens appear only at the front of the context and their influence decays over long responses.

3. Training data curation and alignment strategy

ViSpec emphasizes that VLM datasets with long assistant responses are rare. To address this, it curates a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts (Kang et al., 17 Sep 2025). The data construction procedure is explicitly tied to the target use case of speculative decoding, which benefits most when responses are long enough for multiple-token acceptance to amortize verification cost.

The reported training recipe combines text-only pretraining and multimodal fine-tuning. The paper lists initial text-only pretraining on ShareGPT with 68k dialogue turns and multimodal fine-tuning on 68k LLaVA Visual Instruct Pretrain LCS samples, extended or augmented for response length (Kang et al., 17 Sep 2025). Existing prompts are modified to solicit long, detailed outputs, and target-model generations are used as supervision.

A distinctive concern in ViSpec is shortcut learning. The paper states that training solely on target model outputs can create a risk that the draft model exploits direct access to the target model’s hidden states, rather than learning proper multimodal grounding (Kang et al., 17 Sep 2025). Its training strategy is intended to mitigate this risk. The details describe two corresponding choices: sampled target outputs, rather than always greedy outputs, and multi-token prediction, described as disrupting one-to-one alignment between ground-truth and draft state sequences (Kang et al., 17 Sep 2025).

The paper expresses the draft-target matching objective as a cross-entropy loss between teacher and student token distributions,

$Q$5

where $Q$6 is the target-model probability and $Q$7 is the draft-model probability (Kang et al., 17 Sep 2025). In later comparative work, this token-level supervision became a point of discussion: Variational Speculative Decoding (VSD) characterizes ViSpec as optimizing a KL-only or token-level objective and argues that such training does not directly optimize accepted span or speedup (Zou et al., 5 Feb 2026).

4. Decoding procedure and lossless verification

At inference time, the vision-aware shallow draft model receives compressed image tokens, the text prompt, and the global visual feature supplement on text tokens, and then autoregressively generates speculative continuations (Kang et al., 17 Sep 2025). The target VLM verifies these candidates in parallel.

ViSpec uses the standard speculative acceptance rule. For a proposed token $Q$8 at position $Q$9, the acceptance probability is

$V$0

If a token is rejected, subsequent draft outputs are discarded and generation continues from the target model using the normalized difference distribution, as in standard speculative decoding (Kang et al., 17 Sep 2025). Because acceptance conditions are not relaxed, the framework is described as lossless: the final output distribution exactly matches that of the target model (Kang et al., 17 Sep 2025).

The paper also states that ViSpec adopts dynamic draft trees and multi-candidate token proposals per position, as in EAGLE-2, to maximize acceptance rates (Kang et al., 17 Sep 2025). This places ViSpec within a branch of speculative decoding that seeks speedup not only through a better-aligned draft model but also through richer proposal structures.

Later work made these structural assumptions explicit objects of comparison. SAGE argues that static tree structures remain fixed throughout decoding and therefore fail to adapt to varying prediction difficulty, proposing entropy-guided deeper-narrower or shallower-wider trees instead (Tong et al., 31 Jan 2026). This suggests that ViSpec’s contribution is best understood as draft-side vision adaptation rather than as a final answer to tree-structured scheduling.

5. Empirical evaluation and benchmark interpretation

ViSpec reports experiments on LLaVA-v1.6-Vicuna-7B/13B, Qwen2.5-VL-3B/7B-Instruct, and Pangu-VL-7B, with baselines including Medusa and EAGLE-2 adapted to VLMs by feeding them image embeddings through the VLM vision encoder (Kang et al., 17 Sep 2025). The benchmark suite listed in the paper includes ScienceQA, MM-Vet, MME, TextVQA, COCO Captions, VizWiz, GQA, and SEED-Bench (Kang et al., 17 Sep 2025).

The principal metrics are average acceptance length $V$1 and wall-clock speedup ratio. The paper reports $V$2 speedups on LLaVA and Qwen VLMs, with up to $V$3 on COCO captions, and states that ViSpec achieves consistently higher acceptance lengths across datasets and models than Medusa or EAGLE-2 (Kang et al., 17 Sep 2025). It further describes these results as, to its knowledge, the first substantial speedup in VLM speculative decoding (Kang et al., 17 Sep 2025).

Subsequent benchmark evidence complicates simple cross-paper comparisons. MMSpec, a unified benchmark with 600 multimodal samples across six task categories and ten speculative decoding algorithms, reports that on Qwen2.5-VL-7B, ViSpec attains an overall mean accepted token count of 1.29 and an overall speedup of $V$4, while MSD attains 2.57 and $V$5, and EAGLE-1 attains 2.36 and $V$6 (Shen et al., 16 Mar 2026). MMSpec also finds that throughput speedup alone does not reliably reflect latency performance (Shen et al., 16 Mar 2026). This suggests that ViSpec’s empirical standing depends materially on the model family, batch size, task mix, and latency metric used for evaluation.

Even so, the same benchmark identifies a broader pattern favorable to ViSpec’s design principles: methods explicitly modeling visual context, including ViSpec, achieve much better and more robust acceleration than text-only approaches, especially as batch size increases (Shen et al., 16 Mar 2026). In that sense, ViSpec’s lasting importance lies at least as much in establishing the necessity of vision-aware drafting as in any single headline speedup.

6. Relation to subsequent multimodal speculative decoding research

ViSpec became a reference point for later work on VLM and multimodal speculative decoding. Some papers extended its central insight that the draft should be multimodally informed, while others challenged specific elements of its training or verification design.

MASSV transforms existing small LLMs into effective multimodal drafters by connecting the target VLM’s frozen vision encoder to the draft model through a trainable projector and then applying self-distilled visual instruction tuning; experiments on Qwen2.5-VL and Gemma3 report up to 30% increase in accepted length and up to $V$7 end-to-end speedup on visually grounded tasks (Ganesan et al., 15 May 2025). Relative to ViSpec, MASSV focuses on architecture compatibility within a model family and distribution alignment via self-distillation.

SpecVLM studies systems constraints in VLM speculative decoding, especially prefill dominance by visual tokens and KV-cache pressure. It introduces an elastic visual compressor and an online-logit distillation protocol, reporting $V$8 end-to-end speedups within 5 epochs across LLaVA and MMMU while preserving the target model’s output distribution (Huang et al., 15 Sep 2025). FLASH, in turn, combines latent-aware visual token compression with semi-autoregressive draft decoding and reports up to $V$9 speed-up on video captioning and $$\mathrm{Attention}(Q, V, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$0 on visual instruction tuning tasks (Wang et al., 19 May 2025). These methods share ViSpec’s emphasis on modality-aware drafting but differ in whether they prioritize compression, drafter construction, or parallel draft generation.

Later analyses also targeted ViSpec’s limitations more directly. VSD argues that existing methods, including ViSpec, optimize single greedy trajectories while decoding verifies and ranks multiple sampled draft paths; it reformulates draft training as variational inference over latent proposals and reports up to a $$\mathrm{Attention}(Q, V, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$1 speedup over ViSpec (Zou et al., 5 Feb 2026). SAGE critiques fixed speculation trees and reports up to $$\mathrm{Attention}(Q, V, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$2 decoding speedup for LLaVA-OneVision-72B and $$\mathrm{Attention}(Q, V, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$3 for Qwen2.5-VL-72B through entropy-guided adaptive tree construction (Tong et al., 31 Jan 2026).

In benchmark methodology, MMSpec positions ViSpec within a broader category of vision-aware speculative decoding and uses its findings to motivate ViSkip, a plug-and-play method that disables speculation on highly vision-dependent steps by thresholding cross-attention to visual tokens (Shen et al., 16 Mar 2026). A plausible implication is that ViSpec helped shift the field from asking whether speculative decoding can be applied to VLMs toward asking which parts of the multimodal generation process must remain explicitly vision conditioned, and at what granularity.