MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

Abstract: Vision-LLMs (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-LLMs. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.

• The paper introduces a standardized benchmark of 600 multimodal samples across six tasks, evaluating 10 lossless decoding methods on Qwen2.5-VL-7B and LLaVA-1.5-7B using accepted tokens, speedup, and latency metrics.
• The paper finds that text-oriented speculative decoders often fail on visually grounded generation, with some methods falling below 1× speedup, while vision-aware approaches sustain roughly 2.3–2.6× speedups under larger batch sizes.
• The paper proposes ViSkip, a training-free attention-gated controller that disables speculation when visual relevance is high, improving speed and latency consistency across ViSpec, MSD, and SAM Decoding.

MMSpec addresses a gap in the evaluation of speculative decoding for vision-LLMs (VLMs). While speculative decoding has been extensively benchmarked on text-only LLMs, existing evaluations rely on heterogeneous datasets, models, and hardware setups, and none capture the multimodal-specific phenomena—cross-modal dependencies, visually grounded token distributions, and long multimodal contexts—that determine whether a drafting strategy succeeds. The paper introduces a standardized benchmark of 600 multimodal samples across six task categories, integrates ten lossless speculative decoding algorithms under a unified protocol on identical hardware, and derives three empirical findings that motivate ViSkip, a plug-and-play vision-aware method (2603.14989).

Benchmark design

MMSpec enforces four construction principles: workload diversity, balanced topic coverage, explicit multi-turn support, and method-agnostic measurement. The 600 samples are drawn from seven established sources, with 100 instances per subtask:

Subtask	Source	Avg. output length (tokens)
General VQA	GQA	46.98
Text VQA	TextVQA	63.15
Image Captioning	COCO	191.90
Chart VQA	CharXiv	68.56
Complex Reasoning	MMMU-Pro	285.60
Multi-turn Conversation	ConvBench + MM-MT-Bench	747.65

Output lengths span roughly an order of magnitude, from short factual answers to multi-turn dialogues averaging nearly 750 tokens, which stresses both draft acceptance behavior and per-request latency stability.

The evaluation covers ten lossless methods spanning two families. Training-based methods include EAGLE-1/2/3 (feature-level and token-level auto-regressive drafters with dynamic trees), Medusa (multi-head tree proposals), ViSpec (lightweight vision adaptor compressing visual tokens in the drafting path), and MSD (a multimodal draft model trained with staged text-then-multimodal training). Training-free methods include SAM Decoding (suffix-automaton longest-match), Lookahead (trie retrieval with tree verification), Recycling (reusing discarded candidates as draft-tree nodes), and Prompt Lookup Decoding (n-gram lookup over the prompt). All experiments run on four NVIDIA A100 GPUs with Qwen2.5-VL-7B-Instruct and LLaVA-1.5-7B as target models, reporting Mean Accepted Tokens (MAT) and walltime speedup ratio relative to vanilla autoregressive decoding.

Finding 1: Text-only methods degrade in multimodal settings

The headline result is that neither training-free nor vision-agnostic trained methods transfer reliably to multimodal generation. Model-free approaches such as Lookahead achieve only 1.07× overall speedup on Qwen2.5-VL-7B and 1.17× on LLaVA-1.5-7B, and several subtasks fall below 1× (e.g., Recycling at 0.72× on LLaVA chart QA), meaning the speculative machinery adds overhead without effective drafts. More strikingly, among trained methods, EAGLE-3 collapses entirely: MAT of 0.24 overall on Qwen2.5-VL-7B and 0.04 on LLaVA-1.5-7B, with speedups as low as 0.68–0.73×—i.e., slower than autoregressive baseline despite being the strongest text-only drafter in its family. EAGLE-2 manages only 2.02× on Qwen2.5-VL versus MSD's 2.58×.

The authors attribute this to drafters trained purely on textual signals failing to model visual-conditioned next-token distributions, yielding low acceptance rates during verification. An appendix probe quantifies this directly: logging per-step visual-attention ratios for MSD, ViSpec, and SAM on Qwen2.5-VL-7B shows that steps attending strongly to image tokens (ratio ≥ 0.35) have consistently lower acceptance lengths—for SAM, average acceptance drops from 0.44 to 0.14 tokens between low- and high-visual steps, essentially accepting nothing when generation is visually grounded. This is direct evidence that speculation is least reliable exactly where VLM inference is most distinctive.

Finding 2: Vision awareness matters more at larger batch sizes

Sensitivity analysis across batch sizes shows vision-aware methods (MSD, ViSpec) maintaining roughly 2.3–2.6× speedup on both target models, while non-vision-aware methods such as EAGLE-2 and Lookahead degrade significantly as batch size grows. The proposed mechanism is that batching increases input heterogeneity, amplifying the mismatch between visually uninformed draft predictions and target outputs, which depresses acceptance rates. This finding carries a practical implication for serving deployments: speculative decoding choices tuned at batch size 1 do not necessarily remain optimal under concurrent load, and vision-awareness is not merely a marginal refinement but a scaling requirement.

Finding 3: Throughput speedup misrepresents latency behavior

Per-request latency CDFs reveal that methods with higher aggregate speedups do not always complete individual requests faster; some exhibit unstable behavior across samples, producing long-tail latencies despite favorable averages. Vision-aware methods shift the entire CDF leftward, completing over 50% of requests within shorter wall-clock times. This contradicts common benchmark practice that reports only mean speedup, and implies that throughput-only evaluations can mask tail-latency regressions relevant to interactive multimodal systems.

ViSkip

Motivated by these findings, ViSkip is a plug-and-play controller layered on top of any speculative decoder. At each step $t$, it computes cross-attention weights $A_t$ between the decoder hidden state $h_t$ and visual tokens $V$, takes the maximum attention weight $S_t = \max_i A_t^{(i)}$ as a vision-relevance score, and switches policy against a threshold $\tau$: if $S_t \le \tau$ it runs standard speculative decoding with draft length $K$; if $S_t > \tau$ it falls back to single-token greedy generation from the full model. The rationale follows directly from the probe results: speculation should be suppressed precisely when the next token depends on visual evidence, since drafts are most likely to be rejected there.

Evaluated on Qwen2.5-VL-7B atop three baselines (ViSpec, MSD, SAM Decoding), ViSkip consistently improves speedup across batch sizes, shifts latency CDF curves leftward, and reduces target-model compute time in the latency breakdown. Because it requires no retraining and only reads attention scores already computed by the model, integration cost is minimal. However, the paper reports sensitivity to $\tau$ only in an appendix and does not establish how the threshold generalizes across architectures or attention patterns beyond the tested pair.

Limitations and open questions

Several caveats qualify the conclusions. The benchmark uses 600 samples (100 per subtask), which limits statistical power for per-subtask comparisons, particularly given the large variance the authors themselves document—MSD ranges from 1.80× to over 4× across subtasks on Qwen2.5-VL-7B, illustrating instability that a larger sample would characterize more precisely. Only two 7B-scale target models are evaluated, so findings may not extend to larger or differently architected VLMs. All evaluated methods are lossless; lossy or relaxed-verification variants are out of scope. The non-greedy results at temperature 0.6 show substantial further degradation for training-based methods—for example, ViSpec drops from 3.00 MAT / 2.58× speedup to 1.45 / 1.68× on LLaVA-1.5-7B—which raises an open question about whether draft-target agreement under sampling can be restored by distribution-aware drafting rather than the conservative fallback ViSkip employs. Finally, the vision-relevance signal relies on maximum cross-attention weight, an assumption that may fail for architectures without clean cross-modal attention structure; whether alternative relevance estimators (e.g., learned probes or hidden-state classifiers) improve robustness remains unexamined.

Conclusion

MMSpec provides the first unified benchmark and third-party evaluation of speculative decoding for VLMs, demonstrating that text-oriented methods—including state-of-the-art feature-level drafters—can fall below baseline speedup in multimodal settings, that vision awareness becomes critical under batching, and that throughput speedup alone is an unreliable proxy for latency quality. ViSkip operationalizes these observations into a simple attention-gated switch that improves all tested backends. The benchmark and its step-level acceptance analysis offer a concrete foundation for future work on drafting strategies that explicitly condition on visual context under sampling.