SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

Abstract: Agentic multimodal LLMs (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.

• The paper introduces a speculative framework that leverages lightweight, tool-free inference to bypass sequential agentic tool chains, significantly reducing query processing latency.
• It employs a four-phase speculative funnel—comprising heuristic tool-use judgment, speculative prediction, cognitive gating, and agentic fallback—to selectively handle queries.
• Experimental results demonstrate a 1.1–3.35× speedup and improved accuracy (e.g., from 81.39% to 84.26%) by minimizing redundant processing.

SpecEyes: Speculative Acceleration of Agentic Multimodal LLMs via Speculative Perception and Planning

Introduction and Motivation

Agentic multimodal LLMs (MLLMs) have shifted visual understanding from passive, single-pass perception to iterative, stateful interaction with tool-augmented perception loops, enabling compositional reasoning and complex visual search. However, this paradigm incurs severe end-to-end latency and system-level concurrency bottlenecks due to sequential tool dependencies and per-query state mutation. Each query's agentic depth, defined by the number of tool-calling steps, introduces strict causal dependencies that prevent parallelization and result in a scaling complexity of $\mathcal{O}(BDC)$, where $B$ is the batch size, $D$ is the agentic depth, and $C$ is the per-tool inference cost.

Figure 1: The agentic MLLM pipeline is shown at the top, with sequential, state-dependent tool invocations; the SpecEyes framework (bottom) bypasses unnecessary tool-chains via stateless speculative models and answer-separability gating.

Previous work on efficient reasoning in MLLMs—including token-level speculative decoding, token pruning, and temporal compression—has focused exclusively on accelerating steps within a fixed, serial agentic pipeline and fails to break the bottleneck imposed by agentic depth. Importantly, these methods do not address the core inefficiency: the monolithic execution of perception–reasoning loops even when simpler queries do not require multi-step tool use.

SpecEyes Framework

SpecEyes introduces agentic-level speculative acceleration by leveraging two key observations: (1) a substantial portion of queries can be answered correctly with a lightweight, tool-free MLLM from the original image, and (2) reliable identification of such queries enables their speculative bypass of the expensive agentic loop.

The system is structured as a four-phase speculative funnel:

1. Heuristic Tool-Use Judgment: The agentic MLLM evaluates the necessity of tool use for each query using a lightweight head.
2. Speculative Prediction: A small, stateless MLLM generates answers for queries deemed tool-free, producing output logits.
3. Cognitive Gating: An answer-separability metric evaluates speculative answer confidence, providing a label-free, scale-invariant self-verification without the need for ground-truth labels.
4. Agentic Fallback: Low-confidence queries or those necessitating tool use revert to the full, stateful agentic pipeline.

SpecEyes runs the stateless small model in highly parallel mode, while only a residual set—those queries needing actual tool-calling—is processed sequentially by the agentic model. This architecture fundamentally changes the system-level bottleneck from the entire batch to only the fallback subset, resulting in a multiplicative throughput gain as a function of the speculative acceptance rate.

Figure 2: The SpecEyes pipeline routes queries through the four-phase speculative funnel, with high-confidence predictions accepted immediately and only the low-confidence residual routed to the agentic pipeline.

Cognitive Gating via Answer Separability

At the core of SpecEyes’ reliability is its cognitive gating mechanism. Traditional softmax-based confidence metrics are poorly calibrated, especially in the high-entropy, multimodal setting—often resulting in overconfident acceptance of incorrect answers. SpecEyes introduces a scale-invariant answer separability score, $S_\text{sep}$, which measures the standardized margin between the top-1 prediction and its strongest $K$ competitors for each token position.

Token-level scores are aggregated (defaulting to a min-pooling strategy) to enforce a worst-case guard: a final answer is only accepted if every token has high separability. This minimization paradigm is theoretically motivated to keep the error probability tightly bounded.

Figure 3: Confidence calibration via KDE on V

demonstrates that the min-separability score $S_\text{sep}^\text{min}$ provides sharp discriminability between correct and incorrect answers, exceeding all softmax-based and mean-aggregated alternatives.*

Experimental Results

SpecEyes is evaluated on three benchmarks with distinct profiles: V* (multiple-choice attributes and spatial reasoning), HR-Bench (high-resolution image understanding), and POPE (hallucination probing). The Qwen3-VL-2B model serves as the small, speculative MLLM, with DeepEyes and Thyme as the agentic MLLM backbones.

The system achieves 1.1–3.35× speedup over fully agentic backbones while matching or exceeding their accuracy. For POPE, accuracy increases are observed (e.g., adversarial split: 78.43% $\rightarrow$ 85.13%), which provides evidence that eliminating unnecessary tool trajectories can reduce hallucination errors and improve overall model reliability. SpecEyes (min) achieves an average speedup of 1.73× (DeepEyes backbone), with average accuracy boosted from 81.39% to 84.26%. On challenging, tool-required datasets (HR-Bench 8K), the speedup gain shrinks and can even be sub-unity due to the necessity of fallback and the fixed speculative cost.

SpecReason, an existing token-level speculative reasoning approach, is compared as a baseline and found to consistently slow down inference due to its extra token and turn generation without eliminating agentic depth. SpecEyes instead matches the upper bound of speculative acceleration (draft-only) but with full accuracy recovery via selective fallback.

Ablation Studies

Analysis confirms that:

• Lowering the answer-separability gating threshold monotonically increases speedup but reduces accuracy; however, there exists a significant Pareto region where both are superior to baseline agentic MLLMs.

  Figure 4: Varying the confidence gating threshold reveals an accuracy–speedup tradeoff curve with a wide region dominating the baseline agentic pipeline.

• Increasing batch size benefits throughput due to more efficient amortization of stateless speculative processing; at high batch sizes, the sequential fallback pipeline limits further gains.

  Figure 5: Larger serving batches amplify throughput speedup via efficient speculative parallelism, saturating at the fallback bottleneck.

• The top-K parameter in answer separability tunes speculative aggressiveness: larger K yields higher speedup but degrades accuracy.

  Figure 6: Speedup increases monotonically with K in separability computation, but accuracy degradation cautions against overaggressive speculative gating.

Implications and Future Directions

SpecEyes demonstrates that speculative execution can be lifted from the token level to the agentic level in multimodal LLMs, and that large efficiency gains are accessible without sacrificing—often improving—accuracy. The cognitive gating design is robust, generalizes across agentic architectures, and is minimally sensitive to hyperparameters.

Practically, the SpecEyes paradigm enables orders-of-magnitude improvements in latency and throughput for model serving frameworks, making agentic MLLMs more suitable for real-time or large-scale deployment, and reducing operational costs. Theoretically, it opens avenues toward hierarchical, multi-depth speculative inference, in which progressively deeper speculative models execute a bounded number of tool-use steps before fallback—offering a spectrum of accuracy–latency operating points. Further research could also integrate active learning to optimize the speculative acceptance ratio and extend the paradigm to multi-modal, multi-agent, and continual-learning settings.

Conclusion

SpecEyes advances agentic-level speculative acceleration for MLLMs by combining lightweight, tool-free speculative answering with a robust, label-free confidence gating mechanism and heterogeneous parallel funnel execution. The system achieves strong, quantifiable reductions in end-to-end latency and serving bottleneck severity, validating the hypothesis that the majority of queries can bypass the full agentic loop without performance compromise. This work suggests that future stateful multimodal reasoning systems should architect for speculative inference as a first-class principle, supporting both practical efficiency and theoretical robustness in large-scale AI deployments.