SpecEyes: Accelerating Agentic Multimodal LLMs

• SpecEyes is a framework for agentic multimodal LLMs that bypasses sequential tool invocation through speculative planning and cognitive gating.
• It uses a lightweight, tool-free multimodal model as a speculative planner along with a confidence-based cognitive gate using answer separability to decide tool usage.
• Evaluations on benchmarks like V* Bench, HR-Bench, and POPE show up to 3.35x speedup and accuracy gains up to +6.7% compared to traditional agentic methods.

Searching arXiv for the primary "SpecEyes" paper and adjacent work on agentic multimodal LLMs and speculative acceleration. SpecEyes is an agentic-level speculative acceleration framework for agentic multimodal LLMs that reduces the sequential overhead introduced by iterative visual tool invocation. Its central design uses a lightweight, tool-free multimodal model as a speculative planner, a confidence-based cognitive gate built from answer separability, and a heterogeneous parallel funnel that routes only residual hard cases into the expensive stateful tool-use loop. In evaluation on V* Bench, HR-Bench, and POPE, SpecEyes reports 1.1–3.35x speedup over the agentic baseline while preserving or improving accuracy, with gains of up to +6.7% (Huang et al., 24 Mar 2026).

1. Agentic depth and the computational problem

SpecEyes is defined against the execution model of agentic multimodal LLMs that interleave perception, reasoning, and tool calls. In this setting, an agentic model is formalized as $\mathcal{A} = (\mathcal{S}, \mathcal{T}, \pi)$, where $\mathcal{S}$ is the state space, $\mathcal{T}$ is the tool set, and $\pi$ is the policy. Given a query-image pair, the model evolves through a state trajectory $s_0, s_1, \dots, s_D$, where each step may invoke a visual tool and then fuse the result into the next state. The resulting overhead is termed agentic depth, namely the number of perception-and-tool steps required before answer emission (Huang et al., 24 Mar 2026).

The key systems issue is that this trajectory is causally dependent and therefore serial. Tool choice at step $d+1$ depends on the updated state produced by the output of the tool at step $d$, so the chain cannot be collapsed into a single batched forward pass. SpecEyes characterizes this with a per-query latency that sums LLM reasoning cost and tool execution cost over depth, and with a throughput bound that worsens as total agentic latency accumulates across concurrent queries. In practical terms, high-resolution inspection, OCR-heavy reasoning, and spatially compositional tasks drive up depth, which in turn degrades latency and server-side concurrency (Huang et al., 24 Mar 2026).

A frequent misconception is that this bottleneck is reducible by token-level acceleration alone. SpecEyes argues that the dominant cost is not merely autoregressive decoding, but the stateful sequencing of entire perception trajectories. The framework therefore targets the trajectory level rather than the token level.

2. Architecture and execution flow

SpecEyes is organized as a four-phase pipeline. First, the large agentic MLLM performs a lightweight tool-use judgment and outputs a binary decision indicating whether tools may be required. Second, queries judged tool-free are sent to a small non-agentic multimodal model, instantiated in the reported system as Qwen3-VL-2B, which generates a direct answer without invoking tools. Third, a cognitive gate scores that answer using answer separability and either accepts it or routes the query to fallback. Fourth, all rejected or tool-required cases are executed by the full agentic model with its original multi-step tool loop (Huang et al., 24 Mar 2026).

The large-model backbones used in the reported experiments are DeepEyes and Thyme, both capped at 5 tool-use steps per query. The speculative planner is not trained jointly with the agentic model; SpecEyes is explicitly an inference-time wrapper around pretrained components. The tool-use judgment head is prompt-based rather than a separately trained classifier. This is an important architectural choice: the system does not modify the baseline policy $\pi$ of the agentic model, but inserts an early-routing mechanism ahead of it (Huang et al., 24 Mar 2026).

SpecEyes is distinct from speculative decoding. Classical speculative decoding drafts tokens and lets a larger model verify them, but still executes the same overall reasoning-and-tool trajectory. SpecEyes instead speculates on whether the query can terminate at agentic depth $D=0$, thereby bypassing the entire tool chain. In the present formulation, the speculative planner predicts an answer trajectory, not an explicit sequence of tool actions. The framework therefore performs zero-depth speculative planning, with deeper speculative trajectories reserved as future work (Huang et al., 24 Mar 2026).

3. Cognitive gating and answer separability

The central verification mechanism in SpecEyes is answer separability. Standard confidence surrogates based on maximum softmax probability are treated as insufficient because they are often miscalibrated, inflated by deterministic formatting tokens, and insensitive to how close the top competitor is. SpecEyes instead measures the margin structure of the token logits themselves (Huang et al., 24 Mar 2026).

For a token position $n$, let the logits sorted in descending order be $\mathcal{S}$0. Over the top-$\mathcal{S}$1 logits, define their mean $\mathcal{S}$2 and standard deviation $\mathcal{S}$3. Token-level separability is then

$\mathcal{S}$4

This is effectively a standardized top-logit margin. Because both numerator and denominator respond to rescaling, the metric is substantially more stable than raw softmax confidence under logit magnitude changes. In the reported implementation, the default hyperparameters are $\mathcal{S}$5 and $\mathcal{S}$6 (Huang et al., 24 Mar 2026).

SpecEyes considers three answer-level aggregations: the mean over tokens, the minimum over tokens, and a bottom-fraction average over the lowest-confidence tokens. The preferred form is the minimum, denoted $\mathcal{S}$7, motivated by a union-bound argument over token-level error events. If answer-level failure is the union of token failures, then forcing the weakest token above threshold yields the tightest conservative control. Empirically, kernel density plots over correct and incorrect predictions show the clearest bimodal separation for the minimum aggregation, and this variant yields the best accuracy–speed tradeoff in the reported benchmark suite (Huang et al., 24 Mar 2026).

This gate is explicitly label-free at inference time. Thresholds are calibrated offline by inspecting empirical confidence distributions and selecting operating points that preserve baseline accuracy while maximizing early acceptance. The procedure is therefore closer to self-verification than to supervised calibration.

4. Heterogeneous parallel funnel

The systems contribution of SpecEyes is the heterogeneous parallel funnel, which separates stateless speculative computation from stateful agentic execution. Tool-use judgment and small-model speculation are single-turn, stateless operations and can therefore be fully batched across queries. The fallback agentic loop remains serialized within each query because its state changes after each tool call (Huang et al., 24 Mar 2026).

The funnel is described with two routing fractions. Let $\mathcal{S}$8 be the proportion of queries judged tool-free by the screening phase, and let $\mathcal{S}$9 be the fraction of those that the cognitive gate accepts after speculative inference. Then the residual fraction routed to the full agentic model is

$\mathcal{T}$0

for a batch of size $\mathcal{T}$1. This reduction yields the approximate throughput gain

$\mathcal{T}$2

The reported average values are approximately $\mathcal{T}$3 and $\mathcal{T}$4, giving $\mathcal{T}$5 and a theoretical speedup of about $\mathcal{T}$6, consistent with the observed regime of 1.1–3.35x depending on task mix and backbone (Huang et al., 24 Mar 2026).

This formulation clarifies where SpecEyes helps most. Workloads with a large proportion of globally answerable queries and moderate reasoning demands can be collapsed to depth zero at high rate, so the serial back-end is heavily masked by the batched front-end. Workloads dominated by true high-resolution inspection or OCR-heavy queries produce smaller $\mathcal{T}$7, so speedups narrow and may even disappear.

5. Empirical behavior on multimodal benchmarks

SpecEyes is evaluated on V* Bench, HR-Bench, and POPE. V* Bench contributes the Direct Attributes and Relative Position subsets; HR-Bench contributes 4K and 8K high-resolution tasks; POPE contributes Adversarial, Popular, and Random hallucination splits. All reported latencies include both compute and tool execution times, decoding is greedy with temperature $\mathcal{T}$8, and experiments are run on a single NVIDIA A100 40GB GPU (Huang et al., 24 Mar 2026).

With the DeepEyes backbone, the baseline average accuracy is 81.39%. The best SpecEyes variant, using minimum separability, raises this to 84.26% while delivering 1.73x average speedup. On V* Relative Position, accuracy increases from 82.89% to 89.47% at 1.90x speedup. On POPE Adversarial, Popular, and Random, the corresponding gains are 78.43% → 85.13% at 2.13x, 81.90% → 87.00% at 2.15x, and 88.83% → 90.13% at 2.19x. On HR-Bench 4K and 8K, speedups are more modest—1.13x and 1.08x—while accuracy is preserved or slightly improved (Huang et al., 24 Mar 2026).

With the Thyme backbone, the baseline average accuracy is 82.29%, and SpecEyes with minimum separability reaches 83.99% at 1.42x average speedup. The strongest gains again appear on POPE, while the hardest high-resolution split narrows the advantage: on HR-Bench 8K, accuracy rises from 72.43% to 73.31%, but speed is 0.95x, reflecting the case where speculative overhead is not offset because many queries genuinely require tool-based inspection (Huang et al., 24 Mar 2026).

Several ablations sharpen the interpretation. Lowering the gating threshold increases speedup but eventually reduces accuracy. Larger batch sizes improve speedup by amortizing the stateless front-end. Increasing top-$\mathcal{T}$9 in the separability calculation can make routing too aggressive and hurt accuracy. The comparison against SpecReason is also instructive: in the reported settings it is slower than the baseline, with speedups below $\pi$0, because it adds reasoning turns and tokens without removing the underlying stateful tool chain. Conversely, running Qwen3-VL-2B alone is much faster but materially less accurate. SpecEyes occupies the middle regime in which much of the draft model’s efficiency is retained without surrendering the agentic model’s reliability (Huang et al., 24 Mar 2026).

6. Relation to adjacent methods, limitations, and future directions

SpecEyes is best understood as a system-level routing layer above an agentic multimodal model. In relation to speculative decoding, Medusa-style drafting, token pruning, KV compression, or other efficient multimodal perception schemes, its novelty is that it attempts to remove entire tool trajectories rather than merely accelerating computation inside a fixed trajectory. In relation to planner–executor or multi-agent systems, its planner is deliberately minimal: a tool-free MLLM that speculates only at depth zero, while the original agentic executor remains unchanged (Huang et al., 24 Mar 2026).

Several limitations follow from that design. The current framework only speculates on the case where no tools are needed; it does not yet speculate over shallow multi-step plans such as “one crop then answer.” The small planner is tool-agnostic and does not model the internal effects of tool outputs. Tool-use judgment depends on the competence of the large baseline model itself. Evaluation is restricted to image-centric benchmarks rather than richer multimodal ecosystems such as browsing, databases, or longer-horizon agent workflows. These constraints explain why the largest benefits appear on workloads with many globally answerable queries, while fine-detail high-resolution tasks yield smaller gains (Huang et al., 24 Mar 2026).

The principal future direction is multi-depth speculation, in which the small model would operate at bounded depths $\pi$1 rather than only $\pi$2. Other extensions include stronger confidence estimation, composition with token-level speculative decoding and early-exit schemes, and deployment into broader tool ecosystems. A plausible implication is that SpecEyes defines a general strategy for agentic serving: stateless speculative front-ends absorb easy queries in parallel, while a smaller residual set is left to the stateful serial core. In that sense, SpecEyes is less a replacement for agentic multimodal reasoning than an execution policy for making such reasoning operationally tractable (Huang et al., 24 Mar 2026).