- The paper introduces ActiveVision, a 17-task benchmark measuring distributed visual scanning, sequential traversal, and fine-grained attribute comparison in multimodal language models.
- The paper finds that the best unaided model achieves only 10.6% accuracy versus 96.1% for humans, while increased reasoning effort provides limited gains and does not resolve failures in visual evidence extraction.
- The paper shows that coding agents improve performance to 50.6% on some tasks, but realistic image textures, weak verification, high costs, and poor traversal accuracy prevent tool use from closing the active-observation gap.
Motivation and problem statement
Human vision operates as a closed loop: gaze is continuously redirected by intermediate hypotheses, so that many visual problems are solved through iterative, task-driven inspection rather than a single glance. This position has a long pedigree in psychophysics and computer vision, from Yarbus's demonstration that scanpaths depend on the task prompt to formal arguments that inverse problems such as shape-from-shading become well-posed only for an observer who can control its sensors (2607.16165). The paper asks whether multimodal LLMs (MLLMs) exhibit this active observation. The question is behavioral rather than architectural: although current MLLMs encode an image once as a fixed sequence of visual tokens, attention over those tokens could in principle shift as autoregressive reasoning unfolds. Existing benchmarks do not answer this question—vision-centric suites such as MMVP, MMStar, and CV-Bench are largely saturated, while human–model-gap benchmarks (BLINK, BlindTest) establish shortfalls without isolating which capability is missing.
Benchmark design
ActiveVision comprises 17 procedurally generated tasks organized into three families grounded in documented elemental operations of human vision:
- Distributed Scanning: exhaustive enumeration beyond the subitizing range (e.g., counting bounded faces in a planar graph drawing, connected components, separated Voronoi regions, closed loops in a tangle).
- Sequential Traversal: ordered stepping along a connected structure while maintaining position and running state (arrow chains, color-zone sequencing, maze path tracing).
- Visual Attribute Transfer: fine-grained comparison across regions under visual working memory limits (template match counting, paired-panel difference spotting).
The governing design principle is that every instance carries discriminative visual state exceeding what any single language description can losslessly carry. This is enforced through three properties: arbitrary positions at continuous sampled coordinates, arbitrary shapes synthesized via Fourier-modulated contours or jittered splines (no two instances share a shape), and arbitrary traces with dozens of inflection points. An observer that compresses the image once into language will, by construction, lose the information the answer depends on.
A two-stage pipeline produces the stimuli: a deterministic Python generator emits an exact geometric scaffold with ground truth, then GPT-image-2 re-renders it photorealistically while preserving positions, counts, labels, and topology. Only the rendered image is shown to models. This removes the cartoon-input confound and ensures the imagery resembles real-world inputs; the authors note that the same realistic textures that stress MLLMs also bound what classical CV baselines can recover, which becomes consequential in the tool-use ablation.
Headline results
Evaluation uses exact-match accuracy on a fixed 85-item split (5 instances per generator), pure chain-of-thought with no tools, scored across all reasoning-effort tiers exposed by each API. Three unaided human participants form the baseline. The results are stark:
| Model (highest effort) |
Accuracy |
Notes |
| GPT-5.5 xhigh |
10.6% |
Zero on 11 of 17 tasks |
| Gemini 3.5 Flash high |
8.2% |
|
| Gemini 3.1 Pro high |
5.9% |
|
| Claude Opus 4.7 max |
4.7% |
|
| Claude Fable 5 max |
3.5% |
Tops reasoning/coding leaderboards |
| Claude Opus 4.8 max |
2.4% |
|
| Human average (N=3) |
96.1% |
~9× best model |
No item is solved by all six models, indicating the deficit is not model-specific. A question-only control (image omitted) solves 2/85, matching the none-effort run, ruling out prompt priors as the explanation. The implication is direct: benchmark saturation on MMMU-Pro and CharXiv does not imply that vision-language understanding is solved; a capability humans exercise effortlessly is essentially absent.
Reasoning effort does not close the gap
Scaling GPT-5.5's reasoning effort from none to xhigh raises per-item cost nearly a hundredfold but improves accuracy only from 2.4% to 10.6%. Spending 31× more per item on Fable 5 yields no reliable gain; differences between tiers fall within sampling noise on 85 items. The paper's diagnosis is that models do not run out of deliberation—they fail to extract the right visual evidence. Error anatomy supports this:
- Counting: fitted slopes of predicted versus true counts fall well below identity on all eight counting tasks, with undercounting worsening as scenes grow crowded—the signature of glimpsing rather than exhaustive scanning.
- Traversal: prefix survival on ordered-walk tasks collapses within the first one or two steps; not a single walk among 18 runs is completed exactly. Failure is loss of the walk's frame at its start, not gradual drift.
- Comparison: miss rates on difference tasks reach 73–100% while false alarms stay low; several runs answer "none" on nearly every item—a "same" response bias substituting for perception.
Three autonomous coding agents (Codex on GPT-5.5, Claude Code on Opus 4.8 and Fable 5), operating in fresh sandboxes with only the image and question, improve substantially: Fable 5 reaches 50.6%, Codex 37.6%, Opus 4.8 24.7%. Gains concentrate where tasks admit clean algorithmic reduction—Visual Attribute Transfer reaches 43–66%—while Sequential Traversal resists (Codex solves 1/25) and all three agents score zero on Tangled Loop Counting.
Two failure modes explain why substitution stops there. First, classical primitives (thresholding, connected components, contour tracing) are unreliable on realistic textures: masks dissolve into cobblestone, shadows read as corridor, and counts come out wrong. Second—and more tellingly—weak perception cannot catch these tool errors. In documented rollouts, agents submit truncated routes or mis-binarized flood fills when a glance at their own intermediate overlay would falsify the answer. Tool use therefore shifts the active-observation bottleneck to verification within the agent loop rather than eliminating it. Substitution is also costly: $2.74–$7.63 per item and 12–15 minutes of wall-clock time versus roughly half a minute for an unaided human scoring 96.1%.
Limitations
The paper concedes two substantive constraints. The images, though photorealistic, are synthetic re-renderings from controlled prompts rather than samples of the natural-image distribution; external validity rests on the elemental operations measured (scanning, tracing, comparing) rather than on the renderings themselves. And the anti-description property can erode: as models improve at describing images, tasks may need tightening to preserve the guarantee that no concise summary carries the answer. The agentic evaluation is likewise framed as an ablation measuring tool-orchestrated visual problem solving, not a standalone track; a dedicated benchmark for agentic visual problem solving is left open, as is the question of which architectures or training objectives would actually close the perception–reasoning loop.
Conclusion
ActiveVision converts "active vision" from a rhetorical claim into a measurable quantity. Across 17 tasks engineered so that one-pass compression fails by construction, frontier MLLMs solve at most 10.6% of items against a 96.1% human baseline; additional reasoning effort barely helps, and tool-using agents reach only about half while spending orders of magnitude more time and compute. Because the isolated operations underlie radiology, connectomics proofreading, fingerprint examination, and industrial inspection, the gap quantified here is a prerequisite concern for reliable multimodal deployment, and the benchmark provides a concrete target for architectures and training objectives designed to keep visual evidence in the reasoning loop.