Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Abstract: Multimodal LLMs (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains less understood. We introduce "Mind's Eye", a multiple-choice benchmark of eight visuo-cognitive tasks inspired by classic human intelligence tests and organized under a novel "A-R-T" taxonomy: Abstraction, Relation, and Transformation. The tasks probe core processes of fluid intelligence such as pattern induction, analogical relation mapping, and mental transformation. We evaluate a diverse suite of closed-source and open-source MLLMs and compare their performance with human participants. Humans achieve 80% accuracy, while top performing MLLMs remain below 50%. Error analysis reveals failures in: (i) visual attention allocation, (ii) internal perceptual manipulation, and (iii) weak abstraction of underlying visual concepts. Our findings suggest that current MLLMs exhibit limited visuospatial reasoning capabilities, when compared with human participants, highlighting the need for more cognitively grounded evaluation frameworks.

• The paper introduces the Mind's Eye benchmark to assess MLLMs' abilities in abstraction, transformation, and spatial composition.
• It utilizes a structured ART taxonomy with synthetic SVG tasks to rigorously evaluate visual reasoning and diagnose performance gaps.
• Empirical results show humans achieve 80% accuracy whereas top models remain below 50%, highlighting the need for cognitively grounded AI.

Mind's Eye: A Benchmark of Visual Abstraction, Transformation, and Composition for Multimodal LLMs

Introduction and Motivation

The "Mind's Eye" benchmark addresses a critical blindness in current multimodal LLMs (MLLMs): their limited capacity for higher-order visual cognitive reasoning. While MLLMs have demonstrated strong performance on tasks involving object recognition, image captioning, and text-image correlation, their ability to perform internal manipulations—such as mental rotation, paper folding, and abstraction—is not systematically assessed or developed by existing evaluation frameworks. "Mind's Eye" systematically probes these visuo-cognitive faculties, inspired by classical psychometric theories of fluid intelligence (Carroll's Three-Strata theory), and grounded in experimental paradigms from human intelligence testing.

Benchmark Construction and Taxonomy

The ART Cognitive Framework

"Mind's Eye" formalizes visuospatial reasoning through a tripartite ART taxonomy:

• Abstraction tests require induction of latent regularities or rules (e.g., detecting hierarchical structure or abstract visual relations).
• Relation tasks focus on analogical mapping, correspondence, and structural alignment across distinct elements.
• Transformation evaluates the capacity for mental manipulations such as rotation, folding, or integration of spatial primitives.

This cognitive decomposition facilitates both diagnostic error analysis and theoretical alignment with fluid intelligence constructs.

Task Suite and Dataset Generation

The benchmark consists of eight multiple-choice tasks, each mapping to one or more ART domains. Stimuli are synthesized via programmatic SVG generation, providing strict control over geometric parameters and eliminating reliance on linguistic priors or world knowledge. Each item includes diagnostic distractors that target specific reasoning errors—mirrored transformations, parity errors, or superficial feature matches.

Figure 1: Overview of the eight tasks in the Mind's Eye benchmark, highlighting the diversity of cognitive demands across ART domains.

The benchmark is balanced across the ART axes. Difficulty is parametrized a priori but also calibrated via human consensus, ensuring that item stratification corresponds to human cognitive demand rather than superficial variance.

Experimental Evaluation

Model and Human Performance

Eighteen MLLMs (including closed-source frontier models like GPT-4o and Gemini-2.5, and state-of-the-art open-source models such as Qwen-2.5-VL, LLaVa-1.6, and InternVL) were benchmarked on Mind's Eye. Human performance (n=30, non-expert adult cohort) is measured as an upper bound. The findings are salient:

• Human participants achieve 80% macro-average accuracy across tasks, while frontier MLLMs remain below 50%.
• The largest deficits manifest in transformation and abstraction tasks, especially where mental simulation or rule induction are required.
• MLLM accuracy is invariant to item difficulty, in stark contrast with humans whose performance degrades gracefully with item complexity.

Figure 2: Human-model performance gap stratified by ART domain and item difficulty. Models exhibit flat performance, while humans show expected degradation with increasing difficulty.

Figure 3: Model-task heatmap illustrating that even top-tier MLLMs have significant, heterogeneous weaknesses across tasks—especially Transformation and Abstraction.

Scale, Architecture, and Prompting

Model performance exhibits only moderate correlation with scale (Pearson $r \sim 0.62$), with architectural factors equally influential. Scaling improves performance primarily on perception-heavy tasks but is insufficient to bridge reasoning gaps on transformation and abstraction.

Figure 4: Qwen-2.5-VL accuracy across ART tasks, demonstrating non-uniform gains from scaling and consistent weaknesses on key reasoning measures.

Prompt engineering yields weak, dimension-dependent effects: explicit scaffolding marginally boosts Abstraction tasks but consistently impairs Transformation performance.

Figure 5: Change in accuracy from prompt variations vs. CoT across ART dimensions, showing only limited benefit and frequent harm outside of Abstraction.

Figure 6: Prompting strategy effects show no universal improvement; certain strategies even degrade performance on core cognitive axes.

Detailed Error Analyses

Visual Attention and Reasoning

Attention maps indicate that localized attention correlates with accurate reasoning (OAS$_\text{correct}$ vs. accuracy $r_{pb}=0.34$), but high attention does not guarantee correct answers. Models can "look" at the right region but still fail to integrate perceptual input with reasoning.

Figure 7: Misplaced attention in LLaVa-7B on Mental Transformation, with expected regions of focus marked; the model fails to ground reasoning in visual evidence.

Reasoning Trace Diagnostics

Qualitative and quantitative trace analysis demonstrates that MLLMs:

• Frequently rely on surface heuristics such as color matching or perceptual artifact identification rather than abstract rule application or mental simulation.
• Produce plausible but mechanically ungrounded rationales, as evidenced by CoT analysis across tasks.

  Figure 8: GPT-4o reasoning trace on Mental Transformation, relying on color, not shape manipulation, for option matching.

  

  Figure 9: Paper Folding traces showing superficial reasoning with no evidence of spatio-temporal tracking required to solve the task.

Interventions and Causality

Ablating individual attention heads in Qwen-7B did not yield significant performance drops on transformation tasks, suggesting the absence of localized causal circuits for visuospatial manipulation.

Figure 10: Causal intervention heatmap from attention head knockouts in Qwen-7B on Mental Composition, demonstrating distributed or absent specialized circuits.

Comparative Benchmarking and Theoretical Implications

"Mind's Eye" is the first multimodal benchmark to satisfy six key psychometric and diagnostic criteria: formal cognitive taxonomy, derivation from established assessments, distractor confounding, knowledge independence, parametric control, and scalability. Prior datasets lack multiple dimensions required for robust visuo-cognitive assessment, typically conflating recognition with reasoning.

Figure 11: Demonstrating the dataset’s comprehensive coverage of the ART cognitive framework and task alignment.

By invoking Carroll's Three-Stratum theory, this benchmark directly operationalizes constructs of fluid intelligence in a multimodal setting.

Figure 12: Mapping between Carroll’s fluid intelligence theory and the ART taxonomy targeted by Mind’s Eye.

Limitations and Future Directions

While reliability and calibration are strong due to synthetic data, the current formulation is limited to multiple-choice format and 2D stimuli with 3D implications. Open-ended generative evaluations, naturalistic image sources, and rigorous cross-lingual or expert-baselined human evaluations remain open future directions. Bridging the observed human-model gap may require architectural advances in grounded spatial working memory, explicit transformation operations, and multimodal representation learning that goes beyond scale and dataset engineering.

Conclusion

Mind's Eye exposes a persistent, theoretically significant gap between human and multimodal LLM visuospatial reasoning, even under strong prompting and with frontier models. Scalability and surface-level perception are insufficient substitutes for the mechanistic cognitive operations required for abstraction and transformation. These results both motivate and structurally enable research into cognitively grounded architectures and evaluation protocols for multimodal AI.

Key numerical result: humans achieve 80% accuracy; top-tier MLLMs remain sub-50% on tasks requiring real visual reasoning, with particularly large deficits in Transformation and Abstraction.

Implication: Current methods in scaling and prompt engineering are inadequate for progress on these cognitive axes; new approaches grounded in human-like visual cognition are necessary for further advances in genuine visual intelligence.

(2604.16054)