When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (2511.02779v1)

Abstract: We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal LLMs, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA.

• The paper introduces MIRA, a benchmark that evaluates complex visual chain-of-thought reasoning in multimodal large language models.
• The paper systematically probes MLLMs with 546 multimodal problems across geometry, physics, and puzzles, revealing direct-input accuracies below 20% and performance drops with text-only cues.
• The paper demonstrates that annotated intermediate visual clues boost performance by 33.7% on average, underlining the necessity for unified vision-language architectures.

MIRA: A Benchmark for Visual Chain-of-Thought Reasoning in Multimodal LLMs

Motivation and Benchmark Design

The MIRA (Multimodal Imagination for Reasoning Assessment) benchmark is introduced to address a critical gap in the evaluation of multimodal LLMs (MLLMs): their ability to perform complex reasoning that necessitates the generation and manipulation of intermediate visual representations. Traditional chain-of-thought (CoT) prompting has demonstrated efficacy in improving LLM reasoning by decomposing problems into textual steps, but this paradigm is fundamentally limited for tasks that are intrinsically visual, such as spatial reasoning, geometric construction, and physical simulation. MIRA is constructed to systematically probe these limitations by requiring models to "think while drawing," mirroring human problem-solving strategies that rely on sketches and diagrams.

MIRA comprises 546 multimodal problems spanning 20 task types across four domains: Euclidean Geometry (EG), Physics-Based Reasoning (PBR), Abstract Spatial Logical Puzzles (ASLP), and Causal Transformations (CT). Each instance is paired with manually annotated intermediate visual states and a unique ground-truth answer, ensuring high data quality and reproducibility. The benchmark is designed to require intermediate visual cues for successful reasoning, with tasks ranging from single-step (static) to multi-step (dynamic) visual reasoning.

Figure 1: MIRA categorizes Visual-CoT reasoning tasks into Static (Single-Step) and Dynamic (Multi-Step), with representative examples and a summary of dataset statistics.

Evaluation Protocol

MIRA introduces a three-level diagnostic evaluation protocol to decouple the contributions of visual and textual information:

1. Direct Evaluation (D): The model receives only the initial image and question, measuring end-to-end problem-solving capability.
2. Text-CoT Reasoning (T): The model is prompted to generate a textual chain of thought before providing the final answer, testing text-based reasoning.
3. Simulated Visual-CoT Reasoning (V): The model is provided with annotated intermediate visual clues and textual prompts, simulating a "think-while-drawing" process.

This protocol enables fine-grained analysis of model capabilities and failure modes, providing insights into the necessity of visual reasoning for complex tasks.

Empirical Results and Analysis

Overall Performance

Experimental results reveal that MIRA is substantially more challenging than existing multimodal benchmarks. State-of-the-art closed-source MLLMs (e.g., GPT-5, Gemini 2.5 Pro, o3) achieve less than 20% accuracy with direct inputs, and even the strongest models (GPT-5) reach only 16.5% accuracy. The performance is particularly poor on puzzle tasks, which require meticulous visual reasoning (average 9.5% accuracy), compared to other categories (16.1%).

Figure 2: Left: Example from MIRA with responses from MLLMs and humans, illustrating cognitive gaps; Right: Leading MLLMs perform well on established benchmarks but fail on MIRA, with no model surpassing 20% accuracy with direct inputs.

Impact of Chain-of-Thought Prompting

Contrary to prior benchmarks, Text-CoT prompting does not yield significant improvements on MIRA. In fact, for strong models such as Gemini 2.5 Pro and o3, Text-CoT degrades performance by 18.3% and 14.0%, respectively. This negative impact is most pronounced in categories requiring detailed visual reasoning, indicating that text-only CoT is insufficient for tasks where intermediate visual states are essential.

Visual-CoT as a Solution

Providing annotated intermediate visual clues (Visual-CoT) leads to consistent and substantial performance gains across all models, with an average relative improvement of 33.7%. For example, GPT-5-mini improves from 13.7% to 23.2% accuracy, and physics tasks nearly double in accuracy for proprietary MLLMs. However, open-weight models (e.g., Qwen2.5-VL, GLM-4.5V) show more limited gains, likely due to smaller parameter scales and lack of training on interleaved Visual-CoT data. Unified models with both understanding and generation capabilities (Bagel, Janus-Pro) also benefit, but cannot yet autonomously generate intermediate images during reasoning.

Figure 3: Performance comparison of leading models across Direct, Text-CoT, and Visual-CoT settings, with pass@$k$ and majority voting scores.

Upper-Bound Probing and Prompt Engineering

Expanding the decoding search space via pass@$k$ and majority voting yields only marginal improvements, especially for stronger models, indicating that failures are not due to accidental errors but a fundamental lack of capability. Specialized Text-CoT prompts aligned with Visual-CoT reasoning paths provide modest gains (average +1.4% for closed-source, +1.5% for open-weight models), but these are dwarfed by the improvements from Visual-CoT, underscoring the inherent limitations of text-only guidance.

Case Study: Visual Reasoning Necessity

A representative failure case in Euclidean Geometry demonstrates that even the strongest model (GPT-5) cannot accurately reason about convex hulls and overlapping regions using text alone. The model's text-based reasoning is vague and unreliable, often resorting to "imagine" without concrete spatial manipulation. In contrast, providing a simple visual cue enables precise localization and correct counting, proving that visual grounding is indispensable for many reasoning tasks.

Figure 4: Failure case of Text-CoT on a geometry task; Visual-CoT enables accurate reasoning and correct answer.

Dataset Coverage and Task Diversity

MIRA's task suite covers a broad spectrum of visual reasoning challenges, including convex hull identification, cube counting, mirror transformations, puzzle completion, gear rotation, dice rolling, and more. Each task is designed to require intermediate visual states, either static or dynamic, and is paired with gold-standard visual reasoning trajectories.

Figure 5: Illustrative cases for Convex Hull (left) and Cubes Count (right).

Figure 6: Illustrative cases for Mirror Clock (left) and Cubes Missing (right).

Figure 7: Illustrative cases for Mirror Pattern (left) and Overlap (right).

Figure 8: Illustrative cases for Puzzle (left) and Trailer Cubes Count (right).

Figure 9: Illustrative cases for Puzzle (left) and Trailer Cubes Count (right).

Figure 10: Illustrative cases for Unfolded Cube (left) and Localizer (right).

Figure 11: Illustrative cases for Paper Airplane (left) and Defuse A Bomb (right).

Figure 12: Illustrative cases for Multi-piece Puzzle (left) and Electric Charge (right).

Figure 13: Illustrative cases for Rolling Dice: Top (left) and Gear Rotation (right).

Figure 14: Illustrative cases for Rolling Dice: Sum (left) and Rolling Dice: Two (right).

Implications and Future Directions

The results from MIRA highlight several critical implications for the development of MLLMs:

• Text-only reasoning is fundamentally insufficient for tasks requiring manipulation of visual states; prompt engineering cannot overcome this limitation.
• Visual-CoT provides a substantial but temporary solution; current models benefit from human-annotated visual clues but lack autonomous "think-while-drawing" capabilities.
• Unified multimodal architectures with integrated image generation are necessary to close the gap, but require new training paradigms and datasets that tightly couple visual and textual reasoning.
• Open-weight models lag behind closed-source counterparts in leveraging visual clues, indicating a need for more open, large-scale multimodal pretraining and evaluation resources.

MIRA establishes a reproducible platform and metric system for the rigorous evaluation of visual reasoning in MLLMs, setting a new standard for future research in unified multimodal intelligence.

Conclusion

MIRA exposes the fundamental limitations of current MLLMs in complex reasoning tasks that require intermediate visual representations. The benchmark demonstrates that text-only CoT is inadequate, while Visual-CoT yields significant gains but is not a complete solution. There is an urgent need for models capable of generating and utilizing intermediate visual states during reasoning, with tightly integrated vision-language architectures. MIRA provides the necessary infrastructure for evaluating and advancing such capabilities, and will be instrumental in guiding future developments in multimodal AI systems.