Mind-Bench: Multimodal Knowledge & Reasoning

• Mind-Bench is a benchmark designed for evaluating unified text-to-image and multimodal models on dynamic knowledge grounding and advanced visual reasoning tasks.
• It spans 10 sub-domains—with 500 curated samples—covering both knowledge-driven tasks (e.g., live news, weather) and reasoning-driven tasks (e.g., math, spatial inference).
• The evaluation employs Checklist-based Strict Accuracy (CSA) alongside ancillary metrics to pinpoint gaps between current model capabilities and cognitively equipped, agentic outputs.

Mind-Bench is a rigorously constructed evaluation benchmark designed to assess the capability of unified text-to-image and multimodal models in grounding up-to-date external knowledge and carrying out complex visual reasoning. Mind-Bench emerges from the context of growing recognition that most existing generative models are limited to static text-to-pixel mappings, with insufficient sensitivity to implicit intent and lacking mechanisms for just-in-time knowledge integration or explicit reasoning. Mind-Bench provides a comprehensive set of tasks intentionally crafted to span both knowledge-intensive and reasoning-driven image synthesis, thereby establishing a standard for progress toward agentic, cognitively equipped generative systems (He et al., 2 Feb 2026).

1. Design Principles and Benchmark Scope

Mind-Bench is motivated by the need to systematically stress-test models on two fronts: (1) real-time external knowledge grounding, and (2) advanced multimodal and logical reasoning. Existing image generation evaluations focus predominantly on aesthetic or structural fidelity and tend to neglect factual accuracy, recentness of world knowledge, and deductive capacity. Mind-Bench explicitly fills this gap by incorporating both knowledge-driven and reasoning-driven scenarios. The benchmark contains 500 samples, split evenly across 10 sub-domains. Five are knowledge-driven (reliant on external, often dynamic factual content); five require explicit reasoning, including mathematical, spatial, and commonsense inference.

Knowledge-Driven Sub-domains

• Special Events (live news scenes)
• Weather (spatially and temporally indexed meteorological states)
• Character (factual description or portrayal of IP-protected entities)
• IP (novel product or artifact concepts)
• World Knowledge (historically or culturally grounded objects or events)

Reasoning-Driven Sub-domains

• Life Reasoning (commonsense inference)
• Geo Understanding (interpretation and reasoning over maps, spatial data)
• Math (geometric and algebraic visualization)
• Science & Logic (physical processes and logic puzzles)
• Poem (literary, metaphor-derived imagery)

Samples are distributed uniformly (50 per sub-domain) and constructed to challenge retrieval, disambiguation, and both visual and semantic synthesis.

2. Sample Structure and Data Collection

Each Mind-Bench sample includes:

• Instructional prompt (Iinst), often mirroring real user intent
• Optional input image for image-to-image (I2I) tasks
• Human-curated reference evidence: authoritative text and/or reference images
• A strict evaluation checklist (C = {c₁, ..., cₖ}), specifying all atomic criteria required for successful generation

Prompts are selected and curated by graduate-level annotators to maximize coverage and difficulty. For each item, requisite evidence is retrieved (from news, Wikipedia, etc.), a checklist is constructed (with the assistance of LLMs), and human validation ensures that the criteria are both non-redundant and executable by judges or models.

For example, a prompt might request “a cinematic shot of the 2025 UEFA Champions League final scoreboard showing Spain 2–1 England,” anchoring the evaluation against real-world outcomes and visual detail.

3. Evaluation Protocols and Metrics

The principal evaluation metric for Mind-Bench is Checklist-based Strict Accuracy (CSA), defined by the criterion that a generated image receives full credit only if it satisfies all atomic checklist requirements. Mathematically, for $N$ test samples with per-sample checklist $C_i$, define:

$$\mathrm{Acc_{CSA}} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[\forall\,c\in C_i:\,\mathrm{VQA}(I_{\text{gen},i},c)=1\right]$$

where $\mathrm{VQA}(I_{\text{gen},i},c)$ indicates success on individual checklist item $c$, and $\mathbf{1}[\cdot]$ is the indicator function. This metric penalizes overgeneralization and only rewards outputs satisfying every factual and logical element.

Ancillary metrics include:

• WISE WiScore: Weighted sum of Consistency, Realism, and Aesthetic quality (with weights $w_1=0.7$, $w_2=0.2$, $w_3=0.1$).
• RISEBench Strict-Success Accuracy: Proportion of images scoring highest (5/5) on Instruction-Reasoning, Appearance-Consistency, and Visual-Plausibility.

The evaluation framework thus emphasizes not only semantic fidelity but also compliance with dynamic and precise factual constraints.

4. Model Performance and Baseline Results

Mind-Bench exposes a severe performance gap between current state-of-the-art models and robust knowledge- or reasoning-grounded synthesis.

Model	AccCSA (Overall)
GPT-Image-1	0.17
GPT-Image-1.5	0.21
FLUX-2 Pro	0.21
FLUX-2 Max	0.23
Nano Banana	0.18
Nano Banana Pro	0.41
SD-XL	0.00
SD-3.5 Large	0.04
Bagel	0.02
Echo-4o	0.02
DraCo	0.02
Z-Image	0.02
Qwen-Image	0.02
Mind-Brush (Qwen-Image)	0.31
Mind-Brush (GPT-Image-1)	0.34

Mind-Brush, which augments a T2I backbone with an agentic workflow comprising both dynamic retrieval (Asearch) and explicit reasoning (Areasoning), achieves up to 0.31 AccCSA—a marked improvement over all open-source and most proprietary models. Ablation experiments confirm the orthogonal benefits of explicit retrieval (improving knowledge-driven categories) and chain-of-thought reasoning (improving logic-dependent categories). For instance, on “Special Events,” Mind-Brush (Qwen-Image) attains 0.54 AccCSA, compared to 0.08 for its base model.

5. Analysis of Failure Modes and Domain Challenges

Mind-Bench reveals consistent failure patterns across both open-source and proprietary models:

• Knowledge-Driven Tasks: Models hallucinate details for out-of-distribution entities (e.g., novel IP characters or event-specific artifacts) and exhibit temporal drift, producing outdated or generic artifacts for current events.
• Reasoning-Driven Tasks: Baselines struggle with formal logical deduction (e.g., angle-chasing in geometry), spatial viewpoint inference (map-based tasks), and often produce images that are semantically or structurally incorrect.
• Hallucination and Under-annotation: Generic landscapes or irrelevant object labels frequently substitute for instructed detail. For ingredient or component extraction tasks, non-target objects may be labeled—or true positives missed.
• Ablation Insights: The retrieval agent singularly boosts knowledge-intensive categories, while the reasoning agent is critical for mathematical and logical tasks. When both are present, synergistic gains surpass additive improvement.

6. Implications and Future Research Directions

Mind-Bench highlights the limitations of static, prior-driven generative models and establishes a standard for evaluating hybrid agentic systems that couple retrieval and reasoning within the generation loop. It demonstrates that even leading unified models—when evaluated under strict, checklist-based factual and logical constraints—struggle to approach the requirements of real-world, intent-aligned image synthesis.

Future progress will likely depend on further advances in agentic integration, enhanced evidence retrieval, scaling of step-wise reasoning, and improved mechanisms for dynamic world knowledge acquisition. A plausible implication is that next-generation multimodal foundations will need to tightly couple retrieval, interaction, and explicit reasoning modules to overcome the generalization gaps Mind-Bench exposes (He et al., 2 Feb 2026).