IA-Bench: Agentic Image Generation Benchmark
- Image Agent Bench (IA-Bench) is a benchmark that evaluates agentic text-to-image generation by measuring context gap recovery via planning, reasoning, search, and memory.
- It operationalizes 17 subtasks across 730 test instances to determine if models can extract and use missing context from underspecified user inputs.
- The benchmark employs rigorous checklist-based metrics such as Checklist Accuracy and Pass Rate to expose strengths and failure modes in multi-turn image generation.
Searching arXiv for the benchmark paper and, if available, related benchmark references mentioned in the provided data. Image Agent Bench (IA-Bench) is an evaluation-only benchmark introduced in "Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation" (Zhang et al., 25 Jun 2026) to quantify the agentic capabilities required for real-world text-to-image generation. It is designed around the claim that practical image requests are often underspecified, implicit, or dependent on up-to-date knowledge, producing a "Context Gap" between user context and the sufficient generation context for text-to-image systems. Within that framing, IA-Bench measures whether a model can identify missing context in an underspecified user request, acquire that context via reasoning, search, memory, and feedback, and incorporate it to produce images that satisfy all implicit and explicit constraints (Zhang et al., 25 Jun 2026).
1. Conceptual basis and benchmark objective
IA-Bench is organized around four core image agent capabilities: Plan, Reason, Search, and Memory (Zhang et al., 25 Jun 2026). Its stated goal is not merely to assess rendering quality, but to evaluate whether a model can act like an agent by decomposing tasks, inferring hidden requirements, retrieving up-to-date information, and maintaining dialog history. In this formulation, the benchmark targets real-world text-to-image generation settings in which prompt following alone is insufficient.
The benchmark’s underlying problem formulation is tied to the broader "Context Gap" identified in the Qwen-Image-Agent framework (Zhang et al., 25 Jun 2026). There, user input is treated as partial context rather than a complete specification. IA-Bench operationalizes that view by testing whether systems can recover and use missing context before image synthesis. This suggests that IA-Bench is intended less as a pure image-generation benchmark than as an agent-evaluation benchmark instantiated through image generation.
2. Capability structure and task taxonomy
IA-Bench comprises four capability categories, 17 subtasks, 730 test instances, and 1,801 fine-grained binary checklist items used for evaluation (Zhang et al., 25 Jun 2026). The capability decomposition is explicit and task-centered.
| Capability | # Subtasks | Subtask Names |
|---|---|---|
| Plan | 3 | Composition, Enumeration, Multi-Panel |
| Reason | 6 | Math, Science, Commonsense, Maze, Map, Geometry |
| Search | 6 | Game, Movie, Anime, Celebrity, Stock, Weather |
| Memory | 2 | User Profile, Conversation History |
The Plan capability evaluates decomposition of a high-level goal into concrete visual arrangements. The benchmark examples include counting objects, laying them out in a specified pattern, and splitting a multi-panel scene into sub-images. The Reason capability evaluates inference over implicit requirements before generation, including logical, commonsense, and mathematical or visual reasoning. The Search capability evaluates retrieval of external factual or visual context not present in the prompt, with two sub-branches: IP-related tasks such as Game, Movie, Anime, and Celebrity, and information-driven tasks such as Stock and Weather. The Memory capability evaluates preservation and reuse of information across turns, specifically through User Profile and Conversation History (Zhang et al., 25 Jun 2026).
This taxonomy matters because it makes explicit that "agentic image generation" is being defined as more than tool invocation. It includes structured decomposition, latent-constraint inference, external grounding, and cross-turn state retention. A plausible implication is that IA-Bench can expose failure modes that conventional prompt-response image benchmarks would not isolate.
3. Dataset construction and prompt design
IA-Bench is composed exclusively of held-out evaluation cases; it has no train/val split, and all 730 instances are treated as test cases (Zhang et al., 25 Jun 2026). The benchmark is therefore intended for assessment rather than supervised fitting on benchmark-specific data.
Representative prompts illustrate the design. For Plan–Enumeration, one example is: "Generate an image showing exactly three red apples, four green apples, and five yellow bananas arranged in a spiral layout." For Plan–Multi-Panel, one example is: "Create a 2×2 comic panel: panel 1 shows a cat, panel 2 a dog, panel 3 a mouse, panel 4 them playing together." For Reason–Maze, one example is: "Here is a maze with start (S) and goal (G). Draw only the shortest path from S to G." For Reason–Geometry, one example is: "Draw a square of side length 100 px, then inscribe a circle that touches all four sides." For Search–Stock, one example is: "Show a candlestick chart of Apple Inc. (AAPL) closing price for last Friday." For Search–Game, one example is: "Generate the main character from the video game ‘The Legend of Zelda’ in a heroic pose." For Memory–User Profile, one example is: "User Alex loves oil-painting style and blue color palettes. Now generate a seascape in that style." For Memory–Conversation History, one example is: "Previously you generated a red sports car. Now show the same car from its left side" (Zhang et al., 25 Jun 2026).
The benchmark instances were manually filtered so that the prompt cannot be solved by mere visual priors of large models, the request requires nontrivial context acquisition, and ambiguities are minimized with every requirement captured in the accompanying checklist (Zhang et al., 25 Jun 2026). This curation criterion is central to IA-Bench’s identity. It distinguishes the benchmark from distributions in which memorized iconic imagery or generic aesthetic competence would dominate measured performance.
4. Evaluation protocol and formal metrics
IA-Bench uses a checklist-based evaluation protocol. For each test instance , a human-written checklist enumerates all visual constraints implied by the prompt. After generation, a pretrained vision-LLM automatically judges each item according to
$\mathrm{VLM}(I^i_{\mathrm{gen},\,c^i_j)\;\in\;\{0,1\}$
where 1 means "item satisfied" and 0 means "item violated" (Zhang et al., 25 Jun 2026).
The benchmark defines three principal metrics. Checklist Accuracy (CA) is the average fraction of checklist items satisfied per instance:
$\mathrm{CA} = \frac1N \sum_{i=1}^N \Bigl(\frac1{K_i} \sum_{j=1}^{K_i} \mathrm{VLM}(I^i_{\mathrm{gen},\,c^i_j)\Bigr)\,.$
Pass Rate (PR) is a strict success rate under which an instance is counted as passed only if all its checklist items are satisfied:
$\mathrm{PR} = \frac1N \sum_{i=1}^N \prod_{j=1}^{K_i} \mathrm{VLM}(I^i_{\mathrm{gen},\,c^i_j)\,.$
Image-Agent Score (IA-score) aggregates performance across the four capabilities:
where each of Plan, Reason, Search, and Memory is the micro-average, either CA or PR, on that dimension (Zhang et al., 25 Jun 2026).
The evaluation pipeline is defined in four stages: prompt to agentic context construction, rendering, automated scoring, and aggregation. In the first stage, the model or baseline identifies missing context, invokes sub-modules such as reason, search, memory, and feedback, and produces a final grounded prompt. In the second stage, a fixed image-generation backbone such as Qwen-Image-2.0 produces the output image. In the third stage, a VLM checks each checklist item, and no human scoring is used in the IA-Bench evaluation proper. In the fourth stage, CA, PR, and IA-score are aggregated across subtasks and capabilities (Zhang et al., 25 Jun 2026).
The use of strict PR alongside itemwise CA is methodologically significant. CA rewards partial compliance with a compound instruction, whereas PR requires complete satisfaction of all enumerated constraints. This suggests that IA-Bench is designed to penalize systems that are competent on average but brittle on conjunctions of requirements.
5. Benchmark statistics and empirical profile
IA-Bench contains 17 subtasks, 730 test instances, and 1,801 checklist items (Zhang et al., 25 Jun 2026). The reported capability-level distribution is approximate by instances and explicit by average number of checklist items per instance.
| Capability | # Subtasks | Approx. Instances |
|---|---|---|
| Plan | 3 | ~130 |
| Reason | 6 | ~260 |
| Search | 6 | ~260 |
| Memory | 2 | ~80 |
The average number of checklist items per instance is reported as 2.5 for Plan, 2.7 for Reason, 2.8 for Search, 2.3 for Memory, and approximately 2.47 overall (Zhang et al., 25 Jun 2026). These values indicate that most instances involve multiple simultaneous constraints rather than a single binary property.
The larger allocation to Reason and Search, each with approximately 260 instances, reflects the benchmark’s emphasis on inference and retrieval rather than solely compositional layout or dialog carryover. A plausible implication is that aggregate scores will be influenced more strongly by performance on external grounding and latent-constraint resolution than by memory alone, although the explicit IA-score weighting also separately assigns Memory a smaller coefficient of 0.1 (Zhang et al., 25 Jun 2026).
6. Baselines, reported results, and interpretive findings
The reported comparison set includes proprietary systems such as GPT-Image-1, GPT-Image-1.5, and Nano Banana Pro; open-source models such as Stable Diffusion variants, FLUX, and Qwen-Image-2.0; and prior agentic pipelines such as GenSearcher, GEMS, MindBrush, and SCOPE (Zhang et al., 25 Jun 2026). The excerpted IA-Bench results are as follows.
| Model | IA-score |
|---|---|
| GPT-Image-1.5 | 35.7 % |
| Nano Banana Pro | 42.6 % |
| MindBrush | 30.2 % |
| SCOPE | 30.9 % |
| Qwen-Image-Agent | 45.4 % |
For the same set of systems, Qwen-Image-Agent is reported with 72.9% Plan CA, 65.5% Reason CA, 67.6% Search CA, 73.6% Memory CA, 45.3% Plan PR, 43.7% Reason PR, 46.1% Search PR, and 49.0% Memory PR, alongside an IA-score of 45.4% (Zhang et al., 25 Jun 2026). The paper reports an absolute gain of +2.3 points IA-score over the best prior agentic baseline, SCOPE, and a relative gain of +6.5 points, or 17%, over the strongest closed-source direct generator, Nano Banana Pro (Zhang et al., 25 Jun 2026).
The same source further reports state-of-the-art generalization on WISE-Verified and MindBench, with WISE-Verified overall accuracy of 90.2% versus 87.6% by Nano Banana Pro, and MindBench overall score of 0.42 versus 0.41 by Nano Banana Pro (Zhang et al., 25 Jun 2026). No formal statistical tests were reported, but the paper states that improvements are consistent across three benchmarks and all four dimensions.
Several key findings are explicitly highlighted. Agentic pipelines excel at Plan, Reason, and Search; even strong direct generators such as Qwen-Image-2.0 achieve only approximately 17% IA-score, whereas agentic models exceed 30%. Memory remains challenging; closed-source models with implicit fine-tuned user context still edge out many open-source agentic methods on Memory alone. Unified context-centric frameworks that combine planning and grounding through reason, search, memory, and feedback yield balanced gains across all four dimensions (Zhang et al., 25 Jun 2026).
7. Limitations and prospective extensions
The benchmark description identifies five limitations. First, some real-world requests contain highly implicit context gaps hidden behind subtle cultural or historical references, which neither LLMs nor search easily capture. Second, the boundary between what can be reasoned and what can be retrieved shifts with LLM size and knowledge cutoffs. Third, feedback remains coarse because checklist feedback is generic; stronger task-specific signals or learned reward models could yield larger iterative gains. Fourth, multi-turn settings can lead to context explosion, where accumulated multimodal context exceeds token budgets and requires smarter relevance pruning. Fifth, the full agentic loop incurs substantial API calls and inference cost compared to one-shot generation (Zhang et al., 25 Jun 2026).
The same source outlines several future directions: stronger, knowledge-rich MLLMs for identifying highly implicit gaps; learned tool-use policies for deciding when to reason, search, or retrieve memory; efficient context managers for very long multi-turn dialogs; richer feedback loops combining VLM checklists with human or learned reward models; and benchmark expansion into domains such as interactive 3D scenes or video (Zhang et al., 25 Jun 2026).
These limitations clarify what IA-Bench currently measures and what it leaves open. In particular, the benchmark formalizes a constrained operationalization of agentic image generation through checklist-satisfying context acquisition. This suggests that future benchmark evolution may depend on whether richer interaction regimes, denser supervision, or more open-ended multimodal tasks can be evaluated with comparable rigor without sacrificing the explicitness that the checklist protocol provides.