ATP-Bench: Agentic Tool Planning Benchmark

• ATP-Bench is a benchmark for agentic tool planning in multimodal generation that tests when and which visual tools to deploy for enhanced answer structuring.
• It comprises 7,702 QA pairs across eight categories and 25 visual-critical intents, leveraging a unified five-tool ecosystem including reference, diffusion, search, code, and edit.
• The benchmark employs the MAM framework to rigorously assess planning quality, tool choice, and integrated visual orchestration with high human evaluative correlation.

Searching arXiv for ATP-Bench and closely related benchmark context. ATP-Bench is a benchmark for agentic tool planning in interleaved multimodal generation, introduced in “ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation” (Liu et al., 31 Mar 2026). It evaluates whether a Multimodal LLM can act as a controller that decides when visual content is required, where it should be inserted in a response, and which tool should be invoked to obtain or construct that content. The benchmark contains 7,702 QA pairs, including 1,592 VQA pairs, spans eight categories and 25 visual-critical intents, and is paired with human-verified interleaved ground truths and an execution-independent judge framework called MAM (Liu et al., 31 Mar 2026).

1. Conceptual scope and motivation

ATP-Bench is organized around a specific failure mode in contemporary interleaved text-and-image generation: existing systems often treat image generation and retrieval augmentation as separate paradigms, even though many realistic queries require both factual grounding and creative synthesis in a single response (Liu et al., 31 Mar 2026). In the benchmark’s formulation, a strong system is not merely one that can generate an image or retrieve one; it must identify a visual gap in the answer, choose an appropriate acquisition or manipulation mechanism, and place the resulting visual content coherently in the surrounding prose.

The benchmark therefore targets visual-critical queries: prompts for which image-text interleaving offers substantially higher utility than text alone. The paper motivates this notion through three recurring functions of visual content: visual information augmentation, cognitive acceleration, and structural illustration (Liu et al., 31 Mar 2026). This framing distinguishes ATP-Bench from plain multimodal question answering and from end-to-end image-rendering leaderboards. A common misconception is to view it as a benchmark of final multimedia aesthetics; more precisely, it is a benchmark of planning quality under multimodal tool affordances.

The acronym “ATP” in this work denotes Agentic Tool Planning. A plausible source of confusion is that other benchmark literature uses “ATP” for automated theorem proving, as in TaoBench (Taylor et al., 13 Mar 2026) and GRUNGE (Brown et al., 2019). ATP-Bench belongs instead to the multimodal generation and tool-use literature.

2. Formal task definition and tool space

The benchmark formalizes the input as a visual-critical query $q$ together with a document set

$\mathcal{D} = \{d_1, d_2, \dots, d_n\},$

where each document is represented as

$d_i = (T_i, \mathcal{I}_i),$

with $T_i$ the text and $\mathcal{I}_i$ the associated image set. The model receives an interleaved-generation prompt $p$, the query $q$, and the document set $\mathcal{D}$, and must produce an ordered sequence

$$R = \{s_1, s_2, \dots, s_m\}, \quad s_j \in \mathcal{S}_{\text{text} \cup \mathcal{S}_{\text{tool}}}.$$

Here, $\mathcal{S}_{\text{text}}$ denotes natural-language tokens and $\mathcal{D} = \{d_1, d_2, \dots, d_n\},$0 denotes structured tool-calling instructions (Liu et al., 31 Mar 2026).

The benchmark defines a five-tool ecosystem:

• Reference: cites an existing image from the provided document set via an img_index.
• Diffusion: synthesizes a new image from a descriptive prompt.
• Search: retrieves real-world images through an external search engine.
• Code: generates data-driven visualizations such as charts, plots, and blueprint-style diagrams.
• Edit: modifies a referenced image from the provided document set.

Tool calls use a unified schema: $\mathcal{D} = \{d_1, d_2, \dots, d_n\},$1 The prompting policy requires tool use to be proactive rather than merely reactive: the model is expected to call a tool when it determines that a visualization would materially improve the answer, even if the prompt does not explicitly request an image (Liu et al., 31 Mar 2026).

This tool ontology is central to the benchmark’s design. Reference and search support factual grounding; diffusion and edit support creative synthesis and transformation; code addresses data-heavy or diagrammatic content. The benchmark’s core claim is that realistic multimodal assistance requires these capabilities to be orchestrated jointly rather than treated as mutually exclusive modes.

3. Dataset construction and category structure

ATP-Bench is constructed in two main stages: query collection and ground-truth collection (Liu et al., 31 Mar 2026). For query collection, the benchmark defines eight high-visual-demand categories: Academic, Manual, Recipe, Fashion, Renovation, Product, Travel, and Encyclopedia. Source documents are collected from MRAMG-Bench, RAG-IGBench, and OVEN, with underlying sources including Wikipedia, Xiaohongshu, and arXiv. A taxonomy of 25 visual-critical intents is then used to guide query synthesis.

Query generation is performed by Gemini 2.5 Pro in a two-stage pipeline. In the first stage, the model identifies relevant intents and synthesizes natural text-only queries from source documents. In the second, some of these are converted into VQA-style queries by replacing key textual evidence with images when the intent can be visually grounded. To support the VQA portion, the visual corpus is assembled from web search, generative synthesis using nano-banana, and public benchmarks such as OVEN. The generated queries are then reviewed by ten professional annotators, who remove ambiguous or unnatural prompts, non-grounded VQA pairs, and prompts that do not satisfy the benchmark’s visual-criticality criterion. After filtering, roughly 95% of generated queries are retained (Liu et al., 31 Mar 2026).

Overall dataset statistics are 7,702 queries, 2.15 docs/query, 8.87 images/query, and 4.32 tool calls per ground-truth answer. The category breakdown is as follows:

Category	Queries	Avg. tool calls/GT
Academic	196	1.96
Manual	1,025	4.38
Recipe	1,960	3.06
Fashion	1,051	4.63
Renovation	440	4.79
Product	1,330	4.58
Travel	734	8.00
Encyclopedia	966	3.65

The 25 intents are distributed across these categories. Examples include Framework, Component, Comparison, Dataset, Result in Academic; Operation Guide, Function, Component in Manual; Guideline, Ingredient in Recipe; Hairstyle, Makeup, Outfit, Photography in Fashion; Element, Style in Renovation; ProductIntro, Comparison, Authenticity in Product; Planning, Navigation in Travel; and Animals, Geography, Biography, Architecture in Encyclopedia (Liu et al., 31 Mar 2026).

4. Ground-truth interleaving and annotation protocol

Ground-truth construction follows three stages: textual response generation, image insertion, and fine-grained annotation and refinement (Liu et al., 31 Mar 2026). First, an MLLM produces a high-quality text-only answer grounded in the provided documents and images. Second, another stage identifies visual gaps and inserts tool calls after the relevant paragraphs, using explicit capability boundaries for reference, search, diffusion, edit, and code. The insertion policy forbids mid-sentence placement, filler phrases that merely announce a figure, and repeated use of a reference image. Third, 15 annotators review every tool call.

A tool call is retained only if the expected image materially improves understanding, uses correct parameters, and is placed immediately after the relevant text. Annotators remove redundant or mismatched calls, fix malformed syntax and incorrect tool choices, and move misplaced tags to paragraph boundaries. Entire samples are discarded if they contain systemic errors such as broken markdown structure or serious factual mistakes (Liu et al., 31 Mar 2026).

This annotation regime is important because ATP-Bench does not treat tool tags as incidental metadata. The ground truth is itself an interleaved plan, not merely a plain-language answer accompanied by post hoc labels. The benchmark thereby evaluates the coupling between answer composition and visual orchestration.

5. MAM: execution-independent evaluation of planning

ATP-Bench’s evaluation framework is MAM, a Multi-Agent MLLM-as-a-Judge system designed to assess planning independently of exact tool execution and independently of fixed backend implementations (Liu et al., 31 Mar 2026). This design follows from two assumptions stated in the benchmark: multiple valid visual plans can exist for the same query, and tool backends such as search engines, diffusion models, and editors change over time.

MAM comprises three agents:

• Precision Inspector: evaluates each tool call that appears in the response.
• Recall Inspector: detects missed opportunities for tool use.
• Chief Judge: integrates the reports into a holistic final score.

The Precision Inspector first checks whether a tool call is necessary and whether the chosen tool respects the capability boundary. A call survives only if both 1a_necessity_pass and 2a_tool_choice_pass are true. It then scores four additional properties: 1b Semantic Position, 1c Structural Integrity, 2b Parameter Accuracy, and 2c Format Correctness. For 1b, 1c, and 2b the rubric is 0 = failure, 1 = partial satisfaction, 2 = full satisfaction; for 2c, 2 = strict JSON format, 0 = broken format (Liu et al., 31 Mar 2026).

The Recall Inspector identifies omitted tool use under six explicit omission patterns: explicit deictic failure, contextual reference omission, real-world entity text-wall, creative/abstract visualization omission, unfulfilled image modification, and data & blueprint omission. It outputs a list of missed opportunities and a missed_count.

The Chief Judge maps these signals to a Final Score on a 0–100 scale. The benchmark defines five bands: Excellent (80–100), Good but Flawed (60–80), Mediocre (40–60), Poor (20–40), and Fatal (0–20) (Liu et al., 31 Mar 2026). The main reported metrics are:

• Final Score (FS): holistic 0–100 score.
• Success Rate (SR): precision-style success over emitted tool calls.
• Missed Images (MI): count of missed visual opportunities.
• Tool Adoption Rate: percentage of queries in which a tool is invoked at least once.

The paper also reports set-overlap Precision, Recall, and F1-score for tool invocation against the benchmark ground truth. The ranking induced by this F1 analysis has a high Spearman correlation with the MAM Final Score ranking,

$\mathcal{D} = \{d_1, d_2, \dots, d_n\},$2

which suggests strong agreement between trace-overlap and judge-based evaluation (Liu et al., 31 Mar 2026).

6. Empirical performance and observed failure modes

ATP-Bench evaluates 10 state-of-the-art MLLMs in a default zero-shot setup: Claude Sonnet 4.5, Claude Sonnet 4, Gemini 3 Pro, Grok-4.1 Fast Reasoning, GPT-5, GPT-4o, Qwen3-VL-Plus, Qwen2.5-VL-72B, LLaMA-3.2-11B, and InternVL3.5-14B (Liu et al., 31 Mar 2026). The strongest model overall is Gemini 3 Pro, which achieves:

• Average FS = 79.88
• Average SR = 81.77
• Average MI = 0.49

The next tier consists of Claude Sonnet 4.5, Claude Sonnet 4, Grok-4.1, and GPT-5, all with average Final Scores in the high-60s. Mid-tier systems include Qwen3-VL-Plus and GPT-4o, while Qwen2.5-VL-72B, InternVL3.5-14B, and LLaMA-3.2-11B perform substantially worse (Liu et al., 31 Mar 2026).

Category-level behavior is uneven. Academic and Encyclopedia are comparatively easy because they require less complex tool coordination. Travel and Renovation are hardest. Travel is characterized as MI-dominated, because answers often require many tool calls—ground-truth responses average 8.00 calls—and models frequently omit needed visuals. Renovation is more SR-dominated, because models often detect that visuals are needed but choose incorrect tools or produce poor tool execution plans (Liu et al., 31 Mar 2026).

Tool-use behavior also varies systematically. Reference dominates so strongly that the paper omits it from one adoption-rate figure; all models use it in over 90% of queries. Among the remaining tools, diffusion is most common, followed by search; edit is moderately used, and code is rare. The paper characterizes GPT-5 as conservative, with very high tool-set precision but lower recall, whereas Gemini 3 Pro is more balanced. Qwen3-VL-Plus is described as generation-oriented, leaning toward diffusion and edit (Liu et al., 31 Mar 2026).

Few-shot prompting reveals substantial latent variability. With 3-shot prompting, GPT-4o improves from FS 60.35 / SR 63.05 / MI 1.39 to FS 73.19 / SR 82.01 / MI 0.83, and Qwen2.5-VL-72B improves from FS 53.88 / SR 39.06 / MI 1.64 to FS 72.86 / SR 72.88 / MI 0.65. LLaMA-3.2-11B improves only marginally, from FS 29.60 / SR 25.35 / MI 2.10 to FS 30.20 / SR 26.62 / MI 2.01 (Liu et al., 31 Mar 2026). This suggests that some models primarily lack calibration and exemplars, whereas others lack the underlying planning competence.

The benchmark isolates several failure modes. InternVL3.5-14B is described as suffering from an instruction-following bottleneck, especially in Format Correctness, which is only 27.95%. LLaMA-3.2-11B often inserts unnecessary visuals, reflected in a Necessity score of 44.99%. Mid-tier models such as Qwen2.5-VL-72B often choose reasonable insertion sites but produce inaccurate parameters, such as incorrect img_index values or vague prompts (Liu et al., 31 Mar 2026).

7. Validation, limitations, and benchmark significance

ATP-Bench includes several validation layers for its judge-centric methodology. In a judge-model ablation on an 800-query subset, replacing the MAM agents with Claude Sonnet 4.5 or GPT-5 yields largely stable rankings. Inter-judge Spearman correlations are high: for FS, 0.952, 0.970, and 0.898 across judge pairs; for SR, 1.000, 0.881, and 0.881; and for MI, 0.922, 0.952, and 0.946, all with $\mathcal{D} = \{d_1, d_2, \dots, d_n\},$3 (Liu et al., 31 Mar 2026).

Human agreement studies further support the framework. On 400 queries from all models, agreement rates are 84.00% for the Precision Inspector, 85.88% for the Recall Inspector, and 88.00% for the Chief Judge. In a supplementary human evaluation on 100 responses per model with actual tool execution, MAM rankings correlate strongly with human rankings: $\mathcal{D} = \{d_1, d_2, \dots, d_n\},$4 for Final Score, and

$\mathcal{D} = \{d_1, d_2, \dots, d_n\},$5

for Missed Images (Liu et al., 31 Mar 2026).

The benchmark’s limitations are explicitly stated. It is restricted to text-image interleaved outputs; it exposes only five tools; and it focuses on direct MLLM-as-a-judge evaluation rather than alternative judge pipelines (Liu et al., 31 Mar 2026). In addition, ATP-Bench is presented primarily as an evaluation benchmark; no explicit train/validation/test split is specified in the paper excerpt. A plausible implication is that benchmark reuse will depend not only on dataset access but also on how future work standardizes evaluation subsets and prompting protocols.

Its broader significance lies in shifting the evaluation target from output rendering alone to structured multimodal planning. ATP-Bench argues, and empirically supports, that the key capability gap is not simply image quality or text quality but the ability to decide whether an image should appear, which visual mechanism should supply it, and how that decision should be integrated into the answer. In that sense, ATP-Bench functions both as a benchmark and as a diagnosis framework for multimodal systems that aim to unify factuality and creativity through agentic orchestration rather than through a single monolithic generation pathway (Liu et al., 31 Mar 2026).