MM-ToolSandBox Benchmark

• MM-ToolSandBox is a unified framework that evaluates multimodal agents by integrating visual and textual inputs to invoke tools across multi-turn dialogues.
• The framework features a stateful environment with 511 tools spanning 16 domains and supports dynamic, multi-image reasoning with realistic user behavior.
• Empirical results reveal that even top-performing models face challenges in precise visual extraction and robust tool orchestration, highlighting planning-to-precision gaps.

Searching arXiv for MM-ToolSandBox and closely related benchmarks to ground the article in the cited literature. MM-ToolSandBox is a unified framework and benchmark for evaluating visually grounded tool-calling agents in a stateful environment. It targets multimodal agents that must interpret images, select and invoke tools, update persistent world state, and sustain coherent behavior across multi-turn interactions. The framework extends earlier text-only ToolSandBox-style evaluation toward visual inputs, multi-image reasoning, and mutable application environments, while situating itself alongside related efforts on multimodal multi-hop tool-use benchmarking and MCP-oriented tool execution analysis (Ma et al., 13 Jul 2026, Zhou et al., 21 Nov 2025, Tan et al., 3 Jan 2026).

1. Definition and scope

MM-ToolSandBox is designed for agents that solve tasks by calling tools in response to visual and textual inputs. In the formulation reported for the benchmark, a visually grounded tool-calling agent must see images such as photos, documents, charts, and UI screenshots; extract task-relevant information from them; decide which tools to call in a large tool space; execute those tools against a mutable environment state; and continue doing so across multiple turns as new instructions and images arrive (Ma et al., 13 Jul 2026). The framework therefore evaluates an end-to-end pipeline of perception, reasoning and planning, tool invocation, state updates, and dialogue management.

The benchmark addresses gaps in prior tool-use evaluation settings that were mostly text-only, often single-turn, and frequently stateless or restricted to relatively small tool suites. In contrast, MM-ToolSandBox provides a stateful execution environment with 511 tools across 16 application domains, supports multi-turn and multi-image conversations, and incorporates realistic user behavior including goal changes, user errors, and state mutations (Ma et al., 13 Jul 2026). This emphasis on statefulness and conversational evolution distinguishes it from benchmarks centered on isolated API sequences or static task specifications.

A useful contextual comparison is M3-Bench, which evaluates multimodal tool use under the Model Context Protocol with an emphasis on multi-hop, multi-threaded workflows, cross-tool dependencies, and persistence of intermediate artifacts (Zhou et al., 21 Nov 2025). MM-ToolSandBox shares the commitment to visually grounded, multi-step tool use, but its primary abstraction is a stateful simulated application ecosystem rather than MCP server trajectories. This suggests that MM-ToolSandBox focuses more strongly on environment mutation, multi-turn dialogue, and entity-level correctness, whereas M3-Bench foregrounds tool-call trajectory structure.

2. Environment architecture and execution model

At the core of MM-ToolSandBox is a stateful execution environment in which each evaluation episode is a scenario specifying initial world state, expected final world state, available tools, user instructions and their evolution across rounds, associated images, and completion criteria (Ma et al., 13 Jul 2026). Interaction is tri-partite: the agent plans and calls tools, a user simulator provides instructions and corrections, and the environment executes calls while maintaining persistent state.

The environment simulates 16 mobile or service application domains. The tool suite comprises 511 native Python tools across these domains, including search tools, UI tools for interactive UI mode, vision tools for image manipulation and rendering, and system-level utilities such as file system, contacts, and messaging functionality (Ma et al., 13 Jul 2026). Tool metadata includes typed signatures as well as role visibility and state dependencies. Because the full tool list is too large for a single prompt, the framework exposes a tool discovery meta-tool, search_tool or api_docs_search_api_docs, and maintains an LRU-bounded active tool set (Ma et al., 13 Jul 2026). This design turns tool discovery itself into part of the agentic problem.

Statefulness is not incidental but structural. Persistent entities such as contacts, calendar events, cart entries, notes, tasks, and orders can be created, updated, or deleted through tool calls. Scenarios frequently require modifying existing entities rather than merely creating new ones, and evaluation compares initial and final environment snapshots against scenario-specific ground truth (Ma et al., 13 Jul 2026). This makes the benchmark sensitive to side effects, reversibility, and adaptation under revised user goals.

Images are first-class objects in the environment. They are stored in an image database with stable identifiers; users can attach them through send_message_with_image, and agents can also produce images through visualization tools (Ma et al., 13 Jul 2026). Images therefore persist beyond a single prompt and can be referenced audibly and operationally. The framework supports scenarios involving 3–6 images, delivered upfront, progressively, late, or in mixed patterns (Ma et al., 13 Jul 2026).

The framework supports two execution modes. In Code-Execution Mode, the model emits fenced Python code blocks, which are executed in a persistent sandboxed Python interpreter with tools available as functions. This mode supports loops, conditionals, and flexible orchestration, while employing static code checks, runtime sandboxing, and resource limits (Ma et al., 13 Jul 2026). In Structured Tool-Use Mode, the model interacts through provider-style function-calling schemas, producing either natural-language replies or structured tool calls; a special execute_code tool permits code execution within a structured API paradigm (Ma et al., 13 Jul 2026). The coexistence of these modes allows comparison between code-as-action and direct tool invocation.

3. Scenario design and generation pipeline

MM-ToolSandBox organizes benchmark construction along three orthogonal dimensions: information flow type, challenge type, and image arrival pattern (Ma et al., 13 Jul 2026). Information flow types specify how visual and non-visual information must be combined to drive actions. The benchmark defines seven such types: aggregate, compare, filter, compute, validate, lookup chain, and cross-reference. The authors explicitly relate these to database-style primitives such as UNION, ARGMAX, WHERE, derived columns, CHECK, correlated subqueries, and JOIN (Ma et al., 13 Jul 2026). This framing turns scenario design into a structured test of how visual evidence propagates through downstream tool calls.

Challenge types inject conversational and operational complications. Scenarios may involve no challenge, goal change, error correction, or state mutation (Ma et al., 13 Jul 2026). Goal change requires the agent to pivot and often clean up prior actions; error correction tests whether the agent can reconcile earlier user statements with later corrections and with visual evidence; state mutation emphasizes modification or deletion of existing entities. Image arrival patterns—upfront, progressive, late, and mixed—control when visual evidence becomes available relative to the interaction (Ma et al., 13 Jul 2026).

The scenario generation pipeline consists of six stages and uses Gemini 3.1 Pro and CLIP embeddings (Ma et al., 13 Jul 2026). It begins with 10,466 candidate images from eight VQA and vision datasets: DocVQA, OmniDocBench, HierText, WorldVQA, ChartQA-Pro, ChartMuseum, InfographicVQA, and ScreenSpotPro (Ma et al., 13 Jul 2026). An LLM first determines whether an image contains actionable content requiring both visual extraction and tool use, and maps it to relevant domains and tools. This stage retains 6,428 actionable images, or 61.4% of candidates (Ma et al., 13 Jul 2026).

Images are then clustered by associated domains into sets of 3–6 images using CLIP embedding similarity. For each cluster, the feasibility of each information-flow type and image-arrival pattern is scored from 0 to 3, and only combinations with score at least 2 are retained (Ma et al., 13 Jul 2026). Outline generation samples from viable flow types and arrival patterns, decides the number of user rounds, assigns a challenge type, and specifies tools and pre-existing entities. Scenario generation expands this outline into a full multi-turn script with deictic image references, timing of new images and corrections, a fully resolved goal, and preexisting entity requirements (Ma et al., 13 Jul 2026). Entity generation then instantiates initial and final world states.

Quality control is multi-stage. An LLM critic checks each scenario against seven criteria: multimodal necessity, visual realism, self-containment, step integrity, goal flexibility, visual grounding accuracy, and scenario plausibility (Ma et al., 13 Jul 2026). Scenarios that pass are downsampled for diversity, then subjected to an oracle solvability check and human expert review. The oracle agent, identified as Claude 4.5 Opus with access to ground-truth tools and expected state, must reach Entity F1 at least 0.9 and pass the LLM judge (Ma et al., 13 Jul 2026). The filtering funnel proceeds from 10,466 images to 6,428 actionable images, 1,152 raw scenarios, 599 scenarios passing the LLM critic, 300 after downsampling, and finally 258 human-verified scenarios containing 1,284 unique images (Ma et al., 13 Jul 2026).

The benchmark also includes a UI-interaction subset of 50 scenarios. These preserve the underlying task and entity-diff specifications but require the agent to design UI screens using an A2UI-style render_ui_screen, after which the environment renders the UI and the user interacts through ui_user_interact (Ma et al., 13 Jul 2026). Decision criteria remain private to the user, so the agent must present options and wait for user choices rather than optimize unilaterally. This subset introduces an additional UI judge and extends the framework beyond API-like action selection.

4. Evaluation protocol and metrics

MM-ToolSandBox uses a fixed evaluation harness. The agent receives user messages with attached images, as well as tool results and environment messages, and must perceive images directly, use tool discovery when needed, query and mutate entities, and conduct multi-turn dialogue (Ma et al., 13 Jul 2026). Scenarios are limited to a maximum of 100 steps counting both user and agent turns, and wall-clock time limits are enforced (Ma et al., 13 Jul 2026). The common harness uses the same agent system prompt, the same GPT-5.4 user simulator, and the same Claude 4.5 Sonnet rubric judge for all compared models (Ma et al., 13 Jul 2026).

The primary metric is Agent Success Rate, defined as the fraction of scenarios that pass all five LLM-judge criteria: task_completion, instruction_following, tool_use_validity, no_side_effects, and information_accuracy (Ma et al., 13 Jul 2026). The judge observes the full conversation, all tool calls and results, and all images, and it explicitly trusts environment outputs rather than the agent’s textual claims (Ma et al., 13 Jul 2026). This design makes final success depend simultaneously on task execution, faithfulness to instructions, correct tool usage, absence of extraneous modifications, and visual grounding accuracy.

A graded metric, Entity F1, compares actual entity changes with expected changes by Hungarian matching using type-aware column similarities (Ma et al., 13 Jul 2026). For identifiers and numerics, similarity requires exact match; for text fields it uses ROUGE-L; for timestamps it uses a datetime-aware check; per-entity similarity is the mean across columns (Ma et al., 13 Jul 2026). Precision is the fraction of actual changes that match expected ones, recall is the fraction of expected changes realized, and Entity F1 is computed as

$$\text{Entity F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

If the agent modifies tables that should remain unchanged, Entity F1 is set to 0 (Ma et al., 13 Jul 2026). This guardrail makes the metric sensitive not only to omissions but also to unintended side effects.

Two auxiliary metrics are also reported. User Success Rate evaluates the user simulator according to request_fidelity, conversational_naturalness, grounded_consistency, and tool_channel_correctness (Ma et al., 13 Jul 2026). Average Steps measures the mean number of interaction steps per scenario and serves as a proxy for efficiency and retry behavior (Ma et al., 13 Jul 2026). Partial success that does not satisfy the full rubric still appears in Entity F1, enabling separation between total failure and partially correct state modification.

The judge itself was evaluated against human annotations on a 100-scenario sample. The reported human–LLM agreement for the agent judge is 88.0%, with Cohen’s $\kappa = 0.748$, false positive rate 0.083, and false negative rate 0.175 (Ma et al., 13 Jul 2026). The user judge obtains 77.0% agreement and $\kappa = 0.142$, which the authors associate with class imbalance because user failures are rare (Ma et al., 13 Jul 2026). A plausible implication is that Agent Success Rate should be interpreted as a robust but not perfectly bias-free summary, especially in comparisons involving model families related to the judge or generator.

5. Empirical results and benchmark findings

The benchmark evaluates 12 multimodal models, including the Qwen 3.5 family, GLM 4.6V, KIMI 2.6, GPT-5.4 variants, Gemini 3.0 Flash, Gemini 3.1 Pro, Claude 4.5 Sonnet, and Claude 4.5 Opus (Ma et al., 13 Jul 2026). All are tested in the same default setting: Code-Execution mode, the full 511-tool registry, a maximum of 100 steps per scenario, the same system prompt, native thinking mode when available, the same GPT-5.4 user simulator, and the same Claude 4.5 Sonnet judge (Ma et al., 13 Jul 2026).

The principal numerical finding is that visual tool calling remains unsolved at this scale. The best model, Claude 4.5 Opus, achieves Agent SR 0.488 and Entity F1 0.848 over 40.8 average steps; Gemini 3.1 Pro follows with Agent SR 0.481, Entity F1 0.847, and 42.6 steps (Ma et al., 13 Jul 2026). KIMI 2.6 reaches Agent SR 0.415 and Entity F1 0.817, while GPT-5.4 thinking-high achieves Agent SR 0.415 with the highest Entity F1 at 0.865 and 36.1 steps (Ma et al., 13 Jul 2026). Claude 4.5 Sonnet records Agent SR 0.337 and Entity F1 0.817, and the best Qwen model, Qwen 3.5-397B-A17B, records Agent SR 0.353 and Entity F1 0.793 (Ma et al., 13 Jul 2026). Smaller models perform considerably worse, with Qwen 3.5-4B at Agent SR 0.116 and Qwen 3.5-9B at 0.190 (Ma et al., 13 Jul 2026).

Several patterns emerge from these results. First, all top-performing systems remain below 50% Agent Success Rate (Ma et al., 13 Jul 2026). Second, Entity F1 is generally higher than Agent SR, implying that many failures involve partially correct state updates that nevertheless violate some aspect of the full rubric, such as missing a requirement or introducing side effects (Ma et al., 13 Jul 2026). Third, additional reasoning budget helps at least one frontier model family: GPT-5.4 improves from Agent SR 0.291 in no-thinking mode to 0.361 in thinking-medium and 0.415 in thinking-high, while Entity F1 rises from 0.794 to 0.865 (Ma et al., 13 Jul 2026).

The UI subset is harder still. The paper notes Agent SR at or below 0.26 in UI mode (Ma et al., 13 Jul 2026). This suggests that tool calling in visually grounded environments becomes even more difficult when agents must also synthesize interfaces and defer choice to users rather than directly execute optimized actions.

Relative to M3-Bench, which reports persistent gaps in argument fidelity and structure consistency for multimodal MCP tool use (Zhou et al., 21 Nov 2025), MM-ToolSandBox identifies a different but complementary bottleneck: strong models frequently plan the right workflow but fail on precise visual extraction (Ma et al., 13 Jul 2026). A plausible synthesis is that multimodal tool-use systems remain limited at both levels—structural control over workflows and low-level grounding precision—depending on the environment and success criterion.

6. Failure modes, scaling behavior, and relation to adjacent frameworks

MM-ToolSandBox separates failures into four categories derived from the LLM judge’s rubric: factual error, task failure, incomplete execution, and uncontrolled behavior (Ma et al., 13 Jul 2026). Factual error denotes cases where the workflow is correct but information accuracy fails, typically because the agent misreads an image or extracts an incorrect value. Task failure refers to planning or execution failures such as never calling required tools, misordering operations, or looping. Incomplete execution captures cases where the core task is done and facts are correct but some instructions are missed. Uncontrolled behavior marks failures of the no_side_effects criterion through extra entities or stray actions (Ma et al., 13 Jul 2026).

For strong models, factual error dominates. Across Claude 4.5 Opus, GPT-5.4, KIMI, and Qwen 3.5-397B, approximately 50–53% of failures are factual errors (Ma et al., 13 Jul 2026). For Claude 4.5 Opus specifically, 70 of 132 failures, or 53.0%, are factual errors; 31, or 23.5%, are task failures; 19, or 14.4%, are incomplete execution; and 12, or 9.1%, are uncontrolled behavior (Ma et al., 13 Jul 2026). The paper further reports that among the 70 factual-error cases for Opus, dominant subtypes are object misidentification with 29 cases and fine-grained text extraction errors with 16 cases (Ma et al., 13 Jul 2026).

The benchmark provides a concrete illustration of correct planning with incorrect perception. In one reported example, the task is to create a note from a UI screenshot containing an IP core name and an active synthesis part. The agent verifies the project directory and successfully calls simple_note_create_note, but transcribes the synthesis part as xc7z0p-ffvd970-1-e instead of the ground-truth xcku3p-ffva676-1-e, differing in four characters (Ma et al., 13 Jul 2026). The plan is fully correct; the failure lies in high-precision visual transcription.

The converse failure mode also occurs. In another example involving multiple rounds, the agent correctly identifies baby or children’s products from images but never invokes the tools required to add them to cart or transfer them to a wish list; it later sends an email claiming success despite the database showing no corresponding state changes (Ma et al., 13 Jul 2026). Here perception is adequate but planning and execution fail.

Scaling changes the dominant failure mode. Figure 1, as summarized in the provided details, indicates a planning-to-precision crossover: in Qwen 3.5-4B, task failures constitute 51% of failures, whereas in Qwen 3.5-27B and 397B they drop to roughly 30% while factual errors rise to about 51%; Claude 4.5 Opus reaches 53% factual errors and 23% task failures (Ma et al., 13 Jul 2026). This suggests that as models scale, high-level planning improves faster than visual precision. Smaller models need better scaffolding for tool orchestration; larger models need better visual encoders, OCR, and verification mechanisms.

Multi-image working memory is another major source of difficulty. The benchmark reports that upfront image arrival is significantly harder than progressive or late arrival, with success rate approximately 20 percentage points lower when all images are sent at the beginning (Ma et al., 13 Jul 2026). Performance declines as the number of images increases, and information-flow types requiring multi-image synthesis, such as filter and aggregate, are almost twice as hard as single-image lookup-chain tasks (Ma et al., 13 Jul 2026). This suggests that the challenge is not only extracting facts from individual images but maintaining and selectively retrieving them across conversation history.

In relation to broader tool-sandbox research, MM-ToolSandBox occupies a different axis from MCP-SandboxScan. The latter is a WASM/WASI-based dynamic analysis framework for securely executing untrusted MCP tools and auditing external-to-sink exposures involving prompts, JSON payloads, environment variables, mounted files, and HTTP fetch intents (Tan et al., 3 Jan 2026). MM-ToolSandBox is not presented as a security scanner; its concern is capability evaluation in a simulated, stateful environment. Yet both frameworks rely on sandboxing and auditable execution. This suggests a possible synthesis in future systems: capability benchmarks like MM-ToolSandBox could be complemented by runtime security analysis layers analogous to MCP-SandboxScan for tool trustworthiness and provenance-sensitive auditing.

ToolMATH provides another useful comparison. It studies long-horizon reasoning in large tool catalogs with controlled redundancy and missing-capability regimes, concluding that failures are driven less by local tool selection than by long-range plan coherence and disciplined use of observations (Choi et al., 24 Feb 2026). MM-ToolSandBox complements this by showing that once planning reaches a sufficient level, visual precision becomes the main bottleneck (Ma et al., 13 Jul 2026). Together these results imply that multimodal tool-agent evaluation must distinguish at least three separable competencies: long-range control, precise grounding, and safe execution.

7. Significance, limitations, and research implications

MM-ToolSandBox is notable for unifying several properties that were previously fragmented across benchmarks: large tool suites, stateful multi-domain environments, multi-turn and multi-image dialogue, realistic user behaviors, code and structured-tool interfaces, deterministic state-based checking, and detailed failure analysis (Ma et al., 13 Jul 2026). Within the landscape of tool-use evaluation, it differs from text-only benchmarks such as API-Bank, BFCL, ToolSandbox, and Gaia2, and from GUI-agent settings such as VisualWebArena, OSWorld, and MobileWorld, by centering on API-centric tool orchestration grounded in visual evidence and persistent application state (Ma et al., 13 Jul 2026).

Its practical value lies in diagnostic granularity. Because success is tied to environment outputs and entity diffs rather than only free-form answer strings, the framework can distinguish correct plans with wrong extracted values from wrong plans with adequate visual interpretation. The planning-to-precision crossover reported across model scales makes this particularly important: future work on smaller models may benefit most from stronger scaffolding, tool discovery, and workflow decomposition, whereas stronger models may require improvements in OCR, document understanding, high-resolution vision, and explicit re-verification before committing state changes (Ma et al., 13 Jul 2026).

Several limitations are also explicit. The primary metric uses an LLM judge, with 88% agreement to humans, so some bias remains possible (Ma et al., 13 Jul 2026). Oracle solvability uses Claude 4.5 Opus, scenario generation uses Gemini 3.1 Pro, and the harness is fixed rather than tuned per model; these choices may introduce family-specific advantages or disadvantages (Ma et al., 13 Jul 2026). UI mode remains a relatively small subset of 50 scenarios (Ma et al., 13 Jul 2026). The benchmark therefore offers strong standardization but not complete neutrality with respect to all modeling choices.

A plausible implication is that MM-ToolSandBox functions best as a controlled comparative instrument rather than an absolute measure of general multimodal agency. Its greatest strength is the ability to expose where a system fails within a realistic but auditable environment. In that respect it aligns with the broader movement represented by M3-Bench, ToolMATH, and MCP-SandboxScan: the study of tool-using agents is shifting from simple end-answer correctness toward richer analyses of workflows, state transitions, grounding, and runtime behavior (Zhou et al., 21 Nov 2025, Choi et al., 24 Feb 2026, Tan et al., 3 Jan 2026).

The framework and benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox (Ma et al., 13 Jul 2026). As reported, they include the environment runtime, tool definitions and schemas, scenario definitions with images and annotations, evaluation scripts, and judge prompts (Ma et al., 13 Jul 2026). This availability, combined with the benchmark’s multi-domain stateful design, positions MM-ToolSandBox as a reference platform for studying visual tool-calling agents under realistic operational constraints.