---
title: MM-ToolSandBox Benchmark
url: https://www.emergentmind.com/topics/mm-toolsandbox
type: topic
---

# MM-ToolSandBox Benchmark

Searching arXiv for MM-ToolSandBox and closely related benchmarks to ground the article in the cited literature.
MM-ToolSandBox is a unified framework and benchmark for evaluating visually grounded tool-calling agents in a stateful environment. It targets multimodal agents that must interpret images, select and invoke tools, update persistent world state, and sustain coherent behavior across multi-turn interactions. The framework extends earlier text-only ToolSandBox-style evaluation toward visual inputs, multi-image reasoning, and mutable application environments, while situating itself alongside related efforts on multimodal multi-hop tool-use benchmarking and MCP-oriented tool execution analysis [2607.11818, 2511.17729, 2601.01241].

## 1. Definition and scope

MM-ToolSandBox is designed for agents that solve tasks by calling tools in response to visual and textual inputs. In the formulation reported for the benchmark, a visually grounded tool-calling agent must see images such as photos, documents, charts, and UI screenshots; extract task-relevant information from them; decide which tools to call in a large tool space; execute those tools against a mutable environment state; and continue doing so across multiple turns as new instructions and images arrive [2607.11818]. The framework therefore evaluates an end-to-end pipeline of perception, reasoning and planning, tool invocation, state updates, and dialogue management.

The benchmark addresses gaps in prior tool-use evaluation settings that were mostly text-only, often single-turn, and frequently stateless or restricted to relatively small tool suites. In contrast, MM-ToolSandBox provides a stateful execution environment with 511 tools across 16 application domains, supports multi-turn and multi-image conversations, and incorporates realistic user behavior including goal changes, user errors, and state mutations [2607.11818]. This emphasis on statefulness and conversational evolution distinguishes it from benchmarks centered on isolated API sequences or static task specifications.

A useful contextual comparison is M3-Bench, which evaluates multimodal tool use under the Model Context Protocol with an emphasis on multi-hop, multi-threaded workflows, cross-tool dependencies, and persistence of intermediate artifacts [2511.17729]. MM-ToolSandBox shares the commitment to visually grounded, multi-step tool use, but its primary abstraction is a stateful simulated application ecosystem rather than MCP server trajectories. This suggests that MM-ToolSandBox focuses more strongly on environment mutation, multi-turn dialogue, and entity-level correctness, whereas M3-Bench foregrounds tool-call trajectory structure.

## 2. Environment architecture and execution model

At the core of MM-ToolSandBox is a stateful execution environment in which each evaluation episode is a scenario specifying initial world state, expected final world state, available tools, user instructions and their evolution across rounds, associated images, and completion criteria [2607.11818]. Interaction is tri-partite: the agent plans and calls tools, a user simulator provides instructions and corrections, and the environment executes calls while maintaining persistent state.

The environment simulates 16 mobile or service application domains. The tool suite comprises 511 native Python tools across these domains, including search tools, UI tools for interactive UI mode, vision tools for image manipulation and rendering, and system-level utilities such as file system, contacts, and messaging functionality [2607.11818]. Tool metadata includes typed signatures as well as role visibility and state dependencies. Because the full tool list is too large for a single prompt, the framework exposes a tool discovery meta-tool, `search_tool` or `api_docs_search_api_docs`, and maintains an LRU-bounded active tool set [2607.11818]. This design turns tool discovery itself into part of the agentic problem.

Statefulness is not incidental but structural. Persistent entities such as contacts, calendar events, cart entries, notes, tasks, and orders can be created, updated, or deleted through tool calls. Scenarios frequently require modifying existing entities rather than merely creating new ones, and evaluation compares initial and final environment snapshots against scenario-specific ground truth [2607.11818]. This makes the benchmark sensitive to side effects, reversibility, and adaptation under revised user goals.

Images are first-class objects in the environment. They are stored in an image database with stable identifiers; users can attach them through `send_message_with_image`, and agents can also produce images through visualization tools [2607.11818]. Images therefore persist beyond a single prompt and can be referenced audibly and operationally. The framework supports scenarios involving 3–6 images, delivered upfront, progressively, late, or in mixed patterns [2607.11818].

The framework supports two execution modes. In Code-Execution Mode, the model emits fenced Python code blocks, which are executed in a persistent sandboxed Python interpreter with tools available as functions. This mode supports loops, conditionals, and flexible orchestration, while employing static code checks, runtime sandboxing, and resource limits [2607.11818]. In Structured Tool-Use Mode, the model interacts through provider-style function-calling schemas, producing either natural-language replies or structured tool calls; a special `execute_code` tool permits code execution within a structured API paradigm [2607.11818]. The coexistence of these modes allows comparison between code-as-action and direct tool invocation.

## 3. Scenario design and generation pipeline

MM-ToolSandBox organizes benchmark construction along three orthogonal dimensions: information flow type, challenge type, and image arrival pattern [2607.11818]. Information flow types specify how visual and non-visual information must be combined to drive actions. The benchmark defines seven such types: aggregate, compare, filter, compute, validate, lookup chain, and cross-reference. The authors explicitly relate these to database-style primitives such as `UNION`, `ARGMAX`, `WHERE`, derived columns, `CHECK`, correlated subqueries, and `JOIN` [2607.11818]. This framing turns scenario design into a structured test of how visual evidence propagates through downstream tool calls.

Challenge types inject conversational and operational complications. Scenarios may involve no challenge, goal change, error correction, or state mutation [2607.11818]. Goal change requires the agent to pivot and often clean up prior actions; error correction tests whether the agent can reconcile earlier user statements with later corrections and with visual evidence; state mutation emphasizes modification or deletion of existing entities. Image arrival patterns—upfront, progressive, late, and mixed—control when visual evidence becomes available relative to the interaction [2607.11818].

The scenario generation pipeline consists of six stages and uses Gemini 3.1 Pro and CLIP embeddings [2607.11818]. It begins with 10,466 candidate images from eight VQA and vision datasets: DocVQA, OmniDocBench, HierText, WorldVQA, ChartQA-Pro, ChartMuseum, InfographicVQA, and ScreenSpotPro [2607.11818]. An LLM first determines whether an image contains actionable content requiring both visual extraction and tool use, and maps it to relevant domains and tools. This stage retains 6,428 actionable images, or 61.4% of candidates [2607.11818].

Images are then clustered by associated domains into sets of 3–6 images using CLIP embedding similarity. For each cluster, the feasibility of each information-flow type and image-arrival pattern is scored from 0 to 3, and only combinations with score at least 2 are retained [2607.11818]. Outline generation samples from viable flow types and arrival patterns, decides the number of user rounds, assigns a challenge type, and specifies tools and pre-existing entities. Scenario generation expands this outline into a full multi-turn script with deictic image references, timing of new images and corrections, a fully resolved goal, and preexisting entity requirements [2607.11818]. Entity generation then instantiates initial and final world states.

Quality control is multi-stage. An LLM critic checks each scenario against seven criteria: multimodal necessity, visual realism, self-containment, step integrity, goal flexibility, visual grounding accuracy, and scenario plausibility [2607.11818]. Scenarios that pass are downsampled for diversity, then subjected to an oracle solvability check and human expert review. The oracle agent, identified as Claude 4.5 Opus with access to ground-truth tools and expected state, must reach Entity F1 at least 0.9 and pass the LLM judge [2607.11818]. The filtering funnel proceeds from 10,466 images to 6,428 actionable images, 1,152 raw scenarios, 599 scenarios passing the LLM critic, 300 after downsampling, and finally 258 human-verified scenarios containing 1,284 unique images [2607.11818].

The benchmark also includes a UI-interaction subset of 50 scenarios. These preserve the underlying task and entity-diff specifications but require the agent to design UI screens using an A2UI-style `render_ui_screen`, after which the environment renders the UI and the user interacts through `ui_user_interact` [2607.11818]. Decision criteria remain private to the user, so the agent must present options and wait for user choices rather than optimize unilaterally. This subset introduces an additional UI judge and extends the framework beyond API-like action selection.

## 4. Evaluation protocol and metrics

MM-ToolSandBox uses a fixed evaluation harness. The agent receives user messages with attached images, as well as tool results and environment messages, and must perceive images directly, use tool discovery when needed, query and mutate entities, and conduct multi-turn dialogue [2607.11818]. Scenarios are limited to a maximum of 100 steps counting both user and agent turns, and wall-clock time limits are enforced [2607.11818]. The common harness uses the same agent system prompt, the same GPT-5.4 user simulator, and the same Claude 4.5 Sonnet rubric judge for all compared models [2607.11818].

The primary metric is Agent Success Rate, defined as the fraction of scenarios that pass all five LLM-judge criteria: task_completion, instruction_following, tool_use_validity, no_side_effects, and information_accuracy [2607.11818]. The judge observes the full conversation, all tool calls and results, and all images, and it explicitly trusts environment outputs rather than the agent’s textual claims [2607.11818]. This design makes final success depend simultaneously on task execution, faithfulness to instructions, correct tool usage, absence of extraneous modifications, and visual grounding accuracy.

A graded metric, Entity F1, compares actual entity changes with expected changes by Hungarian matching using type-aware column similarities [2607.11818]. For identifiers and numerics, similarity requires exact match; for text fields it uses ROUGE-L; for timestamps it uses a datetime-aware check; per-entity similarity is the mean across columns [2607.11818]. Precision is the fraction of actual changes that match expected ones, recall is the fraction of expected changes realized, and Entity F1 is computed as

$$
\text{Entity F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
$$

If the agent modifies tables that should remain unchanged, Entity F1 is set to 0 [2607.11818]. This guardrail makes the metric sensitive not only to omissions but also to unintended side effects.

Two auxiliary metrics are also reported. User Success Rate evaluates the user simulator according to request_fidelity, conversational_naturalness, grounded_consistency, and tool_channel_correctness [2607.11818]. Average Steps measures the mean number of interaction steps per scenario and serves as a proxy for efficiency and retry behavior [2607.11818]. Partial success that does not satisfy the full rubric still appears in Entity F1, enabling separation between total failure and partially correct state modification.

The judge itself was evaluated against human annotations on a 100-scenario sample. The reported human–LLM agreement for the agent judge is 88.0%, with Cohen’s $\kappa = 0.748$, false positive rate 0.083, and false negative rate 0.175 [2607.11818]. The user judge obtains 77.0% agreement and $\kappa = 0.142$, which the authors associate with class imbalance because user failures are rare [2607.11818]. A plausible implication is that Agent Success Rate should be interpreted as a robust but not perfectly bias-free summary, especially in comparisons involving model families related to the judge or generator.

## 5. Empirical results and benchmark findings

The benchmark evaluates 12 multimodal models, including the Qwen 3.5 family, GLM 4.6V, KIMI 2.6, GPT-5.4 variants, Gemini 3.0 Flash, Gemini 3.1 Pro, Claude 4.5 Sonnet, and Claude 4.5 Opus [2607.11818]. All are tested in the same default setting: Code-Execution mode, the full 511-tool registry, a maximum of 100 steps per scenario, the same system prompt, native thinking mode when available, the same GPT-5.4 user simulator, and the same Claude 4.5 Sonnet judge [2607.11818].

The principal numerical finding is that visual tool calling remains unsolved at this scale. The best model, Claude 4.5 Opus, achieves Agent SR 0.488 and Entity F1 0.848 over 40.8 average steps; Gemini 3.1 Pro follows with Agent SR 0.481, Entity F1 0.847, and 42.6 steps [2607.11818]. KIMI 2.6 reaches Agent SR 0.415 and Entity F1 0.817, while GPT-5.4 thinking-high achieves Agent SR 0.415 with the highest Entity F1 at 0.865 and 36.1 steps [2607.11818]. Claude 4.5 Sonnet records Agent SR 0.337 and Entity F1 0.817, and the best Qwen model, Qwen 3.5-397B-A17B, records Agent SR 0.353 and Entity F1 0.793 [2607.11818]. Smaller models perform considerably worse, with Qwen 3.5-4B at Agent SR 0.116 and Qwen 3.5-9B at 0.190 [2607.11818].

Several patterns emerge from these results. First, all top-performing systems remain below 50% Agent Success Rate [2607.11818]. Second, Entity F1 is generally higher than Agent SR, implying that many failures involve partially correct state updates that nevertheless violate some aspect of the full rubric, such as missing a requirement or introducing side effects [2607.11818]. Third, additional reasoning budget helps at least one frontier model family: GPT-5.4 improves from Agent SR 0.291 in no-thinking mode to 0.361 in thinking-medium and 0.415 in thinking-high, while Entity F1 rises from 0.794 to 0.865 [2607.11818].

The UI subset is harder still. The paper notes Agent SR at or below 0.26 in UI mode [2607.11818]. This suggests that tool calling in visually grounded environments becomes even more difficult when agents must also synthesize interfaces and defer choice to users rather than directly execute optimized actions.

Relative to M3-Bench, which reports persistent gaps in argument fidelity and structure consistency for multimodal MCP tool use [2511.17729], MM-ToolSandBox identifies a different but complementary bottleneck: strong models frequently plan the right workflow but fail on precise visual extraction [2607.11818]. A plausible synthesis is that multimodal tool-use systems remain limited at both levels—structural control over workflows and low-level grounding precision—depending on the environment and success criterion.

## 6. Failure modes, scaling behavior, and relation to adjacent frameworks

MM-ToolSandBox separates failures into four categories derived from the LLM judge’s rubric: factual error, task failure, incomplete execution, and uncontrolled behavior [2607.11818]. Factual error denotes cases where the workflow is correct but information accuracy fails, typically because the agent misreads an image or extracts an incorrect value. Task failure refers to planning or execution failures such as never calling required tools, misordering operations, or looping. Incomplete execution captures cases where the core task is done and facts are correct but some instructions are missed. Uncontrolled behavior marks failures of the no_side_effects criterion through extra entities or stray actions [2607.11818].

For strong models, factual error dominates. Across Claude 4.5 Opus, GPT-5.4, KIMI, and Qwen 3.5-397B, approximately 50–53% of failures are factual errors [2607.11818]. For Claude 4.5 Opus specifically, 70 of 132 failures, or 53.0%, are factual errors; 31, or 23.5%, are task failures; 19, or 14.4%, are incomplete execution; and 12, or 9.1%, are uncontrolled behavior [2607.11818]. The paper further reports that among the 70 factual-error cases for Opus, dominant subtypes are object misidentification with 29 cases and fine-grained text extraction errors with 16 cases [2607.11818].

The benchmark provides a concrete illustration of correct planning with incorrect perception. In one reported example, the task is to create a note from a UI screenshot containing an IP core name and an active synthesis part. The agent verifies the project directory and successfully calls `simple_note_create_note`, but transcribes the synthesis part as `xc7z0p-ffvd970-1-e` instead of the ground-truth `xcku3p-ffva676-1-e`, differing in four characters [2607.11818]. The plan is fully correct; the failure lies in high-precision visual transcription.

The converse failure mode also occurs. In another example involving multiple rounds, the agent correctly identifies baby or children’s products from images but never invokes the tools required to add them to cart or transfer them to a wish list; it later sends an email claiming success despite the database showing no corresponding state changes [2607.11818]. Here perception is adequate but planning and execution fail.

Scaling changes the dominant failure mode. Figure 7, as summarized in the provided details, indicates a planning-to-precision crossover: in Qwen 3.5-4B, task failures constitute 51% of failures, whereas in Qwen 3.5-27B and 397B they drop to roughly 30% while factual errors rise to about 51%; Claude 4.5 Opus reaches 53% factual errors and 23% task failures [2607.11818]. This suggests that as models scale, high-level planning improves faster than visual precision. Smaller models need better scaffolding for tool orchestration; larger models need better visual encoders, OCR, and verification mechanisms.

Multi-image working memory is another major source of difficulty. The benchmark reports that upfront image arrival is significantly harder than progressive or late arrival, with success rate approximately 20 percentage points lower when all images are sent at the beginning [2607.11818]. Performance declines as the number of images increases, and information-flow types requiring multi-image synthesis, such as filter and aggregate, are almost twice as hard as single-image lookup-chain tasks [2607.11818]. This suggests that the challenge is not only extracting facts from individual images but maintaining and selectively retrieving them across conversation history.

In relation to broader tool-sandbox research, MM-ToolSandBox occupies a different axis from MCP-SandboxScan. The latter is a WASM/WASI-based dynamic analysis framework for securely executing untrusted MCP tools and auditing external-to-sink exposures involving prompts, JSON payloads, environment variables, mounted files, and HTTP fetch intents [2601.01241]. MM-ToolSandBox is not presented as a security scanner; its concern is capability evaluation in a simulated, stateful environment. Yet both frameworks rely on sandboxing and auditable execution. This suggests a possible synthesis in future systems: capability benchmarks like MM-ToolSandBox could be complemented by runtime security analysis layers analogous to MCP-SandboxScan for tool trustworthiness and provenance-sensitive auditing.

ToolMATH provides another useful comparison. It studies long-horizon reasoning in large tool catalogs with controlled redundancy and missing-capability regimes, concluding that failures are driven less by local tool selection than by long-range plan coherence and disciplined use of observations [2602.21265]. MM-ToolSandBox complements this by showing that once planning reaches a sufficient level, visual precision becomes the main bottleneck [2607.11818]. Together these results imply that multimodal tool-agent evaluation must distinguish at least three separable competencies: long-range control, precise grounding, and safe execution.

## 7. Significance, limitations, and research implications

MM-ToolSandBox is notable for unifying several properties that were previously fragmented across benchmarks: large tool suites, stateful multi-domain environments, multi-turn and multi-image dialogue, realistic user behaviors, code and structured-tool interfaces, deterministic state-based checking, and detailed failure analysis [2607.11818]. Within the landscape of tool-use evaluation, it differs from text-only benchmarks such as API-Bank, BFCL, ToolSandbox, and Gaia2, and from GUI-agent settings such as VisualWebArena, OSWorld, and MobileWorld, by centering on API-centric tool orchestration grounded in visual evidence and persistent application state [2607.11818].

Its practical value lies in diagnostic granularity. Because success is tied to environment outputs and entity diffs rather than only free-form answer strings, the framework can distinguish correct plans with wrong extracted values from wrong plans with adequate visual interpretation. The planning-to-precision crossover reported across model scales makes this particularly important: future work on smaller models may benefit most from stronger scaffolding, tool discovery, and workflow decomposition, whereas stronger models may require improvements in OCR, document understanding, high-resolution vision, and explicit re-verification before committing state changes [2607.11818].

Several limitations are also explicit. The primary metric uses an LLM judge, with 88% agreement to humans, so some bias remains possible [2607.11818]. Oracle solvability uses Claude 4.5 Opus, scenario generation uses Gemini 3.1 Pro, and the harness is fixed rather than tuned per model; these choices may introduce family-specific advantages or disadvantages [2607.11818]. UI mode remains a relatively small subset of 50 scenarios [2607.11818]. The benchmark therefore offers strong standardization but not complete neutrality with respect to all modeling choices.

A plausible implication is that MM-ToolSandBox functions best as a controlled comparative instrument rather than an absolute measure of general multimodal agency. Its greatest strength is the ability to expose where a system fails within a realistic but auditable environment. In that respect it aligns with the broader movement represented by M3-Bench, ToolMATH, and MCP-SandboxScan: the study of tool-using agents is shifting from simple end-answer correctness toward richer analyses of workflows, state transitions, grounding, and runtime behavior [2511.17729, 2602.21265, 2601.01241].

The framework and benchmark are publicly available at `https://github.com/apple/ml-mmtoolsandbox` [2607.11818]. As reported, they include the environment runtime, tool definitions and schemas, scenario definitions with images and annotations, evaluation scripts, and judge prompts [2607.11818]. This availability, combined with the benchmark’s multi-domain stateful design, positions MM-ToolSandBox as a reference platform for studying visual tool-calling agents under realistic operational constraints.

Source: https://www.emergentmind.com/topics/mm-toolsandbox