Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use

Abstract: Multimodal LLMs (MLLMs) excel at utilizing digital APIs and increasingly serve as the "brain" of embodied AI, instructing robots to interact with the physical world. In such embodied settings, a central capability is the use of physical tools, which underpins MLLMs' ability to assist humans in real-world tasks. Despite the importance, MLLMs' proficiency in physical tool use remains largely unexplored. To address this gap, we introduce PhysTool-Bench, the first physical tool-use benchmark designed to evaluate MLLMs' ability to comprehend real-world scenarios, identify physical tools, and plan their use. PhysTool-Bench comprises 2,510 queries over 2,678 real-world physical tools spanning diverse domains, including manufacturing, electrical work, agriculture, and healthcare. Concretely, models are evaluated along two primary dimensions: 1) recognizing all physical tools present in the scene, and 2) planning the tool selection and use sequence based on the instruction and visual context. Across 13 leading MLLMs, even the strongest model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes merely 21.0% of queries end-to-end. Our analysis reveals a two-level deficit: MLLMs struggle to perceive tools in realistic scenes, and the much larger drop at the planning stage further indicates a lack of functional commonsense for mapping perceived tools onto task semantics, pinpointing a critical bottleneck for the development of practical embodied AI.

• The paper introduces PhysTool-Bench, a diagnostic evaluation revealing significant cognitive bottlenecks in MLLMs when handling physical tool use.
• It employs a multi-stage pipeline with curated scenarios, distractor injection, and sequential planning, measured by precise metrics such as EM and F1.
• Empirical results show that while MLLMs excel in digital tasks, they struggle with functional commonsense in real-world tool recognition and planning.

Probing MLLMs for Physical Tool Use: PhysTool-Bench and a Cognitive Bottleneck

Motivation and Gap Analysis

The capability gap between digital and physical tool use by Multimodal LLMs (MLLMs) is explicitly defined in the paper "Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use" (2606.10803). While MLLMs reliably manipulate APIs for structured digital tasks, their performance degrades markedly when confronted with physically situated tool use requiring visual reasoning and commonsense mapping in realistic scenes.

Figure 1: The capability divide between digital and physical tool use; MLLMs excel at structured digital tasks via APIs, but fail at visual and physical tool reasoning in real-world contexts.

The paper introduces PhysTool-Bench, a diagnostic evaluation suite specifically targeting physical tool use. The benchmark comprises 2,510 queries spanning 2,678 real-world tools sampled across diverse domains (manufacturing, healthcare, agriculture, electrical work), operationalized along two axes: tool recognition and tool selection with sequential action planning. This approach directly tests the transition from theoretical competence in multimodal reasoning to practical performance as the cognitive engine of embodied AI.

PhysTool-Bench: Construction, Definition, and Quality Assurance

PhysTool-Bench systematically samples tool-use scenarios by integrating a multi-stage pipeline: curated tool bank, task and distractor generation, synthetic scene construction, and rigorous quality control.

Figure 2: Overview of PhysTool-Bench construction: automatic query generation, distractor injection, scene rendering, and continuous refinement with three quality-control stages.

Each query presents a realistic scene image and an instruction requiring models to enumerate all tools (Task I), and to select and sequence the necessary subset (Task II). Distractors are carefully chosen to maximize functional and visual confusion, and step-labeling enforces sequential planning requirements. The dataset spans 57 UNSPSC segments, averaging over eight tools per scene, with a large fraction (86.9%) requiring strict execution order.

Quality control (QC) encompasses: (i) minimal tool set analysis via professional standards, (ii) literal alignment between image descriptions and tool lists, and (iii) human validation to enforce physical realism and prevent artificial cues. This pipeline guarantees high-fidelity data for isolation of cognitive deficits.

Evaluation Framework and Metrics

The study evaluates 13 commercial and open-source MLLMs, including Gemini-3.1-Pro, GPT-4o, GPT-5.2, Qwen3-VL-Plus, and leading open-weight architectures. Evaluations are zero-shot, using prompt templates that enforce scene analysis before tool generation. Metrics are stratified:

• Recognition (Task I): Precision, Recall, F1, against all visible tools.
• Order-Aware Planning (Task II): Exact Match (EM—strict), Task-Completable Rate (TCR—relaxed, allowing extra tools), Success Rate @ $k$ (SR@$k$—prefix performance).
• Order-Agnostic Selection: F1 against ground-truth targets, irrespective of order.

These metrics enable fine-grained analysis separating perception from functional reasoning.

Empirical Results and Cognitive Bottlenecks

Key empirical findings:

• Recognition Deficit: No model exceeds 63% F1 for tool recognition; leading models miss a significant fraction of visible tools.
• Planning Collapse: The best EM on Task II is 20.96% (Gemini-3.1-Pro), with multi-step queries ($k \geq 6$) near zero.
• Instruction Conditioning: Task II does not uniformly boost recognition; only Gemini-3.1-Pro and OVis benefit, implying added reasoning complexity.

The capability degradation with increased planning horizon is sharply illustrated:

Figure 3: Exact-match and prefix success rates for Qwen3-VL-32B-Thinking across target tool count; EM collapses at increasing complexity, exposing difficulty in multi-step planning.

Performance stratified across tool domains further reveals intrinsic category-level difficulty, not attributable to model specifics. Healthcare and Office scenarios—well-structured procedures and distinctive tools—yield better scores, while Manufacturing and Electrical domains, dominated by functionally similar candidates and strict sequencing, remain unsolved.

Figure 4: Task-Completable Rate (TCR) across UNSPSC segments shows consistent model rankings and steep drop in complex, visually overlapping categories.

Error Analysis and Functional Commonsense

A granular decomposition of Task II predictions isolates failure modes:

• Substitution Dominance: Replacing correct tools with functionally similar distractors is the largest error mode.
• Root Cause: Only 22% of missed targets are visual; 41.3% are functional omissions, where models see the tool but fail to map its relevance. 60% of spurious selections are real distractors, further underscoring the functional confusion.
• Ordering Errors: Out-of-order rates are below random baseline, suggesting some sequencing competence, but half of these stem from instruction misinterpretation.

  Figure 5: Failure decomposition across seven MLLMs; Substitute errors dominate, with task-blocking mistakes far more prevalent than task-completable deviations.

Closed-source models outperform open-weight alternatives on most metrics, but the gap narrows in planning-style subtasks, where reasoning-centric architectures show competitive order-aware performance.

Implications and Future Research Directions

The findings establish a two-level cognitive bottleneck in MLLMs: perception is non-trivial but not the limiting factor; functional commonsense, required to link tool affordances with task semantics, is the primary deficit. This diagnosis is corroborated by comparison with specialized object detectors (Grounding DINO, 70.53% recall) and human annotator baselines (75% EM for familiar queries), demonstrating that visual evidence and ground-truth align with informed judgment, and MLLM failures are cognitive, not perceptual.

Closing the gap requires research into explicit grounding of multimodal models in physical reasoning, especially in long-tail domains with high functional overlap and complex tool use patterns. Merely scaling visual encoders will not suffice; integrating structured workflow models, domain-specific knowledge graphs, and hierarchical planning protocols emerges as an urgent direction for embodied AI.

Conclusion

PhysTool-Bench offers a rigorous diagnostic for physical tool use in MLLMs, revealing a distinct capability gap between digital and physical reasoning. The bottleneck lies in functional commonsense, with most failures arising from substituting correct tools for functionally similar distractors. Future progress hinges on deepening the cognitive foundations of multimodal models for grounded action planning.