GeT-USE: Robotic Tool-Use Framework

• GeT-USE is a robotic framework that uses synthetic embodiment extensions to learn and generalize tool affordances for arbitrary objects.
• It employs a two-stage training process that combines simulation-based tool geometry discovery with depth-image-driven real-world deployment.
• Experimental results demonstrate significant improvements, achieving up to 61.7% overall success in challenging 6-DOF manipulation tasks.

GeT-USE

GeT-USE (Generalized Tool-Usage via Simulated Embodiment Extensions) is a robotic learning framework designed to enable bimanual mobile manipulators to flexibly identify, grasp, and utilize arbitrary objects as tools, even in the absence of ideal or predefined tools. The method integrates simulated embodiment extension, deep visuomotor learning, and geometric tool reasoning to achieve high generalization in physical tool-usage across vision-based tasks. GeT-USE is characterized by a two-stage training and transfer strategy: (1) learning in simulation by augmenting the robot’s embodiment with synthetic components to discover beneficial tool geometries, and (2) deploying learned policies to select, grasp, and manipulate real-world objects as tools using only depth-based perception and onboard sensing (Wu et al., 29 Oct 2025).

1. Problem Formulation and Motivation

Robust tool use in robotics entails the identification and use of arbitrary ambient objects as functional tools, especially in settings where the optimal tool is absent or unavailable, or multiple possibly-useful objects are present. This scenario represents a significant challenge for bimanual mobile manipulators for several reasons:

• There is no guarantee of seeing a prototype tool category in the candidate objects.
• Bimanual tasks (e.g., sweeping, hooking, decanting) demand precise 6-DOF control and coordinated movement between arms, mobile base, and grippers.
• The combinatorial space of (object, grasp, manipulation strategy) triples is intractably large for search-based or category-based approaches.

GeT-USE directly addresses this by reframing the problem: the robot learns which geometric properties make an object “tool-like” for a particular task and then distills this spatial reasoning into control and selection policies that generalize to unknown objects and unseen physical instances. This enables deployment to cluttered, real-world tablescapes typical of service robotics, without requiring the presence of optimal tools (Wu et al., 29 Oct 2025).

2. Simulated Embodiment Extensions and Tool Geometry Discovery

The core insight of GeT-USE is that simulated construction of “embodiment extensions”—temporary additions, built from voxelized blocks attached to the robot’s wrist(s)—can expose the robot to a diverse distribution of tool geometries. This process is formalized as learning in a Markov Decision Process (MDP):

$$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathbb{T}, \mathcal{R}, \gamma, \rho_0, \mathcal{H} \rangle$$

where $\mathcal{S}$ is the simulator state, $\mathcal{O}$ the observation (typically two depth images: task + wrist), and actions include the addition of a new block at a configurable relative position and size. A policy $\pi_{gtb}$ (generalized tool-building) incrementally constructs extensions; the constructed “tool” is then evaluated for task success via a privileged simulation-based manipulation, yielding sparse or shaped rewards.

Crucially, both successful and unsuccessful synthetic tools are collected, providing not only data on optimal geometries but also negative examples. This broadens the learning signal and enables downstream selectors to distinguish marginal or nonfunctional geometries (Wu et al., 29 Oct 2025).

3. Vision-based Tool Selector and Transfer to Real-World Objects

The dataset of synthetic embodiment extensions, along with their associated task outcomes, is used to supervise a depth-image-based tool selector $\mathcal{D}_{gts}$. For training, each synthetic tool receiving a successful downstream manipulation is labeled with a binary per-pixel map identifying the “handle” (first blocks affixed to the wrist). Failures are labeled with all-zero maps. The selector is optimized under $l_2$-loss to predict these maps from depth images.

In real-world operation, objects are detected and cropped using a generic 2D object detector. Each is scored by the selector, with the candidate that maximizes the predicted tool-utility activation selected for manipulation. This geometric, rather than semantic, approach enables the system to generalize to unseen objects purely through shape affordances.

By decoupling tool selection from semantic knowledge and relying on geometry, GeT-USE circumvents the limitations of category-based or object-centric datasets and allows for general-purpose, zero-shot transfer of tool-use strategies (Wu et al., 29 Oct 2025).

4. Grasping and Manipulation Policy Learning

After object selection, GeT-USE employs learned visuomotor policies for grasping ($\pi_{gtg}$) and manipulation ($\pi_{gtm}$):

• Grasping: The robot practices grasping successful, detached synthetic tools in random simulated configurations. The policy uses depth images, proprioception, and outputs full-body (base, arms, torso, grippers) commands. A dense reward encodes the proximity/alignment of the grippers to the tool base.
• Manipulation: Once grasped, the robot executes the target task (sweeping, hooking, decanting) using depth input and end-to-end policy learning, with rewards corresponding to task-specific objective completion.

Auxiliary success detectors ($\mathcal{D}_{gs}$ for grasping, $\mathcal{D}_{ms}$ for manipulation) are trained to trigger stage transitions by predicting impending task or grasp success. Policies are trained using standard RL expected return, with shaping rewards and sparse completion signals (Wu et al., 29 Oct 2025).

5. Experimental Results, Baselines, and System Analysis

GeT-USE achieves significant performance improvements over prior approaches to robotic tool use and multiple ablation baselines:

• Overall real-world success: 61.7% across 60 trials (vs. 31.7% for TOG-Net-Internet, 23.3% for TOG-Net-Proc, and 0% for TOG-Net).
• Per-task success: 80% (Hook), 55% (Sweeping), 50% (Decanting).
• Tool selection and full 6-DOF manipulation are necessary for high success, as demonstrated by ablations (removing the tool selector or constraining to top-down grasping drastically degrades performance).
• The approach is robust to the absence of an ideal tool; geometric selection allows “good enough” objects to be used in novel ways.
• The system is tested in cluttered environments with over 20 object categories and no inclusion of canonical tool exemplars (Wu et al., 29 Oct 2025).

The framework requires only a single simulation-to-real transfer per task and no real-world fine-tuning, leveraging the distributional breadth of simulated geometric tool experience.

6. Limitations, Failure Modes, and Scope

GeT-USE explicitly assumes:

• Accurate rigid-body simulation and minimal sim-to-real gap for rigid objects.
• Tasks involving objects that can be reasonably described as rigid, and manipulations compatible with parallel-jaw grippers.
• Tasks that do not require dexterous in-hand manipulation, fine manipulation of deformable or liquid objects, or articulated tool use.

Primary failure cases are attributed to dynamic discrepancies between simulation and reality (especially in contact, vibration, or complex force transfer) and hardware limits in execution precision, especially for challenging tasks like Decanting (where vibration leads to pouring failures).

The approach is not intended for tools that require multi-fingered hands or are highly articulated; extension to such domains would require further conceptual advances in both embodiment extension representation and manipulation learning (Wu et al., 29 Oct 2025).

7. Significance and Distinction from Prior Work

GeT-USE’s methodological contributions are:

• A framework for learning tool affordance geometry through self-generated synthetic embodiment extensions, rather than cataloguing semantic tool categories.
• Deployment of a geometric-visual selector for zero-shot tool generalization and selection.
• Integration of full 6-DOF bimanual control policies trained in simulation and deployed to real-world hardware with no further adaptation.
• Demonstrated scalable performance gains over procedural, dataset, and grasp-only baselines in challenging, cluttered, and open-ended environments.

The framework provides a template for future research into generalizable robotic tool use across broad task and object domains, particularly emphasizing the advantages of geometric over class-based affordance reasoning (Wu et al., 29 Oct 2025).