AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection

Abstract: Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts. Such data is hard to obtain at scale: teleoperation yields valid physical outcomes but is slow and operator-biased, while simulation-based generation is cheap and scalable but cannot certify contact validity. A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop (perception, execution, labeling, and reset) runs without human intervention. We present AutoDex, an automated real-world data-collection system that closes this loop: for each candidate from a replaceable generator, it localizes the object under severe hand-object occlusion with dense 20-camera perception, executes collision-monitored robot motions, labels lift-and-hold success or failure, and actively resets the object between trials to expose additional candidates across stable poses. The result is a reusable database of physically labeled grasp trials that downstream systems can query by retrieval and feasibility filtering. Using AutoDex, we collect 3,593 grasp trials across Allegro and Inspire hands on 100 diverse objects, with synchronized multi-view observations and robot-state logs. For a matched 500-trajectory collection, AutoDex requires 10.3 h versus 49.4 h for teleoperation, yielding a 4.8x throughput improvement, and grasps retrieved from the AutoDex-validated database succeed 76% versus 34% for simulation-only validation. Code and data will be publicly released.

• The paper demonstrates an automated pipeline that achieves a 4.8× throughput gain over human teleoperation for dexterous grasp data collection.
• The system integrates multi-view pose estimation, collision-safe grasp execution, and autonomous object reset for reliable, physically validated grasp trials.
• Real-world experiments show a grasp success rate of 76% with physical validation versus 34% for simulation-only, highlighting the sim-to-real gap.

AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection

Problem Motivation and Context

The reliable synthesis of dexterous robotic grasps on diverse objects remains bottlenecked by the dearth of scalable, real-world datasets annotated with ground-truth physical outcomes. Multi-fingered dexterous grasping, in contrast to parallel-jaw approaches, depends critically on factors that are not robustly modeled or simulated—namely the intricacies of contact physics, slip, compliance, and unmodeled actuator or hand-object-object frictional effects. While simulated grasp generation and geometric feasibility checks enable large-scale candidate proposal, only execution on physical hardware reliably determines grasp validity. However, current real-world data acquisition methods—such as human teleoperation—are fundamentally limited in throughput, generalization, and introduce operator bias. Thus, there is a critical need for an autonomous, end-to-end real-world data collection system that can cycle through perception, grasp execution, outcome labeling, and reset without human intervention.

System Architecture and Methodology

AutoDex is introduced as a fully automated pipeline designed to collect large-scale, physically annotated grasp data across diverse household objects and hardware setups. The system’s core comprises a 6-DoF xArm manipulator equipped with hot-swappable multi-fingered hands and a 20-camera synchronized, LED-lit workcell, enabling dense multi-view perception critical for robust pose estimation under severe object occlusion (Figure 1).

Figure 1: The multi-camera AutoDex workcell with 6-DoF arm, swappable hand, and examples of generated candidate grasps paired with real-world executions across varied scene constraints.

At each cycle, a set of grasp candidates is generated using a modular external generator (here, BODex), which leverages object models and scene constraints to propose kinematically and geometrically feasible grasp poses. These do not guarantee stability under contact dynamics and so require real-world validation. AutoDex’s loop consists of:

1. Robust Multi-View Pose Estimation: 20 high-resolution, synchronized RGB cameras estimate the object’s stable 6D pose, maintaining high accuracy even under occlusion (mean ADD-S $<$1mm with $\geq$8 cameras, Figure 2).
2. Feasibility Screening: Candidates are filtered for IK solvability and collision-free trajectories in the current scene.
3. Safe Grasp Execution: The robot executes candidate grasps with trajectory segments protected by a learned, residual-torque-based collision monitor, which is more robust than factory sensors in detecting unexpected contact loads and avoids spurious aborts.
4. Lift-and-Hold Validation: Grasp efficacy is decided by lifting the object 5cm and holding for 3s, tracked post hoc by high-redundancy multi-view pose tracking.
5. Autonomous Object Reset: Upon exhausting the candidate set for a pose, the robot reorients the object to a new stable pose (requiring active, non-passive placement), enabling exhaustive and efficient exploration of candidate grasps across all object configurations (Figure 2).

Database Construction and Retrieval

The outcome is a database containing thousands of physically executed and labeled grasp trials (success/failure), with each record comprising synchronized robot and camera data, pose trajectories, and scene descriptors. Importantly, the system supports online retrieval: at deployment, given novel scene geometry and object pose estimated from a reduced set of cameras, successful database grasps are filtered for scene feasibility (IK, collision), and the first feasible candidate is directly executed without additional training or manual intervention (Figure 3).

Figure 3: Illustration of the AutoDex pipeline, from candidate execution, physical validation, to deployment-time grasp retrieval.

Experimental Analyses

Throughput and Autonomous Operation

In matched experiments (500 trajectories), AutoDex requires just 10.3 hours to collect all trials, compared to 49.4 hours for human teleoperation—a 4.8$\times$ gain in effective throughput. This gain is achieved not by reducing per-execution time, but by eliminating all human idle time and enabling unattended, overnight operation cycles (Figure 4).

Figure 4: Left—AutoDex throughput compared to teleoperation; Right—Success rate of AutoDex-validated grasp database versus model-screened (simulation-only) candidates.

Effect of Physical Validation

The study demonstrates that geometric/model-based feasibility checks are grossly insufficient. When deploying the AutoDex-validated database (grasps that succeeded in physical trials), the real-world grasp success rate is 76%. In contrast, grasps from a purely model-screened (simulation-validated) database only succeed 34% of the time, even after filtering for scene feasibility at deployment. The discrepancy is even more pronounced in cluttered scene configurations. This contradicts claims that simulation-only pipelines suffice for reliable database construction in dexterous grasping.

Reset Strategies and Unattended Collection

AutoDex’s reset module enables transitions between arbitrarily rare or difficult object pose pairs, maintaining nearly 100% reset success even for transitions with zero probability under passive drop. In contrast, strategies relying on reorient-and-drop or passive settlement fail routinely, and cause unacceptable trial ejections that require human intervention, thereby preventing fully unattended operation (Figure 2).

Figure 2: Left—Reset strategy comparison versus passive transition probability; Right—Pose self-consistency (mean ADD-S) as function of available camera count.

Multi-Camera Perception and System Robustness

By leveraging 20 synchronized RGB cameras, pose estimation is robust to occlusion, lighting, and object geometries. Pose self-consistency improves monotonically with the number of views, saturating at 8+ cameras. Fatal pose errors are virtually eliminated with dense perception, which is critical for high DoF hand operation without operator oversight.

Practical and Theoretical Implications

AutoDex sets a new standard for scalable, real-world data collection in dexterous grasping, decoupling database size and diversity from human operator hours. The demonstrated gains in success rate via physical validation (76% versus 34% for model-only) reveal the persistent sim-to-real gap for high-DoF contact interactions, and highlight that real-world execution is essential for dataset reliability. The results also establish that autonomous reset, collision-safe execution, and dense perception are requisite components for scalable unsupervised data-collection systems in robotics.

Practically, the AutoDex database enables direct retrieval-based in-the-wild deployment, reduces reliance on policy training, and improves the generalization of downstream learning and planning systems. Theoretically, the findings suggest that the persistent inaccuracies of frictional and contact modeling in simulation—particularly under high-dimensional, multi-contact scenarios—are unlikely to be fully bridged by current or near-term simulation advances alone.

Limitations and Future Work

Current limitations include a focus on single-arm, single-hand setups with stable, table-based object presentation. The system does not address mobile manipulation, bimanual tasks, dynamic in-contact re-grasping, or functional manipulation (tool use, handover). The requirement for a large, instrumented multi-camera cell limits immediately practical real-world deployment, though future work may compress these requirements. AutoDex also inherits the coverage constraints of the underlying candidate generator, missing grasps that require dynamic or exaptationist finger motions.

Further advancements could address more agile resets, dynamic contact manipulation, extend the approach to highly cluttered and unstructured environments, and scale down the perception system requirements as robust single- and small-multi-view perception matures.

Conclusion

AutoDex demonstrates the first scalable, fully autonomous real-world pipeline for dexterous grasp dataset collection. It achieves significant throughput gains over teleoperation, produces richly annotated databases of physical grasp outcomes, and enables practical retrieval-based execution with high reliability. Critically, it shows that physical validation is essential for closing the sim-to-real gap in dexterous manipulation—a claim supported by strong empirical results and systematic evaluation across pose, scene, and object categories. The AutoDex pipeline, dataset, and codebase are poised to facilitate new advances in manipulation research, data-driven RL, and robust generalization for dexterous robots.