UltraDexGrasp-20M: Universal Bimanual Grasp Dataset

• The paper introduces UltraDexGrasp-20M, a 20M-frame, multi-strategy dataset that advances universal dexterous grasping for bimanual robots.
• It combines optimization-based grasp synthesis with planning-based trajectory generation, achieving 84% success in simulation and 81.2% in real-world tests.
• The framework leverages synthetic point cloud policy learning to select grasp strategies based on object geometry, size, and mass for robust performance.

Searching arXiv for the cited papers to ground the article in the current record. UltraDexGrasp-20M is a large-scale, multi-strategy synthetic grasp dataset for bimanual dexterous robots, comprising 20,000,000 frames over 1,000 objects. It was introduced together with the UltraDexGrasp framework for universal dexterous grasping with bimanual robots, whose central objective is to select an appropriate grasp strategy and hand posture as a function of object geometry, size, and mass. The framework combines optimization-based grasp synthesis with planning-based demonstration generation, and the associated point-cloud policy, trained exclusively on synthetic data, attains an average success rate of 81.2% in real-world universal dexterous grasping (Yang et al., 5 Mar 2026).

1. Scope, motivation, and task definition

UltraDexGrasp-20M addresses a problem setting that had remained comparatively underexplored relative to parallel-gripper and single-hand grasping: dexterous grasping for bimanual robots in multi-strategy regimes. The motivating difficulty is not merely the kinematic dimensionality of dual-arm, dual-hand systems, but the need to generate grasps that are simultaneously physically plausible, geometrically conforming, and robust to external wrenches while coordinating two manipulators and two dexterous hands (Yang et al., 5 Mar 2026).

The dataset is explicitly framed around universal dexterous bimanual grasping. In this context, “universal” denotes the capacity to choose among multiple grasp strategies according to object properties. The supported strategies are two-finger pinch, three-finger tripod, whole-hand grasping, and bimanual grasping of large objects. Strategy selection is realized by activating different fingertip contact sets on the hands. The paper does not specify the distribution of frames across strategies, but the strategy taxonomy itself is central to the dataset’s design.

A notable design motivation is the claim that synthetic data is required to scale both diversity and quality while avoiding the cost and limited coverage of teleoperation or RL experts. UltraDexGrasp-20M is therefore not merely a corpus of static grasps; it is intended to support policy learning from physically validated demonstrations that encode approach, closure, squeezing, and lifting behavior.

2. Synthetic data-generation pipeline

The UltraDexGrasp pipeline has two principal components: optimization-based grasp synthesis and planning-based demonstration generation. The first initializes hands relative to the object convex hull and optimizes hand poses and contact forces under physical plausibility and geometric conformity criteria. The second uses bimanual motion planning to produce collision-free, coordinated trajectories that execute a selected preferred grasp in closed-loop with PD control and physical validation (Yang et al., 5 Mar 2026).

The high-level procedure begins by importing object assets and robot URDFs into a simulator. The objects are 1,000 from DexGraspNet. During generation, the system randomizes camera pose and joint impedance to reduce sim-to-real gap. For each object, it generates 500 candidate grasps via the optimizer, filters physically implausible candidates, checks inverse-kinematics reachability and collisions with cuRobo, and ranks feasible candidates by SE(3) distance from the current end-effector pose in order to choose the preferred grasp.

Trajectory generation follows a fixed four-stage structure: pregrasp, grasp, squeeze, and lift. The pregrasp is defined as an offset of 0.1 m opposite the palm, and the lift stage raises the object by 0.2 m. Success validation is also explicit: the object must be lifted by at least 0.17 m and held for at least 1 s. This emphasis on executable trajectories differentiates the dataset from purely pose-level grasp annotations. A plausible implication is that UltraDexGrasp-20M is intended as much for policy supervision as for grasp candidate benchmarking.

3. Dataset composition, robot platform, and recorded modalities

UltraDexGrasp-20M comprises 20,000,000 frames over 1,000 objects. The paper evaluates on 600 test objects with variation in shape, size, and mass, but the exact train/validation composition is not specified. A “frame” is a single time step recorded and rendered during demonstration execution and includes observations and commands at that time; the exact sampling rate in simulation is not specified (Yang et al., 5 Mar 2026).

The hardware model underlying the dataset is a dual-arm, dual-hand system with two UR5e arms, each with 6 DoF, and two XHand dexterous hands, each with 12 DoF. XHand is described as a five-fingered dexterous hand, although the exact per-finger joint allocation is not itemized. Simulation is noted as based on SAPIEN. Contact modeling uses point-on-plane contacts with a hard finger model and friction cone constraints.

Per-frame perception is centered on point clouds. The scene point cloud is rendered from cameras, and an “imaged point cloud” of the robot is added using known robot joint states and CAD in simulation to mitigate sim-to-real gap. The simulator also has access during synthesis and planning to the object mesh, object pose and center of mass, robot kinematics, and contact states. However, the paper does not enumerate which of these are stored per frame in the released dataset. Likewise, exact file formats, storage size, and dataset license are not specified.

Aspect	Reported detail
Scale	20,000,000 frames over 1,000 objects
Strategies	Two-finger pinch, three-finger tripod, whole-hand, bimanual
Robot	2×UR5e arms + 2×XHand dexterous hands
Per-frame observation	Scene point cloud plus robot imaged point cloud
Split information	600 test objects reported; exact train/val composition not specified

The object population is drawn from DexGraspNet. Mass and size ranges are reported for evaluation rather than as guaranteed dataset metadata. In simulation, the 600 evaluation objects span masses from 5 g to 1,000 g, with the smallest having longest bounding-box edge below 0.03 m and the largest having shortest bounding-box edge above 0.5 m. In real-world experiments, object masses range from 3.6 g to 1,095 g and volumes from 18 cm$^3$ to 26,400 cm$^3$.

4. Grasp synthesis formulation and physical criteria

UltraDexGrasp formalizes a bimanual grasp pose as

$$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$

where $h \in \{0,1\}$, $\boldsymbol{t}_h \in \mathbb{R}^3$, $\boldsymbol{R}_h \in SO(3)$, and $\boldsymbol{q}_h \in \mathbb{R}^n$ is the hand-joint vector. This representation couples two wrist poses with two hand configurations and is the basic decision variable of the synthesis problem (Yang et al., 5 Mar 2026).

The contact model is the hard finger friction cone

$$\mathcal{F} = \left\{ \boldsymbol{f} \;\middle|\; \|\boldsymbol{f}_{\mathrm{tan}}\| \le \mu \|\boldsymbol{f}_{\mathrm{n}}\|,\;\; f_z \ge 0 \right\},$$

with static friction coefficient $\mu$, normal force $\boldsymbol{f}_{\mathrm{n}} = [0, 0, f_z]$, and tangential component $^3$0. For the $^3$1-th contact, the contact wrench is

$^3$2

where $^3$3 is the vector from center of mass to contact point, $^3$4 is a fixed scaling constant, and $^3$5 is the contact force. The grasp map is

$^3$6

where $^3$7 is the contact position, $^3$8 is the center of mass, and $^3$9 rotates the local contact frame to the object frame. The Grasp Wrench Space is then defined as

$$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$0

The optimization-based synthesis is an upper–lower bilevel procedure. Its decision variables are the bimanual grasp pose and the active-contact forces. The objective combines four terms: a wrench-tracking term that matches achievable wrench to scaled target wrenches $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$1, a contact-distance term $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$2 encouraging fingertip–surface proximity and conformity, a hand–object collision penalty $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$3, and a hand–hand penetration penalty $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$4. The constraints enforce joint limits, contact forces inside the friction cone, and wrist rotations in $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$5. The lower level solves a per-contact QP for the contact forces under friction constraints, while the upper level applies gradient descent on the hand pose; the implementation is reported as using cuRobo and a GPU QP solver.

Preferred-grasp selection among feasible candidates uses an SE(3) distance:

$$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$6

with

$$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$7

This criterion favors grasps close to the current end-effector state, reducing planning burden after physical feasibility filtering.

The paper does not explicitly compute the Ferrari–Canny $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$8 metric and does not explicitly verify force closure through the convex-hull-origin condition. Stability is instead encouraged through wrench tracking, friction-cone constraints, and the physical lifting test. This is an important methodological distinction: quality assessment is operational and task-validated rather than reduced to a single analytic grasp metric.

5. Policy architecture and learning objective

The policy trained on UltraDexGrasp-20M takes point clouds as input. Preprocessing downsamples the point set to 2,048 points using farthest point sampling. The encoder is PointNet++ with two set abstraction layers. The first retains 2,048 points and groups neighborhoods by k-NN with $$g = \left\{ \left( \boldsymbol{t}_h, \boldsymbol{R}_h, \boldsymbol{q}_h \right) \mid h = 0, 1 \right\},$$9, applying per-group local feature extraction via $h \in \{0,1\}$0 convolution, batch normalization, ReLU, and max-pooling. The second downsamples to 256 points with a similar local-feature pipeline. These encoded scene tokens are then consumed by a decoder-only transformer in which action query tokens attend unidirectionally to point features (Yang et al., 5 Mar 2026).

The paper identifies unidirectional attention as a key design choice. Compared to allowing all tokens to attend to one another, unidirectional attention improves performance. The output head predicts a bounded Gaussian, described as a truncated normal distribution over actions, and training minimizes the negative log-likelihood of ground-truth demonstration actions. The exact action parameterization is not specified: the paper does not state whether the control commands are end-effector poses, joint targets, deltas, or absolute commands. That omission limits direct architectural comparison with policies whose action spaces are fully enumerated.

Two ablation results are especially informative. Removing distribution prediction reduces success rate to 73.5%, and removing unidirectional attention reduces it to 68.2%, whereas the full model achieves 84.0% in simulation. These results indicate that both stochastic action modeling and the specific scene-to-action aggregation pattern are critical to policy performance in this setting.

6. Evaluation protocol, scaling behavior, and sim-to-real transfer

Simulation experiments use the full dual-arm dual-hand system with 600 objects, including both seen and unseen instances, and perform 10 trials per object. Small and medium objects are randomly placed in a $h \in \{0,1\}$1 area, while large objects are placed in a $h \in \{0,1\}$2 area. Under this protocol, UltraDexGrasp reports an average success rate of 84.0%, compared with 46.7% for DP3 and 58.8% for DexGraspNet; DexGraspNet cannot handle large objects in this benchmark (Yang et al., 5 Mar 2026).

The breakdown by object regime further clarifies performance. For seen objects, success rates are 78.8% on small objects, 84.3% on medium objects, and 90.4% on large objects. For unseen objects, they are 76.9%, 85.8%, and 87.5%, respectively. The reported improvement over DP3 is +37.3 percentage points on average. The paper also reports a scaling trend in which performance improves as the number of training frames increases. The average success rate of grasp generation itself is 68.5%, and the learned policy surpasses the generation pipeline when the number of training frames exceeds 1M. This suggests that imitation on large synthetic corpora can denoise or regularize imperfections in the generating process rather than merely reproducing them.

Real-world experiments use two UR5e arms placed 0.9 m apart, two XHands, and two Azure Kinect DK cameras in an eye-on-base configuration. Control frequency is 10 Hz. Sim-to-real transfer relies on consistent coordinate frames, intrinsic and extrinsic camera calibration, Statistical Outlier Removal for depth outliers, robot imaged point clouds, and joint impedance randomization in simulation. On 25 real objects with 15 trials per object, the reported average success rate is 81.2%, with 72.0% on small objects, 82.2% on medium objects, and 89.3% on large objects. The paper characterizes this as strong zero-shot sim-to-real transfer and notes that the policy adapts its strategy across tripod, whole-hand, and bimanual grasps.

7. Relation to adjacent datasets, availability, and limitations

UltraDexGrasp-20M occupies a different design point from clutter-centric dexterous grasp benchmarks such as "DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes" (Zhang et al., 2024). DexGraspNet 2.0 reports 1,319 objects, 8,270 scenes, and 427M grasp labels across cluttered scenes, with a single LEAP Hand model and a local-geometry-conditioned diffusion method for grasp generation from single-view depth point clouds (Zhang et al., 2024). UltraDexGrasp-20M, by contrast, emphasizes bimanual robots, four grasp-strategy families, and demonstration trajectories rather than scene-level clutter annotations. This suggests complementary rather than redundant coverage: DexGraspNet 2.0 is clutter-centric and grasp-label-centric, whereas UltraDexGrasp-20M is strategy-centric and demonstration-centric.

The UltraDexGrasp paper explicitly open-sources the data-generation pipeline at the repository https://github.com/InternRobotics/UltraDexGrasp, and it also provides a project page at https://yangsizhe.github.io/ultradexgrasp/ (Yang et al., 5 Mar 2026). However, the availability of the full dataset, its exact format, and its license are not specified in the paper. Reproduction details are therefore stronger for pipeline concepts than for dataset packaging.

Several limitations are reported or directly implied by missing specifications. Although 20M frames across 1,000 objects is large, the distribution among strategies and scenes is not given, which limits fine-grained analysis of curriculum balance and per-strategy bias. Exact data schemas, storage layouts, and split definitions are not specified, which can impede faithful reuse. Contact realism is constrained by the hard finger model, friction-cone constraints, and lift-based validation; tactile sensing, compliance-rich interaction, and more sophisticated impedance or admittance control are not reported. Finally, the task scope is grasp acquisition through pregrasp–grasp–squeeze–lift rather than extended manipulation such as reorientation, handover, or in-hand dexterous manipulation. Future extensions would plausibly require richer demonstrations, more explicit strategy labels, and broader contact-state annotation.