HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

Abstract: Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexterous manipulation focuses on single-object, single-skill tasks. In contrast, our insight is that many sequential tasks require resource-aware grasps that conserve fingers for future actions. In this paper, we study sequential grasp-conditioned dexterous manipulation, where a robot first grasps an object and then performs a second, distinct manipulation subtask while preserving the initial grasp. We introduce HANDFUL, a learning framework that models finger usage as a limited resource and encourages exploration of resource-aware grasps through finger-level contact rewards. These grasps are subsequently selected for downstream tasks via curriculum-based policy learning. We further propose HANDFUL-Bench, a simulation benchmark that introduces sequential dexterous manipulation tasks across multiple secondsubtask objectives, including pushing, pulling, and pressing, under a shared grasp-conditioned setup. Extensive simulation results demonstrate that prioritizing resource-aware grasps improves second-subtask success and robustness compared to a baseline that greedily optimizes the initial grasp before attempting the second subtask. We additionally validate our approach on a real dexterous LEAP hand. Together, this work establishes resource-aware grasp planning as a key principle for multifunctional dexterous manipulation. Supplementary material is available on our website: https://handful-dex.github.io.

• The paper introduces HANDFUL, demonstrating that resource-aware grasp policies significantly improve multi-step dexterous manipulation tasks.
• It employs diverse, finger-constrained grasp policy learning and curriculum-based chaining to effectively reserve resources for sequential subtasks.
• Simulation and sim-to-real experiments show success rates up to ~78% in simulation, highlighting robust performance through optimized finger allocation.

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

Problem Setting and Motivation

The work introduces HANDFUL, a framework for sequential grasp-conditioned dexterous manipulation using resource-aware grasp policies. The central observation is that many multi-step manipulation tasks with dexterous hands require specific fingers to be reserved for subsequent subtasks, rather than always maximizing grasp stability or force closure during the initial grasp. Most prior research has addressed single-object, single-skill settings, which do not necessitate explicit finger resource allocation. In contrast, this work addresses the sequential composition of two manipulation subtasks—e.g., grasping a block and manipulating a second object or interface—where grasp choice is critical for downstream effectiveness.

The method is motivated by issues observed in applications where a dexterous manipulator must maintain force closure on an object while performing secondary manipulations, such as pushing, pressing, twisting, pulling, or grasping an additional object. Conventional approaches that treat all fingers as equally available for each subtask often result in early grasps that block required contact points for later stages, limiting overall task success.

Figure 1: Overview of HANDFUL. Multiple grasping policies with explicit finger activation/inactivation produce grasps that preserve fingers for downstream tasks; grasp and manipulation policies are chained via curriculum learning and retrieval-based sim-to-real transfer.

Methodology

HANDFUL is structured into three main components:

1. Diverse Finger-Constrained Grasp Policy Learning: The system decomposes the hand into “active” and “inactive” fingers for each subtask. The grasping policy is explicitly penalized for activating fingers required by the next subtask, encouraging grasps that conserve those fingers. All unique feasible 1- and 2-finger configurations (over a four-finger hand) are realized as separate grasping policies. Reward functions are tailored to promote proximity of designated fingers while penalizing contacts for those that should remain available.
2. Curriculum-Based Second Subtask Policy Learning and Grasp Selection: For each grasping policy, a downstream manipulation policy (for the second subtask) is trained, conditioned on the terminal grasp state. To maximize sample efficiency, a curriculum is constructed, beginning with low-environmental randomization and increasing over three stages. At each stage, poorly performing grasp-manipulation policy pairs are pruned, allowing the most promising policies to be further refined. Selection criteria include terminal and interim success rates and average episodic return, ensuring both effectiveness and robustness of policy chaining.
3. Compositional Deployment with Sim-to-Real Transfer: The learned policy pairs are deployed in the real world using a retrieval strategy. For each real scene, the best-matching (in pose and condition) simulated grasp trajectory is executed; this avoids the instability and sim-to-real gap seen in direct policy deployment. The approach leverages precise pose segmentation and matching with trajectory datasets collected in simulation.

   Figure 2: Overview of HANDFUL-Bench, a simulation benchmark with multi-step manipulation tasks that share an initial grasp and vary in downstream subtask objectives for systematic evaluation of resource-constrained hand policies.

Benchmark: HANDFUL-Bench

To provide rigorous, reproducible evaluation, the authors present HANDFUL-Bench, a benchmark suite comprising five sequential tasks each requiring (1) grasping an object, and (2) performing a secondary action (pushing another object, pressing a button, twisting a knob, pulling a drawer, or picking a second object) while maintaining the grasp. HANDFUL-Bench is built on ManiSkill with a 7-DoF xArm7 and a 16-DoF LEAP hand. All tasks are designed to create explicit conflict between finger-use requirements for the two subtasks, making naïve single-skill solutions suboptimal.

Each policy receives full proprioceptive and task-relevant state, with action spaces covering both arm and hand. Curriculum and evaluation protocols are standardized to facilitate comparison.

Simulation Results and Analysis

The simulation study demonstrates several key findings:

• Finger-Constrained Grasp Diversity: Enforcing finger allocation constraints and learning multiple grasp modalities significantly improves secondary subtask performance. For instance, methods that neglect finger constraints (treating all fingers as active throughout) fail completely on tasks like two-object grasping, and display high variance on complex manipulation tasks. Explicit finger allocation is essential for robustness.
• Curriculum Learning: The curriculum-based selection pipeline achieves parity or improvement in success rate while reducing training time by ~40%. Early curriculum stage metrics are shown to be reliable predictors of final policy success, avoiding premature elimination of optimal grasp candidates.
• Ablation Performance: Phase-based reward schemes and monolithic policies trained without task decomposition underperform, routinely failing to find stable, sequential solutions due to conflicting reward landscape and inability to consistently maintain proper resource allocation across steps.
• Numerical Results: HANDFUL achieves simulation success rates of up to ~78% (across seeds and tasks) for complex multi-step dexterous tasks, outperforming baselines and ablations with especially notable gaps on tasks requiring significant finger resource conservation.

  Figure 3: HANDFUL with curriculum learning achieves both higher and more stable performance than training without curriculum over the full 10M step RL schedule.

Sim-to-Real and Human Data Comparison

The real-world evaluation leverages the retrieval-based execution on xArm7 with the LEAP hand, using RGB-D cameras for pose estimation. HANDFUL achieves two-stage (both grasp and manipulation) success rates up to 67% on the Push Object task and 26–53% on more challenging tasks, indicating robust transfer given sim-to-real challenges.

Notably, the sim-to-real gap is highly policy-dependent: grasping policies utilizing palm contact are more robust than fingertip-only strategies, and some grasp types generalize better due to improved stability under open-loop execution.

A comparison with policies trained on teleoperated human demonstration data (DP3) reveals that human operators almost exclusively designate the index finger for downstream manipulations, in part a limitation of the teleoperation interface. In contrast, HANDFUL learns to exploit more diverse and non-anthropomorphic hand poses (including significant palm use) to accommodate the task, and outperforms DP3 on four of five tasks.

Figure 4: Real-world evaluation setup with the xArm7/LEAP hand, two RGB-D cameras, and test objects for all subtasks.

Figure 5: Human teleoperated solutions showcase limited pose diversity and a bias toward using the index for subtasks, contrasting with HANDFUL's systematically diverse and resource-aware grasps.

Sim-to-Real Gap and Policy Transfer

Despite high simulation performance, the sim-to-real success rates for grasping policies exhibit significant variability, with some strategies retaining high efficacy and others collapsing due to lack of robustness, especially under open-loop/no feedback execution. Policies that maximize palm contact are most resilient under domain shift, underscoring the importance of robust contact selection in dexterous policies.

Figure 6: Sim-to-real performance scatter plot for the nine grasping policies. While simulation performance is uniformly high, real-world results vary dramatically, manifesting a significant sim-to-real gap.

Limitations and Future Directions

The current experiments rely on state observations and a limited task set. Greater environmental diversity, broader task objectives, and evaluation on alternate robotic hands are necessary for full generalization. The retrieval-based sim-to-real strategy is also vulnerable to modeling errors when environment or object pose differ substantially from those encountered in simulation.

Future extensions should examine (1) task sequences involving more than two subtasks, (2) integration of vision-based feedback, (3) policy adaptation during execution, and (4) more general evaluation of transfer protocols across different manipulation hardware.

Conclusion

HANDFUL provides a methodologically rigorous solution to sequential dexterous manipulation tasks that require explicit finger resource allocation. Through finger-constrained grasp policy learning, curriculum-based chaining, and benchmark-backed evaluation, the work demonstrates the fundamental role of resource-aware pre-grasp planning in multi-step manipulation success. By systematically dissecting the role of grasp modality on composite task achievement, HANDFUL enables better exploitation of hand morphology and sets an experimental foundation for future work on sequential and multifunction manipulation in robotics.