Learning Dexterous Grasping from Sparse Taxonomy Guidance

Abstract: Dexterous manipulation requires planning a grasp configuration suited to the object and task, which is then executed through coordinated multi-finger control. However, specifying grasp plans with dense pose or contact targets for every object and task is impractical. Meanwhile, end-to-end reinforcement learning from task rewards alone lacks controllability, making it difficult for users to intervene when failures occur. To this end, we present GRIT, a two-stage framework that learns dexterous control from sparse taxonomy guidance. GRIT first predicts a taxonomy-based grasp specification from the scene and task context. Conditioned on this sparse command, a policy generates continuous finger motions that accomplish the task while preserving the intended grasp structure. Our result shows that certain grasp taxonomies are more effective for specific object geometries. By leveraging this relationship, GRIT improves generalization to novel objects over baselines and achieves an overall success rate of 87.9%. Moreover, real-world experiments demonstrate controllability, enabling grasp strategies to be adjusted through high-level taxonomy selection based on object geometry and task intent.

• The paper presents GRIT, a two-stage framework that decouples high-level grasp taxonomy selection from low-level finger motion generation.
• The paper reports improved grasp success rates and joint adherence by leveraging 30 discrete canonical grasp templates and a vision-language model.
• The paper integrates a multiplicative reward structure and teacher-student distillation to achieve robust manipulation in both simulation and real-world trials.

Learning Dexterous Grasping from Sparse Taxonomy Guidance: An Expert Summary

Framework Overview and Motivation

The paper introduces GRIT, a two-stage dexterous manipulation framework in which high-level semantic grasp taxonomy selection is decoupled from low-level finger motion generation. The approach addresses three primary challenges in dexterous grasping: (1) efficient generation of grasp configurations conditioned on object geometry and task context; (2) translation of sparse taxonomy-based commands into continuous control actions without explicit demonstration trajectories; and (3) enabling controllability for user intervention during manipulation failures.

GRIT leverages a library of canonical grasp taxonomies, extracted from human grasp patterns, which encapsulate representative finger engagement and contact structures. These discrete taxonomies serve as foundational templates for dexterous control, allowing GRIT to generalize across diverse objects and tasks with minimal supervision.

Figure 1: The framework selects context-appropriate grasp taxonomies based on object geometry and task, executing them through taxonomy-conditioned control policies.

Architecture and Key Components

GRIT comprises three main modules:

1. Taxonomy Library: Encodes 30 canonical grasp templates derived from the Feix et al. taxonomy, each characterized by reference joint configurations and contact structures.
2. Grasp Generation Module: During training, taxonomies are sampled randomly. For inference, a vision-LLM (VLM) predicts taxonomies from scene images and task descriptions, overlaying 3D axes to facilitate spatial reasoning from 2D inputs.
3. Taxonomy-Conditioned Control Policy: Learned through teacher-student distillation. The teacher has privileged access (full object state); the student observes only partial RGB-D data and action-state history.

Training utilizes a novel multiplicative composite reward structure, integrating hand-centric approach terms, object-centric stability constraints, and explicit penalties for unintended contacts. This formulation enforces both stable manipulation and precise adherence to prescribed taxonomy configurations.

Figure 2: Overview of GRIT: taxonomy library, grasp generation (vision-LLM), and taxonomy-conditioned policy via teacher-student distillation.

Evaluation and Numerical Results

Experiments span simulated and real-world manipulations, using YCB and Objaverse datasets. GRIT demonstrates superior generalization performance relative to prior art: Success rates improve from 81.9% (RDG, no taxonomy) and 86.0% (GraspXL, geometric reward shaping) to 87.9%. Top-3 oracle-selected taxonomies achieve 92.3%, indicating VLM-predicted taxonomies are near-optimal.

A key finding is the strong dependency of grasp taxonomy efficacy on object geometry. For fruit/vegetable objects, performance spreads over 25.07% across taxonomies; for packed goods, diameter-oriented grasps outperform precision pinches by 14.85%. These results support the bold claim that explicit discrete taxonomy conditioning provides more robust structural guidance and generalization than continuous pose or geometric region shaping.

Figure 3: The object pose is fixed, wrist is sampled from eight directions; success rates are evaluated for each approach across 30 trials.

Figure 4: Mean joint configuration error compared to taxonomy reference, highlighting adherence capability.

Figure 5: Qualitative examples of generated grasps in both simulation and real-world deployment.

Reward Formulation and Adherence Analysis

A central innovation is the multiplicative reward structure, wherein taxonomy adherence acts as a gating coefficient on task success rewards. Comparative analysis with baseline additive formulations reveals that multiplicative gating consistently achieves high joint adherence and contact precision without hyperparameter tuning, demonstrating stable taxonomy enforcement while maximizing task success rates.

Ablation studies indicate that dense visual features (Basis Point Set, BPS) significantly boost precision grasping performance, with contact precision improved by 28.57% and joint error reduced by 57.83% compared to naive geometric encoding.

Real-World Implications and Deployment

GRIT is successfully deployed on humanoid robots using taxonomy-conditioned controllers as inverse kinematics targets. Experiments show that optimal taxonomy selection is strongly geometry-dependent (e.g., rod objects require finger pinch, spools prefer cylindrical enclosures) and adapts to task context via text prompts (e.g., "squeeze" selects power grasp, "transport" selects precision grasp).

Figure 6: Suitable grasp taxonomies depend on object geometry: illustrated by differential success across taxonomies for rod-like versus spool-like objects in both simulation and real-world trials.

Theoretical and Practical Implications

GRIT presents evidence that sparse discrete taxonomy guidance achieves superior generalization and controllability in dexterous grasping, especially when integrated with vision-language prediction. The framework formalizes a separation between high-level intent (via taxonomy) and low-level execution (via policy), supporting modularity and user intervention during manipulation.

Practically, the system enables real-time adaptation of grasp strategies by adjusting taxonomy selection based on changing object geometry or task intent, benefiting applications in assistive robotics, autonomous manipulation, and robot learning from minimal supervision.

Theoretically, the results call for a reevaluation of compositionality in manipulation learning: discrete semantic guidance can be systematically encoded to improve policy generalization and structural adherence in high-DOF manipulation, challenging the necessity for dense demonstration coverage or direct pose mapping.

Limitations and Speculative Future Directions

Current limitations include reliance on initial visual observations and lack of real-time feedback adaptation for dynamic scenes or moving objects. The taxonomy selection is coarse, and fine-grained semantic targeting (e.g., specifying functional subparts) remains infeasible.

Future directions could include integrating closed-loop reactive visual feedback, expanding taxonomy libraries with functional affordances, and enhancing VLM-driven semantic prediction for robust manipulation in highly dynamic or cluttered environments.

Conclusion

GRIT offers a structured, taxonomy-driven framework for dexterous robot grasping, achieving strong generalization, high adherence to semantic grasp configurations, and real-world controllability. The approach demonstrates that sparse taxonomy guidance, combined with vision-language prediction and multiplicative reward enforcement, is an effective solution for scalable, robust, and adaptable dexterous manipulation.