Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation (1810.01845v1)

Abstract: Human hand actions are quite complex, especially when they involve object manipulation, mainly due to the high dimensionality of the hand and the vast action space that entails. Imitating those actions with dexterous hand models involves different important and challenging steps: acquiring human hand information, retargeting it to a hand model, and learning a policy from acquired data. In this work, we capture the hand information by using a state-of-the-art hand pose estimator. We tackle the retargeting problem from the hand pose to a 29 DoF hand model by combining inverse kinematics and PSO with a task objective optimisation. This objective encourages the virtual hand to accomplish the manipulation task, relieving the effect of the estimator's noise and the domain gap. Our approach leads to a better success rate in the grasping task compared to our inverse kinematics baseline, allowing us to record successful human demonstrations. Furthermore, we used these demonstrations to learn a policy network using generative adversarial imitation learning (GAIL) that is able to autonomously grasp an object in the virtual space.

Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation

This paper presents a comprehensive framework for imitating human hand movements to enable dexterous manipulation using anthropomorphic robotic hands. The significant contributions of the research are the development of a task-oriented hand motion retargeting approach that effectively marries inverse kinematics (IK) with particle swarm optimization (PSO), and the integration of generative adversarial imitation learning (GAIL) to facilitate learning from human demonstrations in a virtual environment.

Core Components and Contributions

The authors identify three key challenges in imitating human hand actions, notably the acquisition of accurate hand data, retargeting this to a suitable hand model, and deriving actionable policies from this data. This complex task is made tractable through the following innovations:

1. Hand Pose Estimation: A state-of-the-art hand pose estimator is utilized to capture the complexities of human hand movements. This serves as the foundational data that feeds into the retargeting model.
2. Hybrid Retargeting Method: The retargeting task, converting hand poses to a 29 DoF hand model, is approached through a novel hybrid methodology that leverages IK for initial pose approximation, supplemented by PSO for task-specific refinement. This combination ensures that the virtual hand not only approximates human hand configuration but also satisfies task-specific objectives, such as effective grasping.
3. Task Objective Optimization: By incorporating task objective optimization into the PSO process, the framework addresses noise from the hand pose estimator and the domain discrepancies between the human hand and the robotic hand model. This specifically allows for improved task execution, with a focus on maximizing the grasping success rate.
4. Generative Adversarial Imitation Learning: Using the gathered hand motion data, the authors employ GAIL to train a policy network capable of autonomously performing object grasp tasks in a simulated environment. This demonstrates the capability of the system to transition from imitation to autonomous task performance.

Experimental Insights

The paper provides compelling numerical evaluations showcasing the capability of the proposed framework to outperform a pure inverse kinematics approach in grasping tasks. Particularly, it reports improvements in the percentage of successful object grasping and retention during task execution. Furthermore, the GAIL-driven learning framework shows potential, with notable though limited generalization from training scenarios to novel, unseen task setups.

Theoretical and Practical Implications

Theoretically, the paper provides insights into the integration of hybrid optimization techniques for motion retargeting tasks, which could be potentially extended or adapted for other high-dimensional manipulation problems. Practically, the methodologies developed have implications for robotics applications where dexterous manipulation is required, such as in robotic-assisted surgery, prosthetics, and advanced manufacturing.

Future Directions

Future research could aim to refine the hybrid PSO method further, possibly through more sophisticated definitions of hand pose energy functions or by exploring reinforcement learning techniques to achieve end-to-end learning objectives for more complex manipulative tasks. Enhancements in real-time performance and the robustness of the system under varying task conditions also remain as promising avenues.

Overall, this paper contributes a systematic approach to enhancing the dexterity of robotic manipulation via learned and retargeted human-hand motions, underlined by a rigorous set of methodologies and promising numerical results. It lays essential groundwork for advancing the capability of robots to perform nuanced and intricate hand-based tasks.