Dexterous In-Hand Manipulation

• Dexterous in-hand manipulation is the dynamic repositioning and reorienting of objects within a robot’s hand using multifinger coordination, complex contact modeling, and compliant hardware.
• It employs analytical modeling, discrete-continuous planning (e.g., DMG, MPC), and optimization-based controllers to achieve precise manipulation under frictional and geometric constraints.
• Recent advancements integrate reinforcement learning, imitation learning, and diffusion policies to robustly manage contact dynamics, improve sim-to-real transfer, and enhance task performance.

Dexterous in-hand manipulation encompasses the class of robotic behaviors in which an object is dynamically reconfigured within a robot’s hand or between robot effectors by leveraging high degrees of kinematic freedom, contact-rich interaction, and complex control policies. Unlike simple grasping, dexterous in-hand manipulation aims for in-grasp repositioning, reorienting, regrasping, or performing functional tasks (e.g., unscrewing a cap, solving a Rubik’s cube) using multifingered, compliant, or dual-arm robotic platforms. The research landscape spans analytical modeling of contact dynamics, sampling- and optimization-based planners, imitation- and reinforcement-learning approaches, and hybrid strategies integrating human demonstrations, tactile feedback, and vision.

1. Fundamental Principles and Modeling

A defining element of dexterous in-hand manipulation is the intricate coordination of multiple contact points—generated by fingers, palms, or dual arms—subject to frictional and geometric constraints. Early frameworks formalize contacts as points, lines, or patches, each characterized by local contact frames and limit surfaces capturing frictional wrench admissibility. Recent models extend to dual-patch frictional contacts, as in the dual-limit-surface framework, where each contact is represented by an ellipsoidal limit surface

$$w^\top A w = 1, \quad A = \operatorname{diag}\big[ (\mu N)^{-2}, (\mu N)^{-2}, (\mu c r N)^{-2} \big]$$

and the total set of feasible contact wrenches is the Minkowski sum of two such surfaces. Critical to these models is the alternating regime of sliding versus sticking: at each time, either one contact slides (w on the limit surface) and the other sticks (w strictly inside the surface), with complementarity constraints enforcing transitions (Dang et al., 2024).

In multi-finger and underactuated hands, models further address the coupling between joint poses, force distributions, object pose, and the dynamical impact of contact breaking or reattachment (“finger gaiting”). Analytical approaches—such as those introduced by the Dexterous Manipulation Graph (DMG)—abstract the state space as a discrete graph of fingertip contact positions and orientations, enabling formal planning over allowable in-hand motion primitives (Cruciani et al., 2018, Cruciani et al., 2019).

2. Planning, Control, and Stability

High-level in-hand manipulation planning typically operates in a hybrid discrete-continuous space, with primitives such as sliding, pivoting, rolling, and regrasping forming the building blocks for complex task synthesis. In the DMG paradigm, nodes encode pairs of (contact position, admissible angular interval), and edges encode whether a feasible primitive (translation/rotation) connects these configurations without collision or violation of constraints (Cruciani et al., 2018, Cruciani et al., 2019). Graph search algorithms, e.g., Dijkstra or A*, minimize path cost (distance, rotation, gripper opening) and can be extended to dual-arm settings for coordinated pushing and handover.

Optimization-based methods, particularly those using model predictive control (MPC), formulate the closed-loop selection of controls as constrained trajectory optimization under nonlinear kinematics, contact complementarity, and stability constraints. The dual-limit-surface method encodes sliding/sticking regimes as second-order cone (SOC) constraints, alternating the “active” contact via parity logic. Robustness to model errors and slip is provided by safety margins in sticking constraints and receding horizon replanning (Dang et al., 2024).

Contact-implicit MPC further advances this by embedding a differentiable, smoothed quasi-dynamic complementarity model into the planning loop. Here, contact sequences and breakages are not explicitly programmed; instead, a smoothed DDP-based planner selects controls and contact modes adaptively, with a compliant low-level feedback controller enforcing generated force profiles (Jiang et al., 2024).

3. Learning, Imitation, and Diffusion-Based Approaches

Recent progress in dexterous in-hand manipulation is marked by a shift toward data-driven methods, leveraging reinforcement learning (RL), imitation learning, and diffusion policy models. End-to-end RL systems, including those employing distributed PPO or SAC (with robustified observation and action spaces), have been shown to learn highly dexterous behaviors such as finger gaiting, regrasping, and object stabilization, contingent on heavy domain randomization and massive simulated experience (OpenAI et al., 2018, Khandate et al., 2023). The OpenAI paradigm demonstrates that policies trained with randomized physics and vision effectively transfer to real hardware, with emergent behaviors like finger gaiting, multi-finger coordination, and controlled pivoting (OpenAI et al., 2018).

Imitation-based strategies employ teleoperation or AR interfaces to collect expert demonstrations, subsequently using motion retargeting and outlier filtering (e.g., HDBSCAN + GLOSH) to ensure high-quality training sets (Koczy et al., 4 Mar 2025). Diffusion policies model the sequence of target actions as samples from a noising–denoising process conditioned on proprioception, effort, and vision, yielding robust and generalizable closed-loop controllers for multifinger tasks such as cap unscrewing.

Hybrid methods incorporate motion primitive dictionaries built from human demonstrations. Trajectories are synthesized by solving convex combinations of primitives to achieve intended fingertip and object poses, with human priors implicitly encoding stability, contact, and collision constraints. Optimization is performed over the primitive weights under velocity and reachability constraints, resulting in rapid planning (sub-second) and implicit satisfaction of manipulation constraints (Hammoud et al., 2024, Hammoud et al., 2023).

4. Hardware, Compliance, and Sensor Integration

Hardware realization spans rigid multi-DOF anthropomorphic hands, soft pneumatic hands, articulated palms, and dual-arm/gripper setups. Compliant hands such as the RBO Hand 3 (Puhlmann et al., 2022) and dexterous soft hands (Sieler et al., 2023) exploit material compliance and high DOF actuation to absorb contact uncertainty and enable robust in-hand manipulation via real-time linear feedback control in the deformation space. These hands demonstrate funnel-based strategies—passive mechanical constraints that stabilize manipulation trajectories over wide state space regions.

Active tactile sensing, including high-resolution depth sensors (e.g., GelSight), is increasingly integrated for in-hand pose estimation, DLO following, and slip detection (Yu et al., 2024, Hu et al., 2023). Real-time tactile feedback enables robust maintenance of grasp in the presence of external disturbances and nonrigid object properties.

The design of the hand substantially affects performance. Open-source, cost-optimized hands such as ISyHand demonstrate that explicit palm articulation boosts early dexterity and learning rates in RL-based reorientation tasks, converging to or surpassing classical rigid hands (Richardson et al., 30 Sep 2025).

5. Performance, Benchmarks, and Empirical Results

Empirical assessment uses metrics such as pose tracking RMSE (position and orientation), task success rate (e.g., Rubik’s Cube solution rate, cap unscrewing completion), repeatability (inter-trial standard deviation), and manipulation stability (slip avoidance). Notably, introducing friction-consistent SOC constraints and mode-switching planners reduces position RMSE from ~25 mm to sub-10 mm and rotational errors from >50° to <5° in real robotic experiments (Dang et al., 2024).

RL-based cube reorientation achieves up to 70+ consecutive successful manipulations post-convergence, with domain randomization deemed indispensable for sim-to-real transfer (OpenAI et al., 2018, Richardson et al., 30 Sep 2025). Visuomotor diffusion policies reach 70–85% completion rates (task-dependent) with properly filtered demonstration sets (Koczy et al., 4 Mar 2025). Hybrid dictionary-optimization approaches rapidly generate pose-accurate in-hand trajectories with median positional errors ≲3 mm and high contact stability, far outpacing classical model-based planners in speed (Hammoud et al., 2024, Hammoud et al., 2023).

6. Challenges, Failure Modes, and Outlook

Persistent challenges include:

• Real-time enforcement of non-convex, high-dimensional collision constraints among fingers and object, addressed by NN-accelerated collision distance fields embedded in sampling/planning algorithms (Gao et al., 2023).
• Drift and stability in open-loop planners, necessitating periodic closed-loop correction or tactile/visual feedback (Dang et al., 2024, Yu et al., 2024).
• Generalization to novel objects, shapes, and manipulation tasks, particularly when relying on narrow demonstration distributions or overfitting to few specialized objects (Koczy et al., 4 Mar 2025, Zhang et al., 28 Feb 2026).
• Sim-to-real discrepancies due to unmodeled frictional, compliance, or object-dependent properties, mitigated by aggressive domain randomization, calibration, and data-driven parameter identification (OpenAI et al., 2018, Hu et al., 2023).

Advances in unified vision–language–action models now allow open-vocabulary, morphology-agnostic policy generation for in-hand manipulation, with discrete codebooks spanning multiple hand designs and strong physical prior enforcement via physics-guided dynamic refinement (Zhang et al., 28 Feb 2026). The integration of rich tactile feedback, dynamic adaptation (self-identification), and learned constraint priors continues to define the leading edge of dexterous in-hand manipulation research.

7. Representative Approaches: Comparative Overview

Approach	Key Methodology	Quantitative Performance/Features
Dual Limit Surface Planner	SOC-constrained mode switching	Real-world RMSE: 4.8 mm, 1.26–4.59° (rot)
Contact-Implicit MPC [2402]	DDP/NLP + contact complementarity	0.3 rad/s, slippage 3.26×10⁻³, robust recovery
RL + Domain Randomization	Massive-scale distributed PPO	70+ reorientations, 90.3% Rubik's Cube SR
Primitive Dictionary Opt.	NMF+QP over human-demonstrated	1.2 mm per-finger error, 97% contact stability
Soft Hand Linear Feedback	Online Jacobian learning, strain	100% success, adapts to actuator failure
Visuomotor Diffusion [2503]	Demo filtering, CNN-diffusion	70–85% task success (unscrewing)

The diversity of approaches reflects the inherent complexity and interdisciplinarity of the field, with the most robust systems integrating model-based planning, learning from experience, compliance, and rich sensory feedback. The field continues to advance toward unified, adaptive solutions with real-world generalizability and physical safety guarantees.