DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model (2510.08556v1)

Abstract: Achieving generalized in-hand object rotation remains a significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a "reality gap" that has limited prior work to constrained scenarios involving simple geometries, limited object sizes and aspect ratios, constrained wrist poses, or customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, to generalize to a wide variety of objects and conditions in the real world. The core of our method is a joint-wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy's actions accordingly. The model is highly data-efficient and generalizable across different whole-hand interaction distributions by factorizing dynamics across joints, compressing system-wide influences into low-dimensional variables, and learning each joint's evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-world interaction data with minimal human intervention. Our complete pipeline demonstrates unprecedented generality: a single policy successfully rotates challenging objects with complex shapes (e.g., animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and rotation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach. Website: https://meowuu7.github.io/DexNDM/

• The paper introduces a joint-wise neural dynamics model that decomposes hand–object interactions to close the sim-to-real gap.
• It utilizes a two-stage specialist-to-generalist training pipeline with autonomous data collection to enhance performance on diverse objects.
• Experimental results show up to 81% improvement over baselines, confirming superior sample efficiency and in-air rotation stability.

DexNDM: Joint-Wise Neural Dynamics for Sim-to-Real Dexterous In-Hand Rotation

Introduction and Motivation

Dexterous in-hand object rotation is a canonical challenge in robotics, requiring robust control under complex, contact-rich dynamics and diverse object geometries. The sim-to-real gap—stemming from mismatched physical parameters, unmodeled effects, and noisy state estimation—has historically constrained policy transfer to the real world, especially for general-purpose manipulation. Prior works are limited by object complexity, size, wrist orientation, or require specialized hardware. DexNDM introduces a sim-to-real framework that leverages a joint-wise neural dynamics model and autonomous data collection to enable a single policy to generalize across a wide spectrum of objects and conditions.

Figure 1: DexNDM enables in-hand rotation of challenging objects with diverse geometries and wrist orientations, supporting teleoperation applications.

Methodology

Specialist-to-Generalist Policy Training

DexNDM adopts a two-stage policy training pipeline:

1. Category-Specific RL Specialists: Train object-category-specific policies using PPO in simulation, with observations including proprioception, fingertip/object states, force/contact signals, wrist orientation, and target rotation axis. The reward combines rotation, goal-pose guidance, and penalties for off-axis motion and excessive torque.
2. Generalist Distillation via Behavior Cloning: Aggregate successful trajectories from all specialists and train a unified generalist policy via supervised learning. This approach avoids the instability of DAgger-style distillation in high-difficulty tasks.

Joint-Wise Neural Dynamics Model

To bridge the sim-to-real gap, DexNDM factorizes the hand-object system by learning per-joint dynamics from each joint's own state-action history, rather than modeling the entire hand-object system. This design:

• Information Bottleneck: Projects high-dimensional system-wide influences (inter-joint coupling, actuation, object loads) into low-dimensional, task-sufficient net effects.
• Sample Efficiency and Generalization: Theoretical analysis (KL contraction via data processing inequality) and empirical results show improved generalization under distribution shift and superior sample efficiency compared to whole-hand models.

Autonomous Data Collection

DexNDM introduces a scalable, fully autonomous data collection strategy ("Chaos Box"):

• Replay policy actions in a container of soft balls, imposing randomized loads and broad coverage.
• Add Gaussian noise to actions to further expand distributional coverage.
• Eliminates catastrophic failures and human resets, enabling large-scale, diverse real-world data acquisition.

Residual Policy for Sim-to-Real Transfer

A residual policy is trained to compensate for the sim-to-real dynamics gap, using the learned joint-wise dynamics model to predict next-state transitions and output action corrections. This is solved via supervised learning on the same trajectory dataset as the base policy.

Figure 2: Method overview: RL specialists, generalist distillation, autonomous data collection, joint-wise dynamics learning, and residual policy deployment.

Experimental Results

Simulation

• The generalist policy outperforms re-implemented AnyRotate by 37–81% on unseen objects and axes.
• Rotation along gravity ($\pm z$) is easiest; generalist policy achieves high rotation reward and time-to-fall across all axes.

Real-World

• DexNDM achieves unprecedented in-air rotation of high-aspect-ratio (up to 5.33), small (down to 2–3 cm), and complex-shaped objects under diverse wrist orientations.
• In downward-facing hand configuration, DexNDM is the first to rotate long objects (10–16 cm) around their long axis for nearly a full circle in the air.
• Outperforms AnyRotate and Visual Dexterity on replicable objects and complex shapes, with superior survival angles and rotation stability.

  Figure 3: Real-world results: rotating challenging objects in the air.

  

  Figure 4: Diverse wrist orientations supported by the policy.

Model Comparison

• Joint-Wise vs. Whole-Hand Dynamics: Joint-wise model matches whole-hand expressivity in high-data regimes, but is significantly more sample-efficient and generalizes better under distribution shift and low-data settings.
• ASAP/UAN Baselines: ASAP and UAN, trained on free-hand data, fail to generalize to object-loaded manipulation; DexNDM's joint-wise model is robust to imperfect data and distributional mismatch.

  Figure 5: Joint-wise dynamics model achieves superior sample efficiency and generalization compared to whole-hand models.

Data Collection and Scaling

• Autonomous data collection is orders of magnitude more efficient than task-relevant or vision-based approaches, which are slow, intervention-heavy, and noisy.
• Performance scales with dataset size; extrapolation suggests task-relevant data would require millions of trajectories to match DexNDM's results with 4,000 autonomous trajectories.

  Figure 6: Analysis of data collection strategies: time efficiency, model performance, and scaling with dataset size.

Teleoperation Application

• DexNDM enables a VR-based teleoperation system (Meta Quest 3) for complex, long-horizon dexterous tasks, such as tool use and assembly.

  Figure 7: Teleoperation system performing complex manipulation tasks.

Ablation Studies

• Ablations confirm the necessity of joint-wise modeling, simulation pretraining, noise injection, object-loaded data, and policy-based replay for generalization and real-world performance.
• Direct fine-tuning of the base policy is unstable; residual policy compensation is robust and effective.

  Figure 8: Ablation paper: joint-wise model design choices and their impact on generalization error and real-world performance.

Implementation Considerations

• Computational Requirements: Training the joint-wise dynamics model and residual policy requires moderate GPU resources (e.g., 8×A10 GPUs for 2 epochs, ~2 days).
• Deployment: The generalist policy and residual compensator can be deployed on standard anthropomorphic hands (e.g., LEAP, Allegro) with torque control at 20 Hz.
• Data Collection: Autonomous strategies are critical for scaling; vision-based pose estimation is impractical for small, occluded, or axis-symmetric objects.
• Limitations: Ceiling is restricted by partial observations; integrating richer signals (e.g., tactile) and joint hand-object modeling are promising future directions.

Implications and Future Directions

DexNDM demonstrates that factorized, joint-wise neural dynamics models, paired with scalable autonomous data collection, can close the sim-to-real gap for dexterous manipulation. This approach generalizes across object complexity, size, and wrist orientation, enabling robust real-world deployment and teleoperation. Theoretical analysis supports the generalization benefits of information contraction, and empirical results validate sample efficiency and robustness.

Future research should explore:

• Integration of tactile sensing and richer observation modalities.
• Extension to bimanual and multi-object manipulation.
• Joint modeling of hand-object dynamics for further expressivity.
• Application to other contact-rich domains (e.g., legged locomotion, tool use).

Conclusion

DexNDM provides a principled, scalable framework for sim-to-real transfer in dexterous in-hand rotation, achieving generality and robustness previously unattainable. Its joint-wise neural dynamics model and autonomous data collection strategy set a new standard for real-world dexterous manipulation, with broad implications for embodied AI and robotics.