DexScrew Framework in Dexterous Manipulation

• DexScrew is a sim-to-real transfer framework that bridges RL, teleoperation, and behavior cloning to master contact-rich screwdriving and fastening tasks.
• It uses a three-stage pipeline—simulation with RL, skill-assisted teleoperation, and tactile behavior cloning—to improve robustness and generalization across diverse geometries.
• Key implementations include multisensory data integration and domain randomization, yielding superior performance over direct sim-to-real methods in realistic scenarios.

DexScrew is a sim-to-real transfer framework designed to address core limitations in dexterous robotic manipulation, focusing on contact- and perception-rich tasks such as nut-bolt fastening and screwdriving with multifingered hands. The framework unifies reinforcement learning (RL) on simplified object models with multisensory teleoperation and behavior cloning leveraging tactile feedback, resulting in robust, generalizable, and autonomous manipulation policies for complex real-world settings. The method is validated on diverse nut and screwdriver geometries and is shown to outperform direct sim-to-real RL approaches in both robustness and generalization (Hsieh et al., 1 Dec 2025).

1. Limitations in Dexterous Manipulation and Motivations

Dexterous manipulation remains constrained by several bottlenecks in both simulation fidelity and multisensory signal acquisition:

• Physics Simulation Gaps: High-fidelity modeling of contact-rich phenomena (e.g., thread engagement, slip, micro-collisions) is computationally prohibitive. Even massively-parallel domain randomization fails to capture critical dynamics, leading to non-robust sim-to-real policies.
• Sensing Gaps: While visual inputs are amenable to domain randomization, simulation of tactile signals remains unreliable, despite tactile feedback being essential for fine-force control and estimating contact phases.
• Teleoperation Bottlenecks: Human demonstration provides real multisensory sequences but scales poorly due to the complexity in mapping human to robotic hand kinematics and the time-intensive nature of data collection.

DexScrew’s core motivation is to leverage the strengths of both RL in simplified simulations and real-world teleoperation while circumventing their respective shortcomings by decomposing the learning process into three distinct stages.

2. Three-Stage DexScrew Pipeline

DexScrew comprises the following sequential pipeline to achieve robust sim-to-real transfer for dexterous manipulation:

1. Simulation RL with Simplified Objects: Policies are trained using rigid-body toy models (e.g., thick prism “nuts” or octagonal/dodecagonal screwdriver “handles” mounted on revolute joints) in IsaacGym. These abstractions maintain task-relevant axes while allowing for cheap, randomized, large-scale RL to induce correct finger gaits. Contact physics employ standard MuJoCo/PhysX models with randomized friction and mass, but omit explicit thread-to-thread forces.
2. Skill-Assisted Teleoperation: The RL policy is encapsulated as a "skill primitive" for hand motion. A human operator, via a Meta Quest 2 VR controller, is limited to controlling wrist/arm movement, triggering the hand primitive as needed. This separation allows efficient collection of real-world demonstrations with high-resolution tactile and proprioceptive feedback, delegating challenging gait patterns to the policy and operator focus to object alignment.
3. Behavior Cloning with Tactile Feedback: The demonstration trajectories are used to train a sensorimotor policy via supervised learning, which fuses tactile, proprioceptive, and temporal data. The result is a fully autonomous closed-loop controller capable of generalizing to unseen object geometries and recovering under perturbations.

3. Detailed Reinforcement Learning and Teleoperation Methodology

Simulation RL: MDP and Objective

• State Space: Robot proprioception and, for oracle training, privileged object state, yielding an embedding $z_t\inℝ^{97}$ (object pose, contact states, PD gains).
• Action Space: Commands for all 12 hand joints,

$$a_t^\mathrm{Hand} = η · f(o_t^\mathrm{RL}) + a_{t-1}^\mathrm{Hand},$$

where $η$ is an action scale.

• Reward Structure:

$$r_t = λ_\textrm{task}·r_t^\textrm{task} + λ_\textrm{energy}·r_t^\textrm{energy} + λ_\textrm{stability}·r_t^\textrm{stability}$$

with components including rotation rewards, proximity, torque penalties, pose penalties, and task-specific thresholds (full definitions in the original paper and its appendices).

• Learning Algorithm: PPO with surrogate loss,

$$L^\mathrm{PPO}(θ) = \mathbb{E}_t [\min(ρ_t(θ) \hat{A}_t, \mathrm{clip}(ρ_t(θ), 1±ε) \hat{A}_t)],$$

where $ρ_t = π_θ(a_t|s_t)/π_{θ_\textrm{old}}(a_t|s_t)$.

• Domain Randomization: Extensive randomization over object parameters (scale, mass, friction), disturbances, and control gains at every environment reset. Early termination upon significant finger drift, sustained object stagnation, or loss of contact.

Skill-Assisted Teleoperation

• Human Operator Interface: 6 DoF arm/wrist control via VR controller, while finger rotation is implemented by the sim-trained policy on demand.
• Data Collected: At 20 Hz, recording (1) hand actions, (2) arm actions from VR input, (3) full proprioceptive state, and (4) raw tactile data from XHand pressure arrays (5 fingers × 120 elements × 3 axes).
• Bootstrapping and Filtering: Only trajectories achieving high task progress are retained ("filtered BC"). No data augmentation beyond feature normalization.

4. Multisensory Behavior Cloning and Network Design

• Inputs: Past $K=5$ steps of concatenated proprioception ($q_{t-K+1:t}\in ℝ^{(12+6)\times K}$) and tactile data ($c_{t-K+1:t}\in ℝ^{5×120×3×K}$).
• Network Modules:

1. Tactile MLP: Extracts features from raw tactile input.
2. Feature Fusion: Tactile features are concatenated with flattened proprioception history.
3. Hourglass Encoder: Stacked-hourglass architecture to capture spatiotemporal relationships.
4. Action-Chunking Head: Predicts a future command sequence $\hat{a}_{t:t+H}\in ℝ^{(12+6)×H}$ for horizon $H=16$.

• Loss Function:

$$\mathcal{L}_\mathrm{BC} = \frac{1}{T}\sum_{t=1}^T \sum_{h=0}^H   \|\hat{a}_{t+h} - a_{t+h}\|_2^2$$

with observation normalization, weight decay $1\times10^{-4}$, and no additional regularization.

• Optimization: Adam optimizer, learning rate $1\times10^{-4}$, 200 epochs, batch size 64, and early stopping on validation loss. Dataset split: 80% training, 10% validation, 10% test, partitioned by object instance.

5. Empirical Results and Robustness Analysis

Performance Metrics

• Progress Ratio: $$(\textrm{rotations completed})/(\textrm{rotations required})\times100\%$$.
• Completion Time: Wall-clock seconds for fully completed (100%) trials.
• Reporting: Mean ± standard deviation over 10 trials per object.

Key Findings

• Nut-Bolt Fastening: The tactile+history BC policy achieved 96–98% success rates and 75–125 s completion times across all tested geometries (including previously unseen types). Tactile-only or history-only models showed significantly lower generalization, especially on irregular geometries, with failure to infer object shape or maintain stable contact.
• Screwdriving: Direct sim-to-real RL policies failed to achieve completion (41.6% progress), while behavior cloning with tactile and history achieved the highest progress (95.0%) and fastest completion (188 s).
• Robustness: The BC policy with both tactile and history demonstrated the ability to re-establish correct contact and motion under external disturbance, outperforming open-loop or non-tactile models which suffered drift, loss of contact, or task failure.
• Statistical Significance: All improvements are statistically significant by t-test at the $p < 0.05$ level ($>2σ$).

Limitations

• Teleoperation data collection is a limiting factor (∼200 min per 122 trajectories).
• The system assumes pre-inserted objects; integrated pick-and-place plus insertion is not addressed.
• No use of vision; occlusion or complex scenes pose challenges for wrist alignment.

6. Implementation, Hardware, and Resources

• Hardware: UR5e (6 DoF) arm, XHand with 12 DoF and high-density tactile arrays, Meta Quest 2 VR interface (60 Hz tracking, 20 Hz control).
• Calibration: Tactile arrays zeroed and scaled with reference weights; joint encoders calibrated via URScript back-drives.
• Software: IsaacGym v0.6.0 (simulation), PyTorch 1.12 (RL, BC), ROS Noetic (teleop), ZeroMQ (VR streaming), and PyTorch/Hydra (BC training).
• Codebase and Reproducibility: Full source code, pretrained models, demonstration datasets, Docker images, and automated scripts are available at https://github.com/dexscrew-project/dexscrew. Quickstart scripts and detailed hyperparameter settings support faithful replication and extension.

Component	Specification / Setting	Source (Section)
Simulation Envs	8192 parallel, 200Hz sim, 20Hz control	§2.4
Hand/Arm	XHand 12DoF, UR5e 6DoF, VR control	§6.1
PPO Hyperparameters	rollout 12, minibatch 16,384, lr=$5\times10^{-3}$	§2.4, Appendix Table IV
BC Training	Adam, lr=$1\times10^{-4}$, 200 epochs, batch size 64	§4.3

7. Significance and Extensions

DexScrew demonstrates that combining sim-trained RL skill primitives, human-in-the-loop teleoperation, and multisensory behavior cloning produces manipulation policies robust to contact uncertainties and geometric variation, outperforming direct sim-to-real approaches especially in the presence of tactile feedback and temporal recurrency. The framework advances the field of dexterous manipulation by providing standardized benchmarks, transparent code, and a reproducible experimental protocol for multisensory sim-to-real transfer (Hsieh et al., 1 Dec 2025). A plausible implication is that integrating vision as an additional sensing modality and scaling semi-autonomous teleoperation pipelines could further expand applicability to long-horizon and cluttered scenes.