SPIDER: Scalable Physics-Informed Dexterous Retargeting (2511.09484v1)

Abstract: Learning dexterous and agile policy for humanoid and dexterous hand control requires large-scale demonstrations, but collecting robot-specific data is prohibitively expensive. In contrast, abundant human motion data is readily available from motion capture, videos, and virtual reality, which could help address the data scarcity problem. However, due to the embodiment gap and missing dynamic information like force and torque, these demonstrations cannot be directly executed on robots. To bridge this gap, we propose Scalable Physics-Informed DExterous Retargeting (SPIDER), a physics-based retargeting framework to transform and augment kinematic-only human demonstrations to dynamically feasible robot trajectories at scale. Our key insight is that human demonstrations should provide global task structure and objective, while large-scale physics-based sampling with curriculum-style virtual contact guidance should refine trajectories to ensure dynamical feasibility and correct contact sequences. SPIDER scales across diverse 9 humanoid/dexterous hand embodiments and 6 datasets, improving success rates by 18% compared to standard sampling, while being 10X faster than reinforcement learning (RL) baselines, and enabling the generation of a 2.4M frames dynamic-feasible robot dataset for policy learning. As a universal physics-based retargeting method, SPIDER can work with diverse quality data and generate diverse and high-quality data to enable efficient policy learning with methods like RL.

• The paper introduces SPIDER, a framework that converts human-object interaction data into dynamically consistent robot control sequences with virtual contact guidance.
• It employs a sampling-based optimization with an annealed kernel and worst-case robustification to overcome noisy data and contact mode mismatches.
• Empirical results show improved success rates, near-real-time performance, and effective sim-to-real transfer across various robot morphologies and manipulation tasks.

SPIDER: Scalable Physics-Informed Dexterous Retargeting

Introduction and Motivation

SPIDER addresses the challenge of transferring human-object interaction data to dynamically feasible and physically robust robot trajectories, enabling efficient learning of dexterous manipulation and whole-body robot control without labor-intensive teleoperation or trajectory-specific reinforcement learning. The approach is motivated by the scalability constraints of current robot data collection, the utilization of abundant, but often noisy, human motion data, and the substantial embodiment gap between human and robotic morphologies. The core problem SPIDER addresses is converting kinematic human demonstrations into robot control sequences that are (i) dynamically consistent, (ii) preserve intended human contact modes, and (iii) generalize across disparate robot morphologies.

Figure 1: SPIDER converts human-object interaction trajectories into robot-trajectories through parallel physics simulation and virtual contact guidance, enabling scalable cross-embodiment demonstration transfer.

Problem Formulation and Architecture

The retargeting task is formalized as a constrained optimization problem:

• Given reference kinematic trajectories from human demonstrations, the algorithm seeks a control sequence $u_{0:T-1}$ minimizing state deviations and control effort, subject to the physical constraints of the target robot and environment dynamics.
• The cost landscape is highly non-convex and discontinuous due to contact-rich manipulation, compounded by noisy or underspecified demonstration data.

To solve this, SPIDER employs a sampling-based trajectory optimizer, leveraging a parallelized physics simulation backend supporting different robot forms (dexterous hand, humanoid) and simulators (MuJoCo Warp, Genesis, Isaac Gym) with efficient GPU batch rollouts.

Figure 2: The SPIDER pipeline processes reconstructed 3D human-object scenes and reference trajectories to generate robot-feasible motion augmented for deployment or policy learning.

The algorithm operates in several key stages:

• Preprocessing: Kinematic state estimation and inverse kinematics for initial alignment of robot and demonstration pose.
• Sampling-Based Optimization: An annealed kernel progressively shifts from global exploration to fine exploitation in trajectory space.
• Virtual Contact Guidance: Explicit virtual constraints maintain the demonstrated human-object contact sequence.
• Robustification: Worst-case (min-max) optimization across a domain-randomized set of physical parameters (e.g., friction, mass) ensures sim-to-real transfer.
• Physics-based Data Augmentation: Systematic variation in environment and object properties aids generalization for downstream learning.

  Figure 4: Contact mode mismatch solved by virtual guidance; SPIDER expands the feasible set towards human-desired contacts (right), avoiding local minima in contact mode (left, middle).

Virtual Contact Guidance

A critical innovation is "virtual contact guidance." Standard sampling-based approaches may optimize an object's global trajectory but not preserve the intended contact mode (e.g., holding a stick between specific fingers). To address ambiguity, SPIDER introduces virtual forces constraining hand-object contact points, activated and gradually relaxed according to a curriculum during optimization. This enlarges the basin of attraction in the solution space, favoring the human-demonstrated contact sequence.

• Contact indicators are computed dynamically, activating the constraint when the demonstration is in sustained contact.
• Constraints are filtered to tolerate demonstration noise—instability or short contacts are ignored to prevent biasing towards artifacts in human data.

Physics-Based Data Augmentation and Robustification

The physics-aware retargeting allows for diverse trajectory augmentation:

• Geometric variation: Changing object meshes, poses, and scales for robust learning.
• Physics variation: Simulating with variable inertial and contact parameters; e.g., retargeting humanoid walk on stairs vs. flat ground, or manipulating under external force constraints.

  Figure 6: Physics-based data augmentation generates diverse robot-object interactions and environmental responses from a single demonstration, improving generalization.

Robustification against model mismatch is achieved by optimizing control to minimize the worst-case cost over sampled parameter variations, ensuring feasibility across a range of physical realities (e.g., different friction or mass parameters).

Empirical Results and Comparative Analysis

Extensive evaluations were performed over six datasets and nine robot morphologies, including both dexterous hands (Allegro, XHand, Inspire, Ability, Schunk) and humanoids (Unitree, Fourier, Booster).

Ablation Studies

SPIDER's annealed kernel and virtual contact guidance significantly improve success rates (up to 18% absolute gain) over sampling-only baselines. The addition of contact guidance achieves the highest rates, confirming the necessity of explicit contact preservation. SPIDER also delivers near real-time trajectory generation speeds of 2.5–3.0 Hz.

Scaling and Generalization

Across 262 retargeted episodes and 2.4M frames, SPIDER maintains high success rates, particularly on datasets (e.g., GigaHands) focused on pick-and-place, with a noted drop in more contact-sensitive datasets (e.g., Oakink, ARCTIC) that depend on precise initializations.

Figure 8: Robot specification chart demonstrates the wide range of hand and humanoid robotic DoFs, sizes, and morphologies successfully supported by SPIDER's unified approach.

Figure 3: SPIDER grounds infeasible human demonstrations with physics, adapting them to different robot hands and their kinematic constraints.

Comparative studies with ManipTrans (RL-based) and DexMachina (functional optimization) reveal that while RL approaches can reach higher ultimate success on the most challenging dexterous tasks, they do so at the cost of an order-of-magnitude increase in computational effort per trajectory. SPIDER maintains competitive success rates coupled with vastly higher throughput, enabling effective large-scale dataset conversion.

Real-World and Sim2Real Experiments

SPIDER's robustified trajectories are directly deployed on physical systems (e.g., Franka Panda with Allegro hand) without additional adaptation, demonstrating reliable execution of dexterous tasks such as lightbulb rotation, spoon picking, and guitar playing.

Figure 5: Open-loop rollout of SPIDER-generated trajectories on physical robots, showing accurate transfer from simulated to real systems for complex single- and dual-arm manipulation tasks.

Humanoid Whole-Body Control

SPIDER generalizes to humanoid locomotion and object interaction, outperforming state-of-the-art kinematic retargeting in joint, position, and orientation tracking errors, while correcting artifacts like foot slippage and penetration unaddressed by kinematics-only baselines.

Applications and Extensions

Integration with RL Policy Learning

Physics-grounded trajectories generated by SPIDER facilitate downstream RL by providing feasible references and nominal controls, thus reducing the need for curriculum or reward engineering. Policies trained with SPIDER-tracked references converge faster and achieve higher task completion (object interactions) compared to training from raw, infeasible human data.

Real2Sim2Real Transfer

SPIDER robustly retargets from noisy 3D reconstructions of single-view RGB videos, correcting mesh penetration and contact artifacts, thus broadening the spectrum of usable demonstrations.

Figure 12: Noisy mesh reconstructions from monocular video are physically grounded and retargeted by SPIDER for direct robot execution.

Implementation and Performance Considerations

• SPIDER is compatible with any differentiable or sample-based physics simulator allowing parallel simulation.
• The core optimizer uses 50–100 Hz control, 1024 particles per batch, and up to 16 optimization iterations per trajectory, efficiently leveraging modern GPU architectures (tested on RTX 4090 and H100).
• The pipeline includes automated filtering and post-processing of demonstrations, contact detection heuristics, and support for geometric and physical domain randomization for data augmentation.

Limitations and Future Directions

• Success and trajectory fidelity are constrained by the quality of human-object 3D reconstructions; severe mesh or pose errors can still yield degraded behavior.
• The lack of explicit force or tactile feedback in most human datasets may limit the recovery of nuanced manipulative behaviors.
• Further augmentation of the virtual contact guidance for complex, multi-contact tasks and automatic task-specific hyperparameter tuning may unlock more challenging applications.

Conclusion

SPIDER represents a scalable, general, and efficient core for human-to-robot demonstration transfer, unifying the physical realism of optimization with the data efficiency of sample-based methods. Its competitive performance, high throughput, and embodiment-agnostic design make it a practical enabler for next-generation dexterous and general-purpose robot learning pipelines. Future progress will emerge from more robust perception-to-3D pipelines, tighter integration with foundation models for generalist policies, and the expanded use of physics-aware augmentation for broad, transferable robotics skills.