Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

Published 13 Jul 2026 in cs.RO, cs.AI, and cs.LG | (2607.11874v1)

Abstract: Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.

Summary

  • The paper presents REGRIND, a minimalist pipeline that uses interaction-preserving motion retargeting and residual RL to learn dexterous manipulation from one human demonstration.
  • It achieves near-perfect simulation success rates (>98%) with keypoint tracking errors of 5-7 mm, and robust real-world performance with errors of 10-28 mm.
  • The approach simplifies reward engineering and enhances policy exploration by combining reference-guided actions, domain randomization, and task-specific curricula.

Minimalist Retargeting-Guided Reinforcement Learning for Dexterous Manipulation

Overview of Problem and Approach

The paper presents REGRIND, a minimalist retargeting-guided reinforcement learning (RL) pipeline for dexterous manipulation using anthropomorphic robot hands. Unlike traditional teleoperation-driven methods, REGRIND achieves policy learning from a single human demonstration, leveraging interaction-preserving motion retargeting and simulation-based residual RL to efficiently cross the human-robot embodiment gap. This approach is motivated by recent successes in humanoid whole-body control, but adapts the methodology for the nuanced, contact-rich dynamics in manipulation tasks.

Interaction-Preserving Motion Retargeting

REGRIND's key innovation lies in its retargeting process, where human demonstrations comprising detailed hand-object motions are mapped onto robotic hands through the construction and optimization of interaction meshes. Spatial and contact relationships between the hand and object are maintained by minimizing mesh deformation, as measured through Laplacian coordinates and temporal smoothness constraints. This ensures physically plausible reference trajectories critical for downstream RL, thereby mitigating artifacts associated with pure kinematic retargeting that fail to respect object contact semantics.

Reference-Guided Residual Reinforcement Learning

The RL formulation utilizes the retargeted trajectory as both motion prior and state distribution for exploration. Policy learning employs a residual action space, where the trained policy modulates reference actions with task-relevant corrections. Observations are carefully structured: actor uses reliable state-based information (joint positions, object pose, previous actions, motion phase), while the critic additionally receives fingertip positions and velocities. The reward function focuses on keypoint-based object tracking, shaped with exponential proximity kernels and augmented with velocity and wrist tracking, plus action magnitude and rate regularization.

Episode initialization leverages reference state sampling (RSI), augmented by dynamic perturbations of the initial object and hand pose, thus enabling broad task coverage and generalization from a single demonstration. Domain randomization (friction, mass, time lag, geometry) and curricula (gravity ramp, stochastic pushes) robustify the policy against sim-to-real discrepancies.

Empirical Evaluation

REGRIND is evaluated on four contact-rich manipulation tasks—scissors and screwdriver operations—across two multi-fingered hands (LEAP, WUJI). Baselines include physics-based retargeting (SPIDER), kinematic retargeting/RL pipelines (DexMachina, Mink IK+RL). On simulated tasks, REGRIND achieves a mean object keypoint tracking error of ~5-7 mm with near-perfect success rates (>98% on all tasks), whereas SPIDER and Mink IK+RL baselines exhibit failure rates (success = 0%), and DexMachina's performance degrades on scissors tasks due to unstable contact modeling.

Real-world deployment shows robust sim-to-real transfer by REGRIND on three tasks, with tracking errors only ~10-28 mm, and success rates of 90-100%. DexMachina-based policies, despite strong simulation performance, fail to transfer effectively on screwdriver tasks, likely due to lack of interaction constraints in retargeting resulting in unrealistic behavior. None of the methods succeeded on WUJI-Scissors, reflecting hardware and modeling limitations.

Generalization experiments demonstrate that REGRIND-trained policies maintain high performance when tested on randomly perturbed object initial conditions. The augmentation strategy allows for spatial robustness without additional demonstrations.

Practical and Theoretical Implications

The results highlight the necessity of interaction-preserving retargeting for RL-based dexterous manipulation, particularly for contact-rich tasks where small sim-to-real gaps compound rapidly. The reference-motion plus residual RL paradigm substantially simplifies reward engineering, nudges exploration toward human-like strategies, and enables rapid policy acquisition from minimal data.

From a theoretical perspective, this work clarifies the distinction between locomotion and manipulation in sim-to-real RL: manipulation demands careful observation/action design, domain randomization, and accurate system identification due to high sensitivity to contact and friction parameters. The embodiment gap is successfully managed via mesh-based metrics, suggesting general applicability beyond the examined tasks.

Practically, REGRIND provides a reproducible pipeline for policy learning on anthropomorphic hands without extensive real-robot data collection or pre-existing robot-specific demonstrations. However, reliance on motion capture for object state and the need for system identification remain bottlenecks for in-the-wild applicability. Future directions include vision-based policy distillation and online adaptation to hardware/scene uncertainties.

Conclusion

REGRIND establishes a minimalist, retargeting-guided RL recipe for dexterous manipulation, yielding high-fidelity policies from minimal human data and robust sim-to-real transfer. The outcomes underscore the critical role of interaction-aware retargeting, trajectory augmentation, and environment randomization in RL for contact-rich manipulation. The framework's modularity and empirical robustness provide a solid foundation for further exploration and deployment of dexterous robotic manipulation systems (2607.11874).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

A simple explanation of “A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation”

What this paper is about

This paper shows a simple way to teach robot hands to use tools (like scissors and screwdrivers) by learning from just one example of a human doing the task. The method is called REGRIND. It first turns a human’s hand-and-object motion into a “robot version” that keeps the important contact and grip details, then trains the robot in a computer simulation to follow that motion, and finally runs the learned skill on a real robot hand.

What questions the researchers asked

The authors focused on four easy-to-understand questions:

  • Does it help to copy not just the shape of the human hand pose, but also how the hand and the object touch and move together?
  • If we preserve those hand–object interactions, does a robot learn faster and better?
  • Can a policy (a learned “how to act” strategy) trained in simulation work on a real robot without more training?
  • Can the robot do the same task even when the starting position is a bit different from the demonstration?

How they did it (in everyday terms)

The method has four main steps:

  1. Collect a human demonstration A person shows how to do a tool-use task (like picking up scissors and cutting). Cameras and tracking systems record where the hand and the object are in 3D over time.
  2. Retarget the motion to the robot while preserving interaction “Retargeting” means translating human hand motion into robot hand motion. Instead of only matching finger poses, the method preserves how the hand and object relate—where the fingers touch, how they’re spaced, and how the object is held.
  • Think of it like copying a dance: rather than copying just how the dancer’s arms bend, you also copy how the dancer interacts with a prop, so the prop moves correctly.
  • To do this, the authors use an “interaction mesh,” a web of key points on both the hand and the object. The robot tries to move so this web changes as little as possible from frame to frame, which keeps the grasp and contacts human-like and physically sensible.
  1. Train a policy in simulation with reinforcement learning (RL) Reinforcement learning is like trial-and-error practice in a video game. The robot gets a “guide route” (the retargeted motion) and learns to follow it while making small corrections.
  • Residual policy: Imagine you have GPS directions (the guide route) and you nudge the steering wheel a bit to stay on track. The policy learns those small “nudge” corrections rather than the entire motion from scratch.
  • Object-centric tracking: Instead of only matching the robot’s finger angles, the robot is rewarded for keeping specific points on the object where they should be. This helps with tools of different shapes.
  • Reference state initialization: Training episodes start at random points along the guide route so the robot practices many parts of the task, not just from the very beginning.
  • Data augmentation: Because there’s only one demo, they “shake things up” by slightly changing the starting position and orientation of the object and warping the early part of the guide route to match. This creates many realistic variations that improve generalization.
  • Domain randomization and curricula: During training, they randomly vary friction and other physics settings and gradually make the task harder (like adding pushes or increasing gravity). This makes the robot more robust to real-world differences.
  1. Deploy to the real robot After training, they run the policy “zero-shot,” meaning no extra training on the real robot. They do careful “system identification” first—basically measuring real robot properties so the simulator matches reality closely. For deployment, a motion capture system provides the object’s position to reduce vision errors.

What they found and why it matters

  • Interaction-preserving retargeting works better: When the robot’s reference motion keeps the correct hand–object contact patterns, the learned policies are more stable and realistic. Baseline methods that just match hand poses (but ignore object contact) often led to bad grasps or finger-object collisions, making learning harder.
  • Strong simulation results: Their method (REGRIND) achieved very high success rates and low tracking errors in simulation on four challenging tasks that involve continuous contact and friction: using scissors and turning a screwdriver, each tested on two different robot hands.
  • Real-world transfer on most tasks: The policies trained with REGRIND worked on real robot hands without extra training for three out of four task-hand combinations. A competing method that seemed good in simulation often failed on real hardware, likely because it learned to exploit simulation quirks and didn’t preserve real contact patterns as well.
  • Generalization from one demo: Even though they used only one human demonstration, the robot could handle changes in the starting position and angle of the object thanks to the trajectory warping and training-time randomization. It didn’t just memorize the demo—it adapted.
  • Important caveat: One setting (scissors with a particular hand) still didn’t transfer well due to bigger differences between the simulator and the real world (for example, motor behavior and imperfect object models). This highlights how contact-rich hand tasks are sensitive to small physical mismatches.

Why this work is important

  • Fewer demonstrations needed: Learning from a single human example is much cheaper and faster than collecting lots of robot-specific data.
  • More natural, human-like grasps: By preserving contact and interaction structure, the robot uses tools in a fluid, human-like way, which tends to be physically stable and effective.
  • Practical recipe for sim-to-real: The paper gives a simple, reusable recipe: preserve interactions when retargeting, learn residual corrections while tracking object keypoints, augment starting states, randomize physics, and carefully match real-world robot settings. This helps others build robust dexterous skills.

What this could lead to next

  • Vision-based deployment: Right now, they use motion capture for the object pose on the real robot. A next step is to replace that with a camera-based system, so the robot can work in everyday environments.
  • Adapting on the fly: The robot could learn to adjust itself during real use (test-time adaptation) to handle different tools, wear and tear, or changing friction.
  • Broader tool-use skills: With this approach, robots could learn many contact-rich tasks—opening packages, using kitchen tools, or helping with daily activities—by watching a few human demos and practicing safely in simulation.
View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a concise list of concrete gaps and open questions that remain unresolved and could guide future research:

  • Perception dependence: The approach requires motion capture for object pose at deployment; it is unclear how performance degrades with vision-only sensing, occlusions, markerless tracking, or noisy pose estimates, and how to best distill the state-based policy into a robust vision/tactile policy.
  • Limited task coverage: Only two single-hand tool-use tasks (scissors, screwdriver) on two hands were evaluated; generalization to broader tool categories, unseen objects, in-hand dexterity, articulated and deformable objects, and bimanual tasks remains untested.
  • Cross-instance and cross-category generalization: Policies were not evaluated on different instances within a category (e.g., various screwdriver sizes/materials) or across categories; methods for category-level skills from a single demonstration (e.g., CAD-parametric conditioning) are missing.
  • Reliance on accurate geometry and physics: Success was sensitive to object meshes, friction, and actuation properties (e.g., WUJI-Scissors failure); there is no automatic system identification pipeline to infer geometry, friction, compliance, and actuator dynamics from data.
  • Actuation and control assumptions: The policy uses position control with PD tracking; the effect of torque/impedance control, non-backdrivable vs backdrivable actuators, and compliance on sim-to-real transfer and grasp stability is not analyzed.
  • Interaction-preserving retargeting design: The method depends on manually defined hand–robot keypoint correspondences and object keypoint sampling; robustness to correspondence errors, keypoint density/placement, and noisy human keypoints is unstudied, and automatic correspondence learning is absent.
  • Dynamics-ignorant retargeting: Retargeting is solved per frame with kinematic constraints and smoothness but no dynamics/contact feasibility; the benefits and costs of dynamics-aware or closed-loop retargeting objectives remain open.
  • Contact modeling fidelity: The reward does not explicitly regulate contact modes or forces; it is unknown whether adding tactile/force objectives or contact-stability terms improves robustness and sim-to-real transfer in friction-critical tasks.
  • Data augmentation scope: Augmentation applies small, planar SE(2) warps with unchanged finger joints and no re-solved contact; the limits before contact semantics break, the value of full 3D warps (including z and non-upright orientations), and adaptive re-retargeting for large perturbations are not explored.
  • Exploration and recovery: Early termination on object deviation and strict reference tracking bias learning toward a single strategy; how to train robust recovery behaviors, tolerate large disturbances, or discover alternative valid strategies remains open.
  • Observation and action design: The actor avoids velocities and force/tactile inputs; the trade-off between sim-to-real robustness and control performance when adding velocity estimates, tactile signals, or richer proprioception is not quantified.
  • Domain randomization and curricula: DR ranges and curricula were hand-tuned; there is no sensitivity analysis or automatic DR selection (e.g., AutoDR/BO), and robustness to larger time delays and unmodeled latencies is not assessed.
  • Failure analysis and adaptation: The root causes of WUJI-Scissors failure (non-backdrivability, friction, geometry errors) are hypothesized but not systematically ablated; the effectiveness of test-time adaptation (e.g., rapid motor adaptation, online system-ID, residual on-robot fine-tuning) is untested.
  • Multi-contact scaling: It is unknown how the interaction-mesh approach scales to multi-hand, multi-object, and environment contacts, or tasks requiring complex regrasping and contact switching.
  • Human-likeness quantification: Claims of “human-like behavior” are not supported by metrics; whether quantitative human-likeness (e.g., pose/contact pattern similarity) correlates with robustness and sim-to-real success is an open question.
  • Safety and compliance in the real world: Impact forces, pinch hazards, and wear were not measured; methods to integrate safety constraints, soft-contact objectives, or real-time safety monitors are not addressed.
  • Computational efficiency and scalability: The wall-clock cost of per-frame SQP retargeting is not reported; feasibility for large datasets, online use, or rapid adaptation to new objects is unclear.
  • Multiple demonstrations and inter-subject variability: The pipeline uses a single demonstration; how to integrate multiple demos, handle stylistic variability, or learn style-agnostic yet robust policies is unexplored.
  • Perception–policy bridging: A concrete recipe for distilling from mocap-state policies to vision policies (data requirements, supervision signals, alignment of coordinate frames, and handling occlusion) is missing.
  • Mapping to new robot hands: Transferring to arbitrary hand morphologies currently requires manual keypoint definition; automated learning of human–robot correspondences and adaptation across diverse hand kinematics is an open problem.
  • Long-horizon autonomy: Robustness over extended horizons, repeated cycles (e.g., prolonged screwing), and failure recovery strategies in the wild were not evaluated.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are actionable use cases that can be deployed now in controlled or semi-structured environments, leveraging the released code and the paper’s validated pipeline.

  • Rapid “one-demo” skill programming for dexterous robot hands in manufacturing cells
    • Sector: robotics, manufacturing, electronics assembly
    • What: Teach screw turning, tool actuation (e.g., scissors-like clamping tools), and similar contact-rich tasks from a single human demonstration; deploy on UR5e + anthropomorphic hand (e.g., LEAP hand or similar form factor).
    • Tools/products/workflows:
    • REGRIND-based skill authoring workflow: capture one human hand–object motion with mocap → interaction-preserving retargeting → residual RL training in Isaac Sim with RSI and augmentation → system identification → deploy.
    • “Skill Packs” for common tool-use tasks (screwdriving, cutting, clamping) pre-trained and parameterized by object pose.
    • Assumptions/dependencies:
    • Access to reliable object pose via motion capture at deployment (as used in the paper).
    • Accurate object meshes and calibrated friction/compliance; careful system identification.
    • Anthropomorphic multi-fingered hand with similar kinematics and a PD control interface.
  • Fast prototyping and benchmarking for hand hardware developers
    • Sector: robotics hardware, academia
    • What: Use the pipeline to evaluate new hand designs (backdrivability, joint layout, fingertip geometry) on contact-rich tasks without large teleoperation datasets.
    • Tools/products/workflows:
    • “Sim-to-Real Tuner” kit: standardized SI procedures (motor gains, delays, friction), domain randomization ranges, and curricula from the paper.
    • Benchmark tasks (scissors/screwdriver) with object-centric keypoint rewards to compare hands.
    • Assumptions/dependencies:
    • Availability of Isaac Sim/Isaac Lab; motion capture for object pose.
    • CAD meshes of tools and hands; consistent URDFs.
  • Data-efficient skill generation for imitation-learning pipelines
    • Sector: software for robotics, foundation models for manipulation
    • What: Use a single human demo to synthesize diverse reference trajectories via REGRIND’s time-varying SE(3) augmentation, providing broader training coverage for downstream IL/RL fine-tuning.
    • Tools/products/workflows:
    • “Trajectory Augmentor” module that dynamically perturbs initial states and blends back to the goal to generate infinite training variants.
    • Assumptions/dependencies:
    • Access to the original single demonstration with hand–object 3D trajectories and object mesh.
  • Training curriculum templates for contact-rich manipulation
    • Sector: robotics software
    • What: Adopt the paper’s practical training recipe—reference-state initialization (RSI), object-centric keypoint rewards, action/observation design, gravity and push curricula—to improve RL convergence on tool-use tasks.
    • Tools/products/workflows:
    • Isaac Lab templates encapsulating curriculum schedules and DR parameters (friction, delays, observation noise).
    • Assumptions/dependencies:
    • Simulator supporting parallelized environments and fine-grained physics randomization.
  • Laboratory and classroom teaching modules for dexterous manipulation
    • Sector: education, academia
    • What: Course labs that walk students from a single human demonstration to sim-to-real deployment on a tabletop task.
    • Tools/products/workflows:
    • REGRIND-based course materials, code repos, and task assets.
    • Assumptions/dependencies:
    • Access to a modest mocap setup (or pre-recorded demos), an anthropomorphic hand, and a 6–7 DoF arm.
  • Safer, more stable initialization for contact-rich RL
    • Sector: robotics safety/ops, QA in R&D
    • What: Use interaction-preserving retargeting as a “safe start” state distribution (RSI) to reduce simulator instabilities from IK-based penetrations and prevent RL from learning implausible grasps.
    • Tools/products/workflows:
    • Retargeting quality checks with interaction-mesh deformation metrics; automated filtering of unstable initial states.
    • Assumptions/dependencies:
    • Properly defined hand–object semantic keypoints and collision pairs.
  • Skill transfer and reuse across similar hands and tasks
    • Sector: robotics integration
    • What: Leverage the object-centric keypoint tracking and residual control design to port skills between similar anthropomorphic hands (e.g., LEAP → custom 16–20 DoF hand) with minor re-tuning.
    • Tools/products/workflows:
    • Hand-keypoint correspondence mapping tool; standardized observation/action interfaces.
    • Assumptions/dependencies:
    • Comparable kinematics and fingertip geometry; updated hand-keypoint mappings.
  • Controlled cell operations for R&D and pilot production
    • Sector: industrial R&D, pilot lines
    • What: Execute repetitive screw-turning or tool actuation tasks in caged or monitored cells where object pose can be tracked and the workpiece is fixtured.
    • Tools/products/workflows:
    • Cell kit: markers for tools/workpieces, calibration routines, and policy deployment scripts.
    • Assumptions/dependencies:
    • Stable fixturing, mocap, and known tolerances on part geometry.
  • Open-source baselines for research on contact-rich sim-to-real
    • Sector: academia, open research
    • What: Use released code/data to evaluate new retargeting objectives, reward designs, and sim-to-real strategies on standardized tasks.
    • Tools/products/workflows:
    • Plug-and-play pipelines for ablation studies; logging/metrics for object keypoint error and success rates.
    • Assumptions/dependencies:
    • Reproducible compute environment (Isaac Sim versioning) and object/hand models.
  • Process guidance for sim-to-real in dexterous settings
    • Sector: policy within organizations (best practices), lab operations
    • What: Adopt the paper’s identified “levers” (observation space, residual action design, domain randomization, curricula, system ID) as internal SOPs to improve transfer reliability.
    • Tools/products/workflows:
    • Checklists and acceptance tests (e.g., tabletop overshoot guardrails, action-rate penalties).
    • Assumptions/dependencies:
    • Access to measurement hardware (for SI), and the ability to enforce reward/termination designs.

Long-Term Applications

These opportunities require further research, scaling, or development—primarily removing motion-capture dependence, handling imperfect perception, and broadening task/hand diversity.

  • Vision-based, in-the-wild dexterous tool use from one demonstration
    • Sector: service robotics (home, retail), logistics
    • What: Replace mocap with onboard perception (RGB-D, VLM guidance, tactile) by distilling the state-based policy into a visuomotor policy; enable household tasks such as opening packages, cutting tags, turning knobs, or using simple tools after a user demo.
    • Tools/products/workflows:
    • “One-Demo Skill Studio” with camera calibration and auto keypoint extraction; policy distillation from state to vision.
    • Assumptions/dependencies:
    • Robust object pose estimation and contact state inference under clutter/occlusion; reliable tactile/force sensing.
  • General-purpose dexterous skill libraries for industry
    • Sector: manufacturing, MRO, field service
    • What: Scalable libraries of contact-rich tool skills (screwdriving variants, wire stripping/crimping, plier/clip operations) with interaction-preserving retargeting to new tools and hands.
    • Tools/products/workflows:
    • Cloud training service with automatic mesh ingestion, retargeting, and domain randomization bundles; on-site SI wizards.
    • Assumptions/dependencies:
    • High-quality CAD/meshes; standardized semantic keypoints for diverse tools; standardized hardware APIs.
  • Bimanual and complex assembly tasks via interaction-aware retargeting
    • Sector: advanced manufacturing, electronics, automotive
    • What: Extend interaction meshes to multi-object, multi-hand contact graphs for tasks like fastening with stabilization, connector insertion, cable routing.
    • Tools/products/workflows:
    • Multi-hand retargeting solvers; dual-arm residual RL controllers with shared object-centric rewards.
    • Assumptions/dependencies:
    • Accurate multi-contact physics; synchronized perception; refined curricula for complex contact transitions.
  • Personalized assistive/rehabilitative dexterous robots
    • Sector: healthcare, assistive tech
    • What: End-user demonstration of personalized tasks (e.g., opening pill bottles, using grooming tools) adapted to the individual’s environment and preferred motions.
    • Tools/products/workflows:
    • Safe demonstration capture (wearables, vision), on-device policy adaptation with interaction-aware retargeting; compliance control layers.
    • Assumptions/dependencies:
    • Medical-grade safety, force limits, fail-safe perception; regulatory approvals; robust perception without markers.
  • Skill sharing and marketplaces for robot hands
    • Sector: software platforms, robotics ecosystems
    • What: Distribute and monetize pre-trained, parameterized dexterous skills that adapt to local tools via retargeting and brief SI.
    • Tools/products/workflows:
    • “Skill marketplace” with metadata (tool type, required keypoints, DR ranges), automated validation suites.
    • Assumptions/dependencies:
    • Interoperability standards for hand keypoints and action spaces; IP/licensing frameworks.
  • Human video–to–robot dexterous skill transfer at scale
    • Sector: academia, software, data platforms
    • What: Combine egocentric video datasets with learned hand–object pose estimators to auto-generate retargeted references and train RL policies for diverse tool uses.
    • Tools/products/workflows:
    • Video-to-MANO-and-object pose pipelines; automated interaction-mesh construction from RGB.
    • Assumptions/dependencies:
    • Reliable 3D hand–object pose from video; dataset coverage of contact-rich tool usage.
  • On-the-fly policy adaptation and online SI for changing tools/environments
    • Sector: field robotics, flexible manufacturing
    • What: Test-time adaptation of dynamics (friction, compliance) and residual policies using brief real-world interactions to maintain stability with worn tools or new materials.
    • Tools/products/workflows:
    • Lightweight online SI modules; meta-RL or context inference from tactile and proprioception.
    • Assumptions/dependencies:
    • Real-time sensing and safe exploration; bounded failure modes.
  • Certification and safety frameworks for contact-rich learned policies
    • Sector: standards/policy, regulatory bodies, enterprise governance
    • What: Develop verification procedures for retargeting-guided RL in contact-rich tasks (e.g., ISO/TS 15066-aligned tests for force/trajectory limits and failure handling).
    • Tools/products/workflows:
    • Simulation test batteries with worst-case DR; conformance scores on object keypoint errors and collision metrics.
    • Assumptions/dependencies:
    • Consensus on metrics and acceptable risk; auditable logs from training and deployment.
  • Prosthetics and exoskeleton control via interaction-preserving transfer
    • Sector: medical devices, rehabilitation robotics
    • What: Retarget human hand-object interaction semantics to assistive devices, using residual controllers to stabilize contact and reduce user effort in tool use.
    • Tools/products/workflows:
    • Interaction-mesh mappings between human and device; mixed-initiative control blending user intent and residual stabilization.
    • Assumptions/dependencies:
    • High-fidelity device sensing/actuation and comfort; clinical evaluations and approvals.
  • Energy and infrastructure maintenance tasks with dexterous tooling
    • Sector: energy, utilities, inspection/maintenance
    • What: Teach robots to operate specialized tools (valve keys, cutters, crimpers) in semi-structured environments from minimal demos; adapt to varying tool geometries via retargeting.
    • Tools/products/workflows:
    • Library of safety-checked tool-use skills; field-hardened perception and SI kits.
    • Assumptions/dependencies:
    • Robust perception in harsh conditions; safety and redundancy; task-specific end-effector protections.

Notes on feasibility across applications:

  • The current pipeline presumes access to precise object pose at deployment (motion capture). Removing this dependency requires reliable vision/tactile perception and policy distillation.
  • Transfer quality is sensitive to sim-to-real gaps in friction, compliance, and object geometry; careful system identification and accurate meshes are critical.
  • Backdrivability and hardware compliance affect stability (non-backdrivable hands showed transfer challenges in the paper); hardware selection matters.
  • Reward and observation design, domain randomization, and curricula significantly impact success; reusing the paper’s templates mitigates risk.
View Paper Prompt  View All Prompts 

Glossary

  • Articulated objects: Objects with joints that can move relative to each other (non-rigid). "covering both rigid and articulated objects"
  • Asymmetric actor-critic: An RL setup where the actor and critic receive different observations, often giving the critic extra state for training stability. "We use asymmetric actor-critic observations."
  • Contact-rich: Involving frequent, sustained, and complex physical contacts between robot and environment or objects. "contact-rich tool-use tasks"
  • Delaunay tetrahedralization: A method to partition 3D space into tetrahedra such that no point lies inside the circumsphere of any tetrahedron, used for mesh connectivity. "The connectivity between the keypoints P are decided by Delaunay tetrahedralization [48]"
  • Domain randomization: Randomizing simulation parameters during training to improve robustness and transfer to the real world. "we apply domain randomization to the physics simulation parameters"
  • Embodiment gap: Differences in morphology and capabilities between human and robot that complicate direct mapping of demonstrations. "because of the embodiment gap between the robot and the human"
  • Exploratory restart distribution: A distribution over initial states used to guide exploration in RL by resetting into informative states. "introduced the concept of exploratory restart distribution"
  • Gravity curriculum: Gradually increasing gravity during training to ease learning of manipulation skills. "We also adopt gravity curriculum to gradually increase the gravity magnitude from zero to its full value"
  • Interaction mesh: A mesh connecting hand and object keypoints to preserve spatial/contact relationships during retargeting. "We construct the source interaction mesh with the object and hand keypoints"
  • Kinematic retargeting: Mapping human motion to robot configurations using kinematics, often ignoring dynamics and contacts. "simple kinematic retargeting"
  • Laplacian coordinates: Mesh-based coordinates capturing local geometric relations, used to penalize deformation. "defined as the change of the Laplacian coordinates of the meshes"
  • Markov Decision Process (MDP): A mathematical framework for sequential decision making with states, actions, transitions, and rewards. "formulated as a finite-horizon Markov Decision Process (MDP)"
  • Motion retargeting: Converting human demonstration motions into robot trajectories while preserving key semantics. "The goal of motion retargeting is to convert the human demonstration into a robot trajectory"
  • Non-backdrivable motors: Actuators that resist passive back-driving, affecting compliance and contact behavior. "the non-backdrivable motors on the WUJI hand"
  • PD controller: Proportional-Derivative controller that tracks target positions/velocities using feedback control. "This control target is then sent to a low-level PD controller."
  • Phase variable: A normalized progress parameter indicating where the system is along a reference motion. "as well as a phase variable ¢ € [0,1]"
  • Proprioception: Internal sensing of a robot’s joint states and body configuration. "The robot proprioception consists of all joint positions"
  • Reference state initialization (RSI): Resetting episodes to states sampled from a reference trajectory to stabilize and accelerate learning. "we adopt reference state initialization (RSI) [46]"
  • Residual RL: Learning an additive correction on top of a nominal/reference action or trajectory. "train a residual RL policy in simulation to track object-centric keypoints"
  • Rot6D: A continuous 6D representation for 3D rotations used in neural networks. "represented by Rot6D [49]"
  • Sequential Quadratic Programming (SQP): An iterative optimization method that solves a sequence of quadratic subproblems to handle nonlinear constraints. "a Sequential Quadratic Programming (SQP)-style solver"
  • Sim-to-real transfer: Deploying policies trained in simulation on real hardware while mitigating modeling gaps. "sim-to-real transfer in dexterous manipulation"
  • System identification: Estimating physical parameters of the real system to align simulation and hardware behavior. "zero-shot to hardware with careful system identification"
  • Teleoperation: Controlling a robot remotely by a human operator to collect demonstrations or perform tasks. "retargeting has mainly supported teleoperation and demonstration generation"
  • URDF: Unified Robot Description Format, an XML format describing robot kinematics and visuals for simulation/control. "the corresponding robot keypoints we define on the robot URDFs"
  • Zero-shot: Deploying to a new setting (e.g., hardware) without additional task-specific training or fine-tuning. "transfers the resulting policy zero-shot to hardware"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 3 tweets with 105 likes about this paper.

Sign Up for Free