Tele-Catch: Adaptive Teleoperation for Dexterous Dynamic 3D Object Catching

Abstract: Teleoperation is a key paradigm for transferring human dexterity to robots, yet most prior work targets objects that are initially static, such as grasping or manipulation. Dynamic object catch, where objects move before contact, remains underexplored. Pure teleoperation in this task often fails due to timing, pose, and force errors, highlighting the need for shared autonomy that combines human input with autonomous policies. To this end, we present Tele-Catch, a systematic framework for dexterous hand teleoperation in dynamic object catching. At its core, we design DAIM, a dynamics-aware adaptive integration mechanism that realizes shared autonomy by fusing glove-based teleoperation signals into the diffusion policy denoising process. It adaptively modulates control based on the interaction object state. To improve policy robustness, we introduce DP-U3R, which integrates unsupervised geometric representations from point cloud observations into diffusion policy learning, enabling geometry-aware decision making. Extensive experiments demonstrate that Tele-Catch significantly improves accuracy and robustness in dynamic catching tasks, while also exhibiting consistent gains across distinct dexterous hand embodiments and previously unseen object categories.

• The paper introduces an adaptive shared autonomy framework combining deep RL, diffusion policy with unsupervised 3D representation, and the DAIM mechanism to enhance dynamic object catching.
• It employs a multi-stage architecture that unifies human teleoperation with autonomous policy priors through quality-guided data collection and geometry-aware control.
• Empirical evaluations show success rate improvements from 35.3% to 54.7% overall, reaching up to 86.7% in specific categories across diverse robotic hand setups.

Tele-Catch: Adaptive Teleoperation for Dexterous Dynamic 3D Object Catching

Motivation and Problem Statement

Dexterous dynamic object catching via teleoperation presents distinct technical challenges beyond static manipulation: input timing is highly sensitive, pose estimation may be unreliable due to rapid relative motion, force regulation is prone to over- or under-compensation, and retargeting error compounds with fast movements and embodiment discrepancies. Conventional teleoperation and existing shared-autonomy frameworks fail to robustly support dynamic object catching due to their inability to adaptively arbitrate control authority or leverage high-fidelity geometry. Tele-Catch directly addresses this gap by introducing an adaptive shared autonomy paradigm, focusing specifically on dexterous robotic hands engaged in catching dynamic 3D objects under human-in-the-loop guidance and real-timed glove-based teleoperation.

Figure 1: Teleoperation in dexterous manipulation, highlighting four failure modalities intrinsic to dynamic object interaction: premature grasp, incorrect grasp pose, excessive force, and retargeting errors.

Tele-Catch Framework Overview

Tele-Catch unifies human teleoperation and autonomous policy priors through a multi-stage architecture:

1. Deep RL Policy Acquisition: A PPO-driven policy is trained in simulation using a composite shaped reward, explicitly regularizing for kinematic accuracy, stable contact, energy efficiency, and dynamic adaptation. This policy demonstrates robust catching but remains susceptible to embodiment-specific teleoperation errors.
2. Quality-Guided Data Collection: Successful RL trajectories are curated and paired with point cloud streams, strictly filtered to ensure consistent dynamic grasp quality.
3. Diffusion Policy with Unsupervised 3D Representation (DP-U3R): Diffusion policies are augmented with unsupervised point cloud embeddings. Local geometric nuances and global structure are extracted from noise-perturbed point clouds; the global feature enters the observation space of the DP, yielding geometry-aware control that generalizes across categories and kinematic layouts.
4. Dynamics-Aware Adaptive Integration Mechanism (DAIM): Glove input is fuzzily retargeted to the robot and injected into diffusion denoising. Control authority is adaptively weighted by both the denoising step and the object's translational and angular velocities, ensuring that the learned policy dominates during rapid object motion while teleoperation guidance prevails under quasi-static conditions.

   Figure 2: Modular Tele-Catch pipeline spanning RL training, data curation, DP-U3R point cloud augmentation, and real-time shared autonomy with adaptive glove signal integration.

Technical Contributions

DAIM (Dynamics-Aware Adaptive Integration Mechanism):

DAIM introduces step-wise, object-state-dependent weighting for teleoperator actions versus policy outputs, leveraging a cosine schedule modulated by logistic scaling on object velocities. This mitigates the inherent unreliability of human input when rapid object acceleration is present, while preserving user intent during slow dynamics. The result is a stable, non-disruptive fusion, as opposed to abrupt or brittle threshold-based arbitration.

DP-U3R (Diffusion Policy with Unsupervised 3D Representation):

Point clouds are systematically perturbed and encoded; local and global features undergo attention-based fusion, with only the global structure embedded to ensure inference efficiency. This unsupervised 3D representation is coupled with policy training via a bifurcated loss: reconstruction (L2 point-level) and diffusion noise prediction. The point cloud encoder thus grounds the policy’s action selection in geometric context, boosting both specificity and transferability across unfamiliar geometries.

Empirical Evaluation and Numerical Results

The framework is evaluated using both ShadowHand (24-DoF) and Linkerhand (16-DoF) in diverse dynamic catching scenarios. Metrics include success rate (SR, percentage of stable catches) and diffusion action denoising MSE. The mean SR rises from 35.3% (pure teleoperation baseline) to 54.7% with Tele-Catch, with category-specific SR reaching 86.7% (Teapot) and marked gains in challenging categories such as Truck and Showerhead.

Ablation studies showcase the impact of backbone choice: substituting DP and DP3 in place of DP-U3R results in lower SR and higher MSE, evidencing that unsupervised 3D structure directly benefits action coherence. The DP-U3R backbone achieves significantly lower action-noise MSEs and more uniform performance across both regular and thin-edged geometries.

Figure 4: Qualitative analysis illustrating synchronization between glove input, robotic hand motion, and point cloud perception as a function of Tele-Catch’s adaptive policy fusion.

Sim2Real tests involve Manus glove teleoperation and Xhand hardware with RGB-D sensing, confirming that the system is robust to hardware transfer (with policy retraining) and can generalize to unseen object categories.

Figure 6: Sim2Real deployment, with unedited real-world catches implemented using the full Tele-Catch framework.

Failure analysis attributes most unsuccessful trials to four canonical error modes identified in the introduction, with retargeting error and rapid impact (thin object edge contact) remaining the dominant challenges under real-world noise.

Generalization and Sensitivity Analysis

Tele-Catch retains strong cross-embodiment viability: retraining the DP-U3R backbone for Linkerhand yields SR improvements (73.3% vs. 40.0%) mirroring those seen with the canonical ShadowHand. Unseen-category trials (Eraser, Telephone, Bottle) confirm DP-U3R’s geometry-aware policy robustness, delivering substantial zero-shot generalization without retraining.

Sensitivity sweeps of DAIM’s dynamics weighting hyperparameters (βv, βω) reveal that attenuating angular velocity influence (β_ω) stabilizes policy blending under discontinuous rotation estimates, while β_v governs the balance between reactivity and teleoperation responsiveness.

Figure 3: Sensitivity analysis highlighting adaptive control transitions between policy and glove input for the ring finger, validating DAIM’s dynamic blending.

Theoretical and Practical Implications

The Tele-Catch framework demonstrates that modular integration of policy priors and unsupervised 3D perception establishes robustness unattainable by teleoperation or fixed-policy automation alone. The DAIM mechanism represents a new standard for arbitration between human intent and learned priors, dynamically and smoothly adapting authority as a function of scene and task state. Geometric context, via point cloud encoding, amplifies policy coherence under unfamiliar objects and embodiments. The results indicate that such architecture could be extended to broader domains where non-stationary human-robot handover or rapid shared autonomy is required.

Limitations and Future Directions

Residual failure cases underscore the enduring difficulties of contact-rich, high-DOF tasks under perception error and kinematic mismatch. The need for precise object state estimation remains a practical constraint for full sim2real transfer. Future directions include tighter coupled pose-tracking, learning shape-conditioned DAIM schedules, and exploring richer action-conditional priors leveraging foundation models for more abstract dynamic tasks.

Conclusion

Tele-Catch systematically reconceives teleoperation for the dexterous dynamic object catching regime by combining policy diffusion, unsupervised 3D geometric encoding, and dynamics-aware integration of human input. The resulting shared autonomy yields measurable and repeatable improvements over direct teleoperation, with strong empirical robustness across object categories, robotic hand designs, and real-world transfer scenarios. The modular approach and adaptive arbitration mechanism outlined here provide a foundational scaffold for advancing the study and deployment of human-in-the-loop dynamic manipulation in complex embodied AI systems.