DragMesh-2: Physically Plausible Dexterous Hand-Object Interaction with Articulated Objects

Abstract: Dexterous interaction with articulated objects is important for household, assistive, and humanoid manipulation, where multi-finger hands can provide compliant contact patterns beyond parallel-jaw grasping. However, articulated-object manipulation differs from static-object manipulation: the target part cannot be directly actuated, and its motion must emerge through sustained physical hand--handle contact. This makes the transition from object-centric articulated generation to hand-driven dexterous hand--object interaction non-trivial, since geometric trajectory replay or open-loop execution does not model the contact dynamics required to move the articulated part. Moreover, policies trained only for task completion under fixed dynamics can overfit nominal contact loads, especially without tactile or force feedback, and may degrade when the contact load changes. To address these challenges, we present DragMesh-2, a contact-driven framework for dexterous interaction with articulated objects that extends articulated interaction from object-centric generation to hand-driven dexterous hand--object interaction, where articulated motion must arise through physical contact. We further propose PICA, a physically informed contact-aware training mechanism that injects physical signals into policy learning without tactile or force feedback, improving robustness and task success under changing contact loads. Finally, we conduct systematic evaluation across multiple damping conditions and articulated-object categories to study robustness under contact-load variation, and provide a pure-geometry dexterous interaction resource to support future loco-manipulation and humanoid hand--object interaction research. Across seven GAPartNet objects, DragMesh-2 achieves stronger robustness under contact-load variation than the compared methods while maintaining high task success across damping conditions.

• The paper introduces a contact-driven manipulation paradigm that achieves high success rates using a physically informed, RL-based training (PICA).
• It employs temporal encoding and damping randomization to enhance policy robustness when handling articulated objects under non-nominal damping conditions.
• The study validates DragMesh-2 via a pure-geometry dataset and ablation studies, demonstrating superior performance over conventional PPO-based methods.

Physically Plausible Dexterous Manipulation with DragMesh-2

Motivation and Problem Definition

Manipulation of articulated objects in household, assistive, and humanoid contexts demands handling complex motion constraints and multi-finger contact patterns. Unlike rigid objects, parts of articulated objects (such as drawers, doors) cannot be directly actuated; motion must be induced through sustained, physically plausible hand-handle contact. Prior approaches—primarily geometric trajectory replay or open-loop execution—fail to capture the required contact dynamics, rendering them non-robust under varying physical conditions (e.g., increased damping). Conventional RL-based policies tend to overfit nominal dynamics in the absence of force/tactile feedback, manifesting degraded performance when confronted with real-world physical variability.

DragMesh-2 Framework

DragMesh-2 proposes a strictly contact-driven manipulation paradigm, where control is confined to a high-DoF SMPL-X hand, and articulated object parts move solely through induced physical interaction. The approach eliminates any direct action channels to object joints, enforcing articulated motion to emerge via contact mechanics. The framework thus shifts the focus from object-centric articulation to dexterous hand-object interaction, making contact regularization and physical plausibility central to policy learning.

Physically Informed Contact-Aware (PICA) Training

PICA advances policy robustness under unseen (OOD) contact loads without explicit tactile or force feedback. Physical signals such as contact maintenance, detachment risk, action-boundary regularization, damping variation, and temporal contact response are injected into the RL pipeline. The mechanism is instantiated on top of PPO, augmented by:

• Contact maintenance and detachment signals: Penalizing unsafe separation and saturated actions, and rewarding sustainable contact.
• Damping randomization: Training with randomized object joint resistance to mitigate overfitting.
• Temporal encoder (GLA): Conditioning policy behavior on short-term interaction history, allowing contact-state awareness absent direct force input.
• Auxiliary supervision: Causal-window predictions of recent object response, palm-handle distance, detachment probability, and tracking stress inform temporal feature learning.

These additions bias policy learning away from shortcut behaviors and toward physically plausible, contact-conditioned strategies. Ablation studies underscore the necessity of coupling temporal encoding with explicit physical signals.

Dataset and Evaluation Protocol

DragMesh-2 introduces a pure-geometry dataset, comprising reference contact trajectories generated heuristically from annotated GAPartNet geometry. These trajectories serve as expert grasp initializations, define task normalization scales, and provide tracking benchmarks without relying on any RL artifact. The benchmark encompasses seven GAPartNet objects, spanning multiple categories and joint types, and systematically evaluates performance under damping multipliers ($\times1$ nominal, $\times2$ mild, $\times4$ OOD).

Evaluation metrics extend beyond task success to include action saturation (clip099) and detachment-failure rate. This comprehensive protocol prevents training reward or nominal task completion from hiding instability under physical loads.

Numerical Results and Analysis

DragMesh-2 with PICA achieves superior robustness under contact-load variation relative to geometric primitives and RL baselines, including PPO variants with recurrent or Transformer encoders.

• Success Rate: PICA delivers deterministic success rates of $0.89$ at $\times1$ and $0.56$ at $\times4$ damping, outperforming state-only PPO ($0.27$ at $\times4$) and Transformer-PPO ($0.09$ at $\times2$0).
• Omissions reveal failure: Geometric primitives (parallel-jaw) succeed only on a single object and perform poorly as contact load increases.
• Ablations: Removing physical signals or temporal encoding significantly degrades robustness; their combination is necessary for stable contact behaviors under OOD dynamics.
• Training protocol: Longer nominal training increases action saturation and collapses robustness; reporting OOD metrics and physical diagnostics is essential for checkpoint selection.

Limitations and Future Directions

Despite its advances, DragMesh-2's policy remains bounded by the lack of direct force/tactile sensing and its position-increment action interface. Under strong damping, inferred contact state from kinematics is insufficient for stable manipulation. Per-object heterogeneity in results indicates that further gains require enhancements to the contact interface: inclusion of wrist force/torque outputs, integration of tactile/contact feedback, and possibly mode-switching for light/heavy pulling. The dataset and method are amenable to extension toward whole-body loco-manipulation, harnessing geometry-guided motion-scale priors for humanoid coordination.

Implications

Practically, DragMesh-2 defines a robust, contact-driven manipulation benchmark and a reusable dataset resource for future research in dexterous HOI and loco-manipulation. Theoretically, its results underscore that physically plausible manipulation demands policy architectures intertwined with explicit contact diagnostics and temporal representation. The separation of task progress optimization from physical plausibility constraints should be a guiding principle for stable, generalizable policy learning in contact-rich robotics.

Conclusion

DragMesh-2 establishes a contact-centric framework for dexterous manipulation of articulated objects, rendering motion induction strictly through hand-object contact. Its PICA training regimen—anchored in physically informed signals, dynamics randomization, and temporal modeling—consistently improves robustness under variable contact loads, without force or tactile feedback. The released geometry-based interaction dataset and the contact-aware protocol constitute valuable tools for advancing stable, realistic manipulation research, with clear pathways toward extensions encompassing force-aware control and whole-body coordinated interaction.