DragMesh-2: Hand-Driven Articulated Interaction

• DragMesh-2 is a contact-driven framework for dexterous hand-object interaction that leverages sustained physical contact to drive articulated object motion.
• It integrates a simulated 51-DoF hand, Isaac Gym’s contact dynamics, and PICA—a physically informed training mechanism—to learn adaptive, robust policies.
• The framework demonstrates resilience under varied damping conditions, emphasizing trade-offs between maintaining contact and managing action saturation.

Searching arXiv for the specified paper and closely related work to ground the article. {"query":"(Zhang et al., 13 Jun 2026) DragMesh-2 dexterous articulated objects DragMesh", "max_results": 5} I’m checking the exact arXiv entry and nearby related papers for accurate citation context. {"query":"(Zhang et al., 13 Jun 2026)", "max_results": 10} DragMesh-2 is a contact-driven framework for dexterous interaction with articulated objects, introduced to extend articulated interaction from object-centric generation to hand-driven dexterous hand-object interaction, where articulated motion must arise through physical contact rather than direct actuation of the target part (Zhang et al., 13 Jun 2026). The formulation targets settings in which the target part cannot be directly actuated and its motion must emerge through sustained physical hand-handle contact. Within this setting, DragMesh-2 combines a simulated multi-finger hand, contact-driven articulated dynamics in Isaac Gym, and PICA, a physically informed contact-aware training mechanism that injects physical signals into policy learning without tactile or force feedback. The reported study evaluates robustness under changing contact loads across seven GAPartNet objects and multiple damping conditions, and also releases a pure-geometry dexterous interaction resource for future loco-manipulation and humanoid hand-object interaction research (Zhang et al., 13 Jun 2026).

1. Problem formulation and scope

DragMesh-2 is motivated by a distinction between static-object manipulation and articulated-object manipulation. In the latter, the target part cannot be directly actuated, and its motion must emerge through sustained physical hand-handle contact. The paper therefore treats the transition from object-centric articulated generation to hand-driven dexterous hand-object interaction as non-trivial, because geometric trajectory replay or open-loop execution does not model the contact dynamics required to move the articulated part (Zhang et al., 13 Jun 2026).

The framework makes this distinction explicit by contrasting DragMesh 1 and DragMesh-2. DragMesh 1 performed open-loop, geometry-guided articulation by replaying part poses, whereas DragMesh-2 enforces that only the hand is actuated and object parts move only in response to persistent contact. The resulting task is described as “contact-driven pulling,” in which the loop is closed through physical interaction rather than through replay of object-centric motion. This suggests that the central research question is not merely whether a part can be opened, but whether a dexterous policy can maintain effective contact while adapting its finger forces and motions under altered interaction dynamics.

A second component of the problem statement concerns robustness. Policies trained only for task completion under fixed dynamics can overfit nominal contact loads, especially without tactile or force feedback, and may degrade when the contact load changes. DragMesh-2 addresses this by explicitly studying contact-load variation through joint-damping randomization during training and controlled damping shifts at evaluation. A plausible implication is that the framework is intended as a benchmark for contact-conditioned adaptation as much as for articulated opening itself.

2. Contact-driven system model

The simulated hand is a 51-DoF SMPL-X right hand, consisting of 6 wrist DOFs and 45 finger joints, floating in space. The policy issues 51-dimensional increments to joint PD targets, and a low-level PD controller executes the commands (Zhang et al., 13 Jun 2026). This action design means that articulation is mediated through hand posture adjustments rather than through any explicit object-joint command channel.

The articulated objects come from GAPartNet with part-level geometry and joint annotations, which may be revolute or prismatic. At reset, a single target part, defined as the part with the largest motion range, and its handle are selected, and the hand is placed in a reference grasp pose. The object joint is uncontrolled; motion can occur only via simulated rigid-body contact with the hand. In consequence, the policy must generate object motion indirectly by producing contact forces at the handle.

Contacts are simulated in Isaac Gym. At each contact point, a linear spring-dashpot model computes normal and tangential forces:

$$F_n = k_n \delta_n + b_n \dot{s}_n,\qquad F_t = k_t \delta_t + b_t \dot{s}_t$$

where $\delta$ is penetration, $\dot{s}$ is relative velocity, and $k$ and $b$ are stiffness and damping. The paper also gives a single-point contact-force expression:

$F_c = k_n \delta_n + k_t \delta_t.$

The articulated joint obeys torque balance:

$\tau = B(\theta)\dot{x} + g(\theta) + \tau_c$

with $B$ the damping coefficient, $g$ gravity, and $\tau_c$ the net moment of contact forces. The joint torque on the part arises solely from contact:

$\delta$0

This formulation establishes a strict separation between actuation and articulation. The policy can alter only hand motion; the articulated part responds only if the induced contact geometry and contact forces generate sufficient joint torque. That design is central to the paper’s notion of physical plausibility.

3. PICA and physically informed training

PICA is introduced as a physically informed contact-aware training mechanism that injects physical signals into policy learning without tactile or force feedback (Zhang et al., 13 Jun 2026). Despite the absence of explicit force or tactile sensors, the policy observes hand joint positions and velocities, handle pose, palm-handle distance $\delta$1, object joint angle $\delta$2, and task-scale features consisting of progress and remaining distance. It also receives a short history of past PD tracking errors and actions:

$\delta$3

These history terms are assembled into a blocked input

$\delta$4

and fed to a Gated Linear Attention (GLA) temporal encoder. The resulting history representation $\delta$5 is combined with current-state features for actor-critic learning. The intended effect is to let the policy infer latent contact state from kinematic consequences of recent actions.

From the last $\delta$6 steps, the environment computes four auxiliary targets:

1. $\delta$7, the recent joint response;
2. $\delta$8, the maximum palm distance;
3. $\delta$9, a detachment proxy;
4. $\dot{s}$0, tracking-error stress.

The PICA auxiliary loss is

$\dot{s}$1

The total optimization objective is

$\dot{s}$2

where $\dot{s}$3 is the standard clipped surrogate policy loss, $\dot{s}$4 is value-function MSE, and $\dot{s}$5 is a penalty on $\dot{s}$6 to discourage action saturation.

The paper states that, together with contact-distance and detachment penalties in the reward and with randomized joint damping, PICA biases learning toward stable, contact-conditioned strategies. This suggests that PICA is not a replacement for explicit contact sensing, but an inference-oriented surrogate that makes latent interaction loads partially observable through motion history and PD tracking residuals.

4. Policy architecture, observations, and actions

The current-state encoder is an MLP with layer widths $\dot{s}$7 and ELU activations, producing a feature denoted $\dot{s}$8 (Zhang et al., 13 Jun 2026). The history encoder applies a linear projection of each $\dot{s}$9 to 64-dimensional tokens and processes them with a 4-head GLA module; the last token yields a 64-dimensional $k$0. These are concatenated and passed to two heads: an actor head that outputs Gaussian mean $k$1 with fixed $k$2, and a critic head that outputs a scalar value $k$3. The full network contains approximately $k$4 million parameters.

The observation space is approximately 100-dimensional and comprises hand joint positions (51 D), hand joint velocities (51 D), handle pose (7 D: position plus quaternion), palm-handle distance (1 D), object joint angle and velocity (2 D), and task-scale features consisting of normalized progress, remaining distance, and scale factor (3 D). The action space is 51-dimensional and consists of incremental PD-target offsets in $k$5 clipped to joint limits, with $k$6 rad.

This architecture is technically notable for how it partitions information. Current kinematic state and task progress are handled by a high-capacity feedforward encoder, while latent interaction structure is delegated to a temporal module operating on PD-error and action history. A plausible implication is that the design assumes physically relevant hidden variables, such as contact load and impending detachment, are better inferred from short-horizon temporal traces than from instantaneous state alone.

5. Experimental protocol and benchmark setting

The evaluation uses seven objects from GAPartNet across three categories and two joint types: StorageFurniture drawers (2 instances), StorageFurniture doors (3 instances), Dishwasher door (1 instance), and Microwave door (1 instance) (Zhang et al., 13 Jun 2026). The task begins from a reference grasp state obtained from a pure-geometry demo trajectory, and success requires opening the target part by at least 50% of its motion range through hand contact only.

Contact-load variation is implemented through object-joint damping randomization. During training, the object-joint damping is multiplied at reset by a random factor drawn from $k$7. During evaluation, the damping multiplier is fixed to $k$8 (nominal), $k$9 (mild shift), or $b$0 (strong out-of-distribution). Both deterministic execution, using $b$1 only, and stochastic execution, using sampled actions, are tested for 20 episodes each.

The benchmark explicitly compares DragMesh-2 against State-only PPO, Flat-history PPO, GRU-PPO, Transformer-PPO, a Parallel-jaw primitive, and Trajectory-tracking. The inclusion of trajectory-tracking is important because it separates geometric replay from contact-aware policy adaptation. The paper’s framing does not imply that geometry alone is useless; rather, it argues that geometry-guided execution is insufficient when the contact dynamics shift and the target part must move only through persistent physical interaction.

6. Quantitative results, ablations, and interpretation

The main deterministic success rates, averaged over seven objects, are as follows (Zhang et al., 13 Jun 2026):

Method	$b$2	$b$3	$b$4
DragMesh-2 (PICA)	0.89	0.80	0.56
State-only PPO	0.58	0.44	0.27
Flat-history PPO	0.35	0.25	0.15
GRU-PPO	0.51	0.33	0.30
Transformer-PPO	0.27	0.09	0.00
Parallel-jaw primitive	0.14	0.14	0.14
Trajectory-tracking	1.00	0.71	0.71

The robustness metrics further emphasize worst-case and failure-mode behavior. The worst-case success across $b$5, denoted $b$6, is 0.56 for DragMesh-2 and 0.27 for State-only PPO. Under strong damping $b$7, DragMesh-2’s action-saturation proxy, clip0.99, is approximately 0.45, versus greater than 0.9 for State-only PPO. The detachment-failure rate under $b$8 is approximately 0.43 for DragMesh-2 and approximately 0.90 for baselines.

The ablations isolate both main components. Removing physical signals while keeping GLA (“w/o PICA”) yields $b$9 success of 0.36. Removing the GLA temporal encoder while keeping PICA (“w/o GLA”) yields $F_c = k_n \delta_n + k_t \delta_t.$0 success of 0.43. The full method reaches 0.56 at $F_c = k_n \delta_n + k_t \delta_t.$1 in deterministic evaluation. The paper interprets this as evidence that both physical-signal supervision and temporal encoding are needed.

Several explanatory claims are stated directly. Contact-maintenance and detachment penalties prevent the policy from simply “ripping” and losing grip. Action-boundary regularization through $F_c = k_n \delta_n + k_t \delta_t.$2 avoids saturated commands that fail under higher load. Damping randomization reduces overfitting by exposing the policy to variation in joint impedance. Temporal encoding combined with auxiliary supervision helps the policy internalize a representation of how recent actions produced object response, or failed to do so. Taken together, these points support an interpretation of robustness as emerging from contact-conditioned adaptation rather than from trajectory memorization.

7. Limitations, misconceptions, and future directions

The paper identifies several limitations (Zhang et al., 13 Jun 2026). Without force or tactile sensors, contact state must be inferred from kinematic errors, and light pulling under very high damping remains challenging. Because the action space is restricted to fixed incremental-position limits, direct regulation of grip force is not available, and the policy often pushes to actuator limits. The study also reports per-object heterogeneity, stating that no single policy dominates every instance.

These limitations clarify a potential misconception. DragMesh-2 is not presented as a fully sensed dexterous manipulation system; it is instead a framework that seeks physically plausible articulated interaction under partial observability, using only kinematic and task-derived signals. Similarly, the comparison with trajectory-tracking does not establish that trajectory replay is universally ineffective. The reported numbers show that trajectory-tracking attains 1.00 at $F_c = k_n \delta_n + k_t \delta_t.$3 and 0.71 at both $F_c = k_n \delta_n + k_t \delta_t.$4 and $F_c = k_n \delta_n + k_t \delta_t.$5, but the paper’s broader argument is that open-loop execution does not model the contact dynamics required to move the articulated part and is not, by itself, a contact-aware solution.

The future directions listed in the paper follow directly from these constraints. Proposed extensions include enriching the action space with wrist force/torque outputs, augmenting observations with contact-force or tactile feedback, introducing mode-switching such as light versus heavy pulling or latent environment factors inferred online, and extending from hand-only interaction to full-body humanoid loco-manipulation using the released pure-geometry trajectory dataset as a motion prior. Code and dataset are available through the project repository and project page specified in the paper.