Arm-Aware Guided Dexterous Grasp Generation with Arm-Agnostic Grasp Models

Abstract: Dexterous grasp generation that considers arm-related constraints is crucial in real-world scenarios involving arm environment collision avoidance, workspace boundary grasps, and consecutive grasping. Existing hand-centric grasp models, which primarily focus on the floating hand's pose, are insufficient for such cases. Conventional arm-aware methods either rely on rejection sampling to discard infeasible samples or require retraining on arm-specific data, leading to low sample efficiency under adverse conditions or limited generalization across different robots and environments. To overcome these limitations, this letter presents an arm-aware dexterous grasp generation framework that leverages pretrained arm-agnostic grasp models while integrating arm and environmental information only at inference time. Specifically, we formulate arm-aware constrained grasp generation as a joint optimization of hand pose and arm configuration, and derive closed-form gradients for arm-related constraints. Assuming the hand pose distribution is represented by a diffusion model, we prove that gradient-based optimization is equivalent to guided diffusion sampling, steering near-feasible samples toward the feasible region. Through comprehensive evaluation involving 10k objects across 6 scenarios, we demonstrate that the proposed framework generates feasible grasps in highly constrained settings with significantly higher probability, highlighting its advantages in real-world applications. Supplementary materials and appendix are available at https://arm-aware-dexgrasp.github.io/.

• The paper introduces an inference-time guidance method that maps arm joint-space constraints into closed-form Cartesian wrist-pose gradients for a pretrained, arm-agnostic diffusion model.
• Experiments across 10,892 objects show feasible-grasp rates increasing from 15.31% to 88.87% in a UR5 narrow-corridor scenario, while improving overall and object-level success without retraining.
• The method transfers across UR5 and Franka arms, operates in roughly 0.5–2 ms per successful grasp, and reduces consecutive-grasp joint distance from 0.88 to 0.56 radians, but can trade grasp quality for feasibility.

Overview

This paper addresses a persistent gap between hand-centric dexterous grasp generation and the practical requirements of real-world manipulation: arm–environment collision avoidance, workspace-boundary reachability, and joint-space proximity for consecutive grasping. The authors propose an inference-time guidance framework that injects arm-related constraints into a pretrained, arm-agnostic diffusion model over wrist poses, avoiding both the sample inefficiency of rejection sampling and the poor generalization of arm-specific retraining (2608.16351). The central technical contribution is a principled mapping from joint-space (arm configuration) constraints to Cartesian-space (wrist pose) denoising updates, realized through closed-form constraint gradients.

Problem formulation as bi-level constrained optimization

The method formulates arm-aware grasp generation as a variational problem over the sampling distribution $\pi$: minimize $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$, where $p_\theta$ is the pretrained wrist-pose diffusion model conditioned on a partial object point cloud, and $g(\bm{x})$ is a value function defined by an inner constrained optimization over arm configurations $\bm{q}$ in the IK solution set $$\mathcal{Q}(\bm{x}) = \{\bm{q} \mid \mathrm{FK}(\bm{q}) = \bm{x}\}$$. Collision avoidance and hand reachability are treated as hard inequality constraints handled by a primal–dual scheme with an adaptively updated Lagrange multiplier $\lambda_c$; joint proximity is a soft penalty. Solving the outer problem yields an exponentially tilted distribution $$\pi^*(\bm{x}) \propto p_\theta(\bm{x}\mid\mathcal{O})\exp[-g(\bm{x})]$$, which converts directly into a DDIM-compatible denoising update that subtracts $\frac{\beta_t}{2}\nabla g(\hat{\bm{x}}_0)$ at each step. Notably, gradients are evaluated on the predicted clean pose $\hat{\bm{x}}_0$ rather than the noised sample $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$0, which the authors argue—and ablate—yields smoother, more consistent guidance under collision constraints.

Closed-form gradients for arm-related constraints

The key difficulty addressed is differentiating through the IK map. By viewing the IK solution $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$1 as the minimizer of a damped least-squares problem, sensitivity analysis gives $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$2, independent of the reference configuration. The resulting value-function gradient decomposes into a direct term in $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$3 plus an indirect term propagated through the damped pseudo-inverse of the geometric Jacobian. Three constraints are instantiated analytically: reachability via forward-kinematics error; collision avoidance via minimum SDF values over sphere-based link models with a SoftPlus activation producing sigmoid-shaped distance-dependent scaling; and joint proximity as a quadratic penalty toward a reference configuration $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$4. Multiple IK branches are enumerated with a geometry-based analytical solver, and redundant DoFs (Franka) are sampled uniformly. This construction requires only analytical kinematics and environment SDFs, so it transfers across arms and scenes without data synthesis or retraining—a property the experiments confirm on UR5 and Franka with a single hand-centric model trained once on roughly 50k BODex-synthesized tabletop grasps.

Simulation results

Evaluation spans 10,892 DGN test objects with 10 samples each (~100k grasps) across six scenarios featuring high obstacle coverage and workspace-boundary placements. The headline result is a dramatic improvement in feasibility: in the Narrow Corridor scene with UR5, the feasible rate rises from 15.31% (rejection sampling) to 88.87%, and overall success rate from 8.91% to 47.17%, with object success rate reaching 91.77%. On Franka, gains are consistent though smaller (e.g., FR 10.58% → 59.18% in S1), attributed to Franka's larger collision volume and more limited workspace. A recurring trade-off appears throughout: guidance slightly degrades grasp quality among feasible grasps (GSR drops several points relative to rejection sampling), because steering perturbs the learned distribution—but SR and OSR improve substantially since far more samples land in the feasible region. Ablations show each constraint governs its target metric (collision-only maximizes CFR; reachability-only maximizes RFR), while jointly applying them yields the best combined feasible rates, indicating that fully resolving the reachability–collision trade-off remains open.

Against re-implemented constraint-aware baselines—Projected Denoising and Post-Sampling Refinement—the proposed method achieves the highest GSR, SR, and OSR in all tested scenes. PD attains comparable or higher constraint satisfaction but lower GSR due to local optimization over joint samples without iterative optimal-IK selection, requiring large batches for adequate exploration. PSR ignores the target distribution during refinement and yields the lowest GSR except near workspace boundaries.

Sensitivity analyses indicate robust behavior: performance saturates quickly in $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$5 and $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$6; the SoftPlus sharpness $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$7 stabilizes metrics given SDF magnitudes of $$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$8–$$\mathrm{KL}(\pi \Vert p_\theta) + \lambda_g \mathbb{E}_{x\sim\pi}[g(\bm{x})]$$9 m; ten DDIM steps suffice for near-maximal OSR; and the number of enumerated IK solutions matters sharply for UR5—fewer than four solutions degrade FR to 2.74% because solutions lie in disconnected configuration-space sub-manifolds, whereas Franka's connected solution set is less affected. Inference costs approximately 0.5–2 ms per successful grasp on a single RTX 3090 Ti, faster than rejection sampling per successful grasp because fewer samples are discarded; SDF queries dominate runtime (56.8% of component time).

Real-world validation

Using a LEAP Hand on UR5 with an Azure Kinect depth camera and a learned grasp-quality evaluator (87% test-set prediction accuracy), the authors execute grasps of eight everyday objects in corridor and shelf scenes, reporting top-10 success counts between 6/10 and 10/10 per object. Practical demonstrations include downward grasps in deep containers, side grasps behind obstacles, and collision-aware bi-manual handover. For consecutive regrasping, the joint proximity guidance reduces average joint-space distance between successive grasps from 0.88 rad to 0.56 rad. Failure analysis attributes 7 of 20 failures to guidance-degraded edge grasps, 9 to open-loop execution and planning inaccuracies, and 4 to mass/friction estimation errors—the latter two outside the scope of the grasp generation method itself.

Limitations and open questions

The paper is candid about several limitations. Guidance perturbs the learned distribution and can degrade grasp quality, particularly when the arm remains in collision until late in denoising, creating conflict between constraint satisfaction and convergence to high-quality grasp regions; the proposed null-space projection recovers GSR but reduces FR with no net SR gain, so maintaining quality without sacrificing feasibility is unresolved. Because guidance-induced refinement is inherently local, the method primarily converts near-feasible samples into feasible ones and relies on multiple initial samples for global coverage; integrating guidance with improved initial sampling strategies is left as future work. The choice between gradients at $p_\theta$0 versus $p_\theta$1 is scene-dependent rather than settled. Finally, the evaluator's sensitivity to noisy, incomplete real-world point clouds limits downstream filtering reliability.

Conclusion

This work provides a mathematically grounded bridge between arm-agnostic dexterous grasp priors and arm-level feasibility, showing that closed-form constraint gradients derived through damped-pseudo-inverse IK sensitivity can be injected into guided diffusion sampling to raise feasible-grasp rates by factors of three to six in highly constrained scenes, across arms and environments, with no retraining. The residual tension between constraint satisfaction and grasp quality, and the locality of gradient-based correction, define the open problems this framework leaves for subsequent research.