DexEvolve: Evolutionary Optimization for Robust and Diverse Dexterous Grasp Synthesis

Abstract: Dexterous grasping is fundamental to robotics, yet data-driven grasp prediction heavily relies on large, diverse datasets that are costly to generate and typically limited to a narrow set of gripper morphologies. Analytical grasp synthesis can be used to scale data collection, but necessary simplifying assumptions often yield physically infeasible grasps that need to be filtered in high-fidelity simulators, significantly reducing the total number of grasps and their diversity. We propose a scalable generate-and-refine pipeline for synthesizing large-scale, diverse, and physically feasible grasps. Instead of using high-fidelity simulators solely for verification and filtering, we leverage them as an optimization stage that continuously improves grasp quality without discarding precomputed candidates. More specifically, we initialize an evolutionary search with a seed set of analytically generated, potentially suboptimal grasps. We then refine these proposals directly in a high-fidelity simulator (Isaac Sim) using an asynchronous, gradient-free evolutionary algorithm, improving stability while maintaining diversity. In addition, this refinement stage can be guided toward human preferences and/or domain-specific quality metrics without requiring a differentiable objective. We further distill the refined grasp distribution into a diffusion model for robust real-world deployment, and highlight the role of diversity for both effective training and during deployment. Experiments on a newly introduced Handles dataset and a DexGraspNet subset demonstrate that our approach achieves over 120 distinct stable grasps per object (a 1.7-6x improvement over unrefined analytical methods) while outperforming diffusion-based alternatives by 46-60\% in unique grasp coverage.

• The paper introduces an asynchronous, simulator-in-the-loop evolutionary optimizer that directly refines wrist poses, joint states, and closing commands using robustness, contact, and diversity objectives.
• DexEvolve generates more than 120 distinct stable grasps per object—1.7–6× more than analytical seeds—and achieves 46–60% higher unique-grasp coverage than diffusion models trained on those seeds.
• The method supports non-differentiable human-preference rewards and diffusion distillation, enabling natural-looking grasp refinement, partial-observation inference, and successful real-world cabinet-handle manipulation.

Overview and Motivation

DexEvolve addresses a persistent bottleneck in dexterous grasp synthesis: the separation between grasp generation and physical validation. Analytical methods based on force-closure optimization, including conic programming formulations (2002.01530) and differentiable relaxations such as GraspQP (Zurbrügg et al., 20 Aug 2025), produce diverse candidates under simplified contact models, while high-fidelity simulators like Isaac Sim are typically used only as post-hoc accept/reject filters. This two-stage paradigm is sample-inefficient: most proposals are discarded during verification, and surviving grasps remain suboptimal because generation never receives direct physical feedback. The authors argue that high-fidelity simulation should instead be treated as a black-box objective to be optimized directly, rather than as a binary filter.

The core technical obstacle to direct simulator-based refinement is that physics evaluation is expensive (requiring force application over a rollout horizon) and non-differentiable in engines like PhysX, precluding gradient-based optimization. Prior evolutionary approaches to grasping have been restricted either to parallel-jaw setups or to wrist-pose optimization with fixed hand configurations (Huber et al., 2024). DexEvolve extends simulator-in-the-loop refinement to full dexterous grasp configurations—wrist pose, joint states, and closing commands—for a multi-fingered hand.

Method

Grasp representation and seed initialization

A grasp is parameterized as $G = (\chi, q, \Delta q_{cmd})$, where $\chi \in SE(3)$ is the wrist pose, $q \in \mathbb{R}^{n_q}$ are joint angles, and $\Delta q_{cmd}$ is the delta joint command for gripping force. The pipeline initializes from an analytically generated seed set produced by GraspQP, though the framework is agnostic to the seeding strategy; random initialization is also evaluated.

Asynchronous evolutionary refinement

Refinement runs an asynchronous, massively parallel genetic algorithm directly in Isaac Sim, with early rejection of failing rollouts. Three components distinguish it from naive evolutionary search:

Novelty-gated archive insertion: A candidate is inserted only if its distance to the nearest archive member exceeds a threshold $\tau$; otherwise it replaces that neighbor only if it improves fitness. Distance is a weighted metric over wrist position, orientation (Euler angles), and joint configuration. This quality-diversity mechanism, following MAP-Elites-style novelty gating (Mouret et al., 2015), explicitly suppresses mode collapse.

Density-aware tournament selection: Fitness is reweighted by inverse local density within a ball of radius $r$, so individuals in crowded regions of grasp space are selected less often. In the degenerate case where all grasps coincide, selection weight drops by a factor of $|\mathcal{G}|$; isolated individuals retain raw fitness. The authors note that insertion and selection serve complementary purposes: selection steers reproduction toward promising but underexplored regions, while insertion guarantees final population diversity.

Structured crossover and adaptive contact resampling: Naive gene-wise crossover of full grasp configurations produces mostly infeasible offspring. Instead, crossover swaps entire finger configurations between parents at a fixed split point, exploiting the observation that grasps cluster into taxonomies with similar finger configurations. For every offspring, active contact points are recomputed via ball query plus farthest point sampling ($n_c = 12$), and closing commands are solved by least squares against desired normal forces through the stacked contact Jacobians—avoiding rejection of valid grasps due to stale contact assignments.

The fitness function combines an environment-lifetime score (robustness under sequential disturbance forces along all six canonical axes), a weak SDF-based distance-to-contact penalty, and an interpenetration penalty, evaluated over up to 10,000 asynchronous environment steps with an archive cap of 1,024.

Gradient-free steering

Because the optimizer never requires gradients of the objective, arbitrary non-differentiable signals can be folded into fitness. The paper demonstrates human preference alignment: roughly 1,000 pairwise annotations of grasp naturalness train a PointNet++ reward model under a Bradley-Terry loss (with a tie regularizer for "similar" labels), which is added to the fitness with weight 10. Qualitatively, unguided refinement yields stable but awkward grasps (e.g., approaching handles at unnatural angles), whereas preference-guided refinement recovers human-like approach directions and finger placements without sacrificing stability.

Diffusion distillation

For deployment from partial observations, the refined distribution is distilled into a conditional diffusion model built on the DexGraspAnything architecture [zhong2025dexgrasp], with three extensions: it diffuses the complete grasp state—a 21-dimensional vector comprising translation, a 6D rotation representation, and twelve joint angles—rather than position only; it conditions on both the observed point cloud and forward-kinematics-derived hand keypoints; and it adds keypoint-consistency and SDF-based penetration losses, with the penetration term exponentially decayed across diffusion timesteps so collision penalties dominate only near clean samples. The model (~9.7M parameters) trains in about two days on a single RTX 4090. Training-time augmentations include normal-biased depth noise, occlusion spheres, and SE(3) augmentation (restricted to handle-axis rotations for handles).

Handles dataset

The paper contributes 90 geometrically distinct door-handle and knob assets modeled on commercial IKEA products (190 with texture variations), released as articulated USD assets for Isaac Sim. Collision meshes are recursively subdivided to at most 1 cm triangle edges because coarse meshes cause interpenetration artifacts under Isaac Sim's SDF-based contact handling. Each handle carries a prismatic joint along z plus two compliant auxiliary joints with ±1.5 cm travel; this compliance deliberately rejects weak grasps that hold the handle by only millimeters, pushing solutions toward stronger force closure. The dataset fills a gap relative to prior synthetic handle assets (Wallach, 2019) by pairing assets with grasp predictions and physically reproducible hardware.

Experimental results

Evaluation uses the XHand gripper, a 50-asset subset of DexGraspNet objects [wang2023dexgraspnet], and 50 handles assets, with stability assessed via the standard disturbance protocol in Isaac Lab. Metrics are success rate, DistinctStableGrasps (DSG) at three quantization resolutions, and marginal entropies over position, orientation, and joints.

Three headline findings emerge:

Comparison	Result
Evolutionary vs. analytical seeds	>120 distinct stable grasps per object, a 1.7–6× improvement, with DSG@20cm near saturation
Evolutionary vs. diffusion trained on seeds	46–60% higher unique grasp coverage (115 vs. 72 unique grasps at 32 seeds; 118 vs. 81 at 128 seeds)
Entropy	Refinement achieves mean entropy ≈3.0 vs. 2.4–2.6 for diffusion

Two claims deserve emphasis. First, stability gains do not trade off against diversity: refined distributions preserve or increase entropy, indicating extrapolation beyond the seed manifold rather than mere consolidation. Second—and presented by the authors as surprising—evolutionary refinement initialized from random populations reaches competitive success rates and unique-grasp counts even when zero random seeds succeed initially. The caveat is material: randomly initialized runs exhibit markedly lower orientation and joint entropy, so analytical seeding remains important for multimodal coverage. Convergence analysis shows both stability and diversity metrics plateau by roughly 10k environment steps after rapid gains in the first 2.5k steps, supporting the method's sample efficiency.

Diffusion baselines improve substantially over raw seeds themselves (≈20 → 71 unique grasps at 32 seeds), confirming that generative interpolation expands feasible coverage—but they remain strictly below simulator-in-the-loop refinement, since implicit learned priors cannot enforce contact dynamics the way explicit rollouts do. Diffusion remains essential for inference speed and partial-observability conditioning, so the two stages are complementary rather than competing.

Real-world deployment

The system is validated on a Franka Panda arm with an XHand. Perception fuses seven RGB-D views using Depth Anything V3 monocular depth lifting (Lin et al., 13 Nov 2025), co-aligned with sensor depth for scale consistency—an approach motivated by specular and transparent surfaces that defeat raw depth sensors. Grasps are sampled from the diffusion model with optional test-time penetration-gradient guidance, planned with cuRobo MotionGen over voxelized scene geometry (typically succeeding within 15 s), and executed under joint impedance control. Multiple successful cabinet-opening sequences across knob, bar, and shell handle styles demonstrate sim-to-real transfer across distinct grasp modes. A supplementary reachability study finds that 7-DoF arms retain 20+ reachable grasps in worst-case handle poses versus fewer than 10 for 6-DoF arms, providing a quantitative justification for redundant kinematics in dexterous manipulation.

Limitations

The paper is candid about three constraints. First, recovery from poor initializations comes at a diversity cost: random-seeded refinement produces lower-entropy distributions than analytically seeded runs, so strong initialization materially affects final manifold coverage. Second, PhysX does not reliably resolve contacts for initially colliding configurations, forcing explicit de-penetration steps after each mutation (up to two gradient-descent steps on penetration energy with clipped step sizes), which adds non-trivial overhead. Third, the deployment pipeline's multi-stage structure—multi-view capture, monocular depth lifting, fusion, diffusion sampling, motion planning—is too slow for responsive end-to-end operation. An additional unstated dependency is that the disturbance-based success criterion and the compliant-handle design jointly define "stability"; results may not transfer to evaluation protocols with different disturbance profiles.

Conclusion

DexEvolve demonstrates that reframing high-fidelity simulation as a differentiable-free optimization target—rather than a verification filter—yields large gains in both the quantity and diversity of physically feasible dexterous grasps: over 120 distinct stable grasps per object (1.7–6× over analytical seeds), superior to diffusion-trained alternatives by 46–60% in unique coverage, with successful hardware transfer. Its gradient-free structure additionally admits non-differentiable steering signals, demonstrated through human preference alignment. The open question the work leaves is whether the observed dependence on seed quality for final diversity can be removed—for instance, by combining random initialization with stronger exploration operators—or whether analytical seeding will remain a necessary ingredient for covering the full grasp manifold.