DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Abstract: Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments. However, existing benchmarks remain limited in task and data diversity, embodiment coverage, or controllable visual variation, hindering studies of cross-task and cross-embodiment generalization. We present DexVerse, a large-scale and modular benchmark for dexterous manipulation. DexVerse includes 100 tasks spanning a broad range of manipulation skills, including object grasping and relocation, articulated-object interaction, functional tool use, bimanual coordination, non-prehensile control, contact-rich behaviors, multi-goal execution, and long-horizon multi-stage task completion. It supports 3 robot arms and 6 dexterous hands, and is extensible to new tasks, assets, and embodiments. To evaluate visuomotor generalization, DexVerse provides configurable visual variations in textures, background, lighting, and camera viewpoints. We further provide a VR-based teleoperation interface and 3,180 demonstrations with synchronized proprioceptive, RGB, depth, point-cloud, and state observations. We benchmark representative methods, including Diffusion Policy, DP3, OpenVLA, and $π_{0.5}$, across 19 tasks. Results reveal substantial challenges in task generalization and visuomotor robustness, establishing DexVerse as a promising testbed for general-purpose dexterous manipulation. Project page: https://ycyao216.github.io/DexVerse.site

• The paper introduces DexVerse, a benchmark that decouples tasks from embodiments to systematically evaluate 100 diverse dexterous manipulation tasks.
• It employs a configuration-driven modular design and a scalable VR-based teleoperation system to collect 3,180 multimodal trajectories across varied settings.
• The evaluation reveals that current methods struggle with fine-contact tasks, underscoring the need for improved sensory and embodiment alignment in dexterous policy development.

DexVerse: A Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Motivation and Related Work

The absence of benchmarks that span multiple axes of dexterous generalization—broad task type coverage, multi-embodiment support, controllable visual variation, and synchronized teleoperation demonstrations—critically restricts progress toward general-purpose dexterous manipulation policies. Prevailing benchmarks emphasize either gripper-based, isolated-task manipulation (e.g., CALVIN, LIBERO) or restricted-object, single-embodiment dexterous tasks (e.g., DexArt, Bi-DexHands, DexJoCo), with limitations in embodiment diversity, visual domain control, or unified demonstration-data collection. This separation impedes analysis of policy robustness under both sensory and embodiment shifts, essential for deploying dexterous skills in real environments.

DexVerse directly targets these deficits, providing 100 tasks with wide interaction diversity, 3 arms and 6 dexterous hands, scientifically structured visual variation, parallel RL support, a 3,180-trajectory multimodal dataset, and canonical observation interfaces for optimal comparison of competing methods. Unlike prior datasets, DexVerse decouples tasks from embodiments via configuration-driven modular design, supporting extensibility and systematic evaluation of multi-task and multi-embodiment transfer regimes.

DexVerse Environments and Task Taxonomy

DexVerse environments are architected for modularity and generalizability. Each environment instance is specified by structured configuration: its scene, assets, robot configuration, sensory interfaces, and randomization regimes are parametrized and composable. Task definitions follow a formal tuple design $$\mathcal{T} = (\Omega,\mathcal{S}_0,\mathcal{O},\mathcal{A},\mathcal{G})$$, with precise success predicates, supporting abstraction over observation/actuation modalities.

DexVerse enumerates 100 tasks grouped into 8 manipulation categories:

• Primitive: Atomic pick/place and interaction primitives.
• Functional: Object-specific activities exploiting physical affordances.
• Articulation: Control of objects with jointed components.
• Non-prehensile: Tactile and environmental contact-based regulation.
• Contact-rich: Fine-alignment, force-control, multi-contact demands.
• Bimanual Coordination: Dual-arm-hand tasks enforcing explicit stabilization or handover constraints.
• Multi-goal: Compositional tasks requiring simultaneous or sequential goal satisfaction.
• Long-horizon: Temporally extended, multi-stage procedures.

  Figure 1: Visualization of selected tasks from the DexVerse environments.

This categorization emphasizes interaction complexity as opposed to strictly object or actuator type. In addition to primitives, non-prehensile and contact-rich regimes expose failure modes not adequately revealed by gripper benchmarks, notably under real-world deployment conditions.

Figure 2: Visualization of task progression of the 5 long-horizon tasks in DexVerse environments.

The decoupling of environment logic from embodiment specification allows for transparent swapping-in of distinct hand-arm pairs, with default and override configurations for each task class. Asset libraries are assembled from SAPIEN, ManiTwin, Isaac Sim, and synthesized sources, expanded with image-to-3D mesh generation for coverage completeness.

Multi-Embodiment and Visual Variation

DexVerse abstracts robot embodiments as composable, parametrizable configurations. Combinations of Franka Research 3, UR10e, and xArm 7 arms with Sharpa Wave, WUJI, Shadow, Inspire, Allegro, and LEAP hands are fully specified, with explicit joint count, morphology, workspace, and dynamic limits. Both anchored and floating-hand variants are supported for systemic ablation.

Visual aspects—including object/textures, environment backgrounds, lighting, and viewpoint parameters—are independently randomized per episode for policy generalization benchmarking. Non-visual stochasticity is also configurable (object pose, actuation properties), supporting robust domain randomization and transfer studies.

Demonstration Corpus: Teleoperation Interface and Dataset Specification

A scalable VR-based demonstration collection pipeline enables rapid dataset expansion over new embodiments. Apple Vision Pro, integrated via CloudXR and Isaac Lab APIs, streams simulated feedback to the operator and captures hand/wrist pose for IK-based robot control. Embodiment adaptation is systematically solved using optimization-based dex-retargeting.

3,180 trajectories are produced across 61 tasks and 6 hand types, each trajectory recording action-state pairs with full synchronized multimodal observation streams (proprio, RGB, depth, point-cloud, simulator state). Environments can replay exact states to avoid simulation divergence. The observations are grouped by accessible policy input modes (policy, proprio, contact, rgb, depth, pointcloud, etc.), with observation-mode presets for controlled evaluation of modality contributions.

Baseline Policy Evaluation

The baseline analysis benchmarks four representative policy families on a subset of 19 tasks:

1. $\pi_{0.5}$: Vision-language-action (VLA) transformer, web-scale pretrained, fine-tuned for chunk-based action prediction via flow matching.
2. OpenVLA: Large-scale open-source VLA, modified to regress continuous joint targets.
3. Diffusion Policy (DP): State-only, per-task conditional action diffusion, chunk-based planning.
4. 3D Diffusion Policy (DP3): Point-cloud plus proprioception conditioning.

Evaluation is closed-loop, 50 rollouts per task, standardized observation/action regimes, no privileged information.

Numerical Results and Findings

• Mean online success rate peaks at 34% (DP3 and $\pi_{0.5}$), with OpenVLA lagging at 19%. No method dominates across all skill families; performance is highly skill-dependent.
• VLA web-scale pretraining does not confer generalization advantage in high-DoF dexterous control; hand/arm control priors from internet data do not readily transfer beyond perception.
• No single observation paradigm is optimal: 2D vision suffices for Pick-and-Lift, 3D geometry is critical for Functional Tool Use, and language/flow-matching helps Articulation/Precision regimes.
• All methods fail on precision contact and sub-centimeter alignment: Tasks such as PushT, InsertPen, and SlideUtilityKnife achieve nearly zero success, exposing open gaps in fine contact, sustained force reasoning, and policy transfer beyond behavioral cloning.

Implications and Future Directions

Practically, DexVerse exposes the unsolved nature of general-purpose dexterous manipulation policies. Despite recent advances, policies fail to generalize across both visual domains and robot morphologies, and current architectures are fundamentally limited in high-dimensional, fine-contact regimes. The non-dominance of VLA models over appropriately structured diffusion policies calls for systematic study of sensory and embodiment alignment, and the explicit ablation facilities in DexVerse enable hypothesis-driven disentangling of modality and embodiment priors.

The DexVerse environment and dataset provide a scalable foundation for novel architecture development (e.g., hierarchical models, contact-proprio fusion, embodiment-aware latent spaces), as well as for benchmarking approaches to cross-domain adaptation, robust RL with sparse rewards, and real-to-sim transfer. Future work should expand real-robot evaluation, multi-embodiment demonstration variety, and task compositionality for broader RL and imitation learning baselines.

Conclusion

DexVerse defines a rigorous, extensible benchmark for systematic study of dexterous multi-task, multi-embodiment manipulation. With structured variation across task, embodiment, and observation regimes, it directly exposes persistent failures in current policy learning and lays the groundwork for advancing robust, generalizable dexterous robot intelligence.