Dex4D: Sim-to-Real Dexterous Manipulation

• Dex4D is a sim-to-real dexterous manipulation framework that formulates tasks as goal-conditioned MDPs with dense point cloud-based reward shaping.
• It employs an AP2AP objective with a teacher–student policy, using video-based 3D point tracking and transformer networks to achieve robust real-world generalization.
• Empirical evaluations demonstrate significant improvements in success rate and task progress over baselines, highlighting its effectiveness across diverse tasks and object scenarios.

Dex4D is a sim-to-real dexterous manipulation framework that enables a robot hand to perform diverse object-centric tasks by leveraging a task-agnostic point track policy. Dex4D’s approach is centered around learning in simulation via an Anypose-to-Anypose (AP2AP) objective, then employing a novel combination of video-based 3D point track generation and transformer-based policy to achieve robust real-world generalization and zero-shot deployment without additional finetuning. Dex4D’s contributions span a goal-conditioned Markov Decision Process (MDP) formulation, dense reward shaping using point clouds, a teacher–student policy distillation mechanism, and a pipeline for flexible closed-loop deployment across previously unseen objects and tasks (Kuang et al., 17 Feb 2026).

1. Markov Decision Process Formulation and Reward Shaping

Dexterous manipulation is formulated as a goal-conditioned Markov Decision Process

$$\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma, \mathcal{G} \rangle,$$

where the state $s_t$ comprises robot proprioception (joint angles $\theta_t$, velocities $\dot\theta_t$), the previous action $a_{t-1}$, and optionally privileged quantities (torques, fingertip forces/distances), though at deployment only partial 3D object observations (“paired points”) are available. The action space $\mathcal{A}\subset\mathbb{R}^{22}$ includes 6-DoF incremental arm joint targets and 16-DoF LEAP finger joint angles. The transition function $\mathcal{T}$ is provided by physics simulation (Isaac Gym) or real robot dynamics. Goals $\mathcal{G}$ are represented as sets of 3D object points at the desired pose.

The AP2AP objective is to learn a goal-conditioned policy $\pi(a_t|s_t, g_t)$ that can manipulate any object from any initial pose to any target pose, specified by goal $g$. Optimization is performed via

$s_t$0

with Proximal Policy Optimization (PPO) in simulation to obtain a teacher policy, followed by DAgger for distillation to a student policy under partial observations.

Rewards are not based on direct pose error but instead use dense feedback from surface point clouds. At each timestep, $s_t$1 points $s_t$2 are uniformly sampled on the object’s surface and compared to the corresponding target points $s_t$3; the average distance $s_t$4 between corresponding points informs several reward terms. The aggregate reward is

$s_t$5

where each component captures aspects such as goal proximity, fingertip–object distances, energy minimization, and action regularization. Rewards are stage-weighted during training to facilitate curriculum progression.

2. Neural Policy Architecture and Paired Point Encoding

Dex4D utilizes a teacher–student pipeline, with the teacher policy trained with privileged information and full observability, and the student policy operating under partial observability suitable for deployment.

Teacher Network:

• Inputs: privileged state and Paired Point Encoding, where pairs $s_t$6 are processed with shared MLPs and aggregated using mean-max pooling (PointNet style) to obtain a goal feature $s_t$7.
• Actor and critic comprise 4-layer MLPs ([1024, 1024, 512, 512]), taking as input [proprioception; last action; $s_t$8] and outputting the action or value estimate.

Student Network:

• Inputs: tokens from proprioception, previous action, and masked paired points ($s_t$9, masking to simulate occlusions).
• Each data type is tokenized via separate MLP or small PointNet into $\theta_t$0 embeddings.
• The token sequence is processed with a 4-layer transformer encoder (multi-head self-attention).
• Outputs are generated by two heads: action prediction (from the action token, yielding a 22D action vector) and a dynamics head (from state tokens, for predicting the next state).
• Training employs a combination of imitation and dynamics losses:

$\theta_t$1

Inference proceeds by tokenizing inputs, processing them through the transformer, then decoding the action from the corresponding token.

3. Simulation Training Regimen and Policy Distillation

The training pipeline begins with 3,200 diverse object meshes from UniDexGrasp, each randomized in initial pose. A multi-stage curriculum is employed:

• Stage 1 (0–15k PPO steps): single “bottle” type, high arm speed, simple resets.
• Stage 2 (15–25k): slower arm, more tolerant resets.
• Stage 3 (25–50k): full object diversity, conservative learning rate, 5 Hz control.

Domain randomization is applied for observation noise, action noise, PD gains, friction, mass parameters, and external perturbations (random linear/angular velocity every 4 s).

Teacher Training: conducted with PPO (4,096 parallel environments; $\theta_t$2, KL target $\theta_t$3, GAE $\theta_t$4, $\theta_t$5). Point cloud size is 128 for the teacher, with training completed in 2–3 days on an A6000 GPU.

Student Distillation: DAgger is used for 25k steps in 3,200 environments, with 64-point masked paired point encoding, transformer depth 4, and learning rate $\theta_t$6. Total training time is approximately 20 hours.

Training curves demonstrate rapid mean reward convergence for the teacher, with ablation studies showing clear advantage for Paired Point Encoding and transformer-based policy over alternative designs.

4. Sim-to-Real Zero-Shot Deployment via Video-Based Point Tracks

At deployment, Dex4D leverages a video-driven planning paradigm to obtain object-centric point tracks specifying desired goal poses:

• Video Generation: Given a language instruction $\theta_t$7 and a single depth-RGB frame $\theta_t$8, a video $\theta_t$9 is generated using Wan2.6. An initial object mask is computed via SAM2, after which CoTracker3 tracks $\dot\theta_t$0 2D points over the video. Video-Depth-Anything yields per-frame relative depths. Median matching calibrates depth to metric scale, enabling lifting of 2D tracks $\dot\theta_t$1 to 3D tracks $\dot\theta_t$2.

• Online Execution (Real Robot): The control loop utilizes a 6-DoF xArm6 with 16-DoF LEAP hand, RealSense D435 for depth, and Apriltags for calibration.

  1. Target point set $\dot\theta_t$3 initialized; CoTracker3 run online to obtain current 2D locations, backprojected to 3D.
  2. Paired points $\dot\theta_t$4 (masked if occluded), together with proprioceptive tokens and previous action, are input to the student policy.
  3. Predicted action $\dot\theta_t$5 is executed on the robot.
  4. If average distance $\dot\theta_t$6, the next point in the track is selected as the target.
  5. Loop repeats until completion of the point track sequence.

No real-world finetuning is conducted at any stage.

5. Empirical Evaluation and Results

Simulation experiments span six tasks: Apple2Plate, Pour, Hammer, StackCup, RotateBox, Sponge2Bowl, evaluated by average success rate (SR) and task progress (TP, average waypoints reached):

Method	SR-avg	TP-avg
NovaFlow	0.345	0.448
NovaFlow-CL	0.437	0.608
Dex4D	0.600	0.712

Closed-loop (CL) planning baselines improve over open-loop by +9.2% SR and +16% TP. Dex4D provides an additional +16.3% SR and +10.4% TP over closed-loop motion planning. Ablations (MLP encoding, nonpaired features, absence of attention or world modeling) result in significant performance reductions, confirming the importance of paired point encoding, transformer attention, and explicit action-world model design.

Real-world experiments across four tasks (LiftToy, Broccoli2Plate, Meat2Bowl, Pour; 10 trials/task) yield:

Method	Successes (n=40)
NovaFlow-CL	10
Dex4D	19 (+22.5% SR)

Qualitative observations indicate Dex4D executes dynamic grasp correction and can handle severe occlusions ($\dot\theta_t$710 visible points), while baseline methods frequently fail due to noisy Kabsch estimation or object drops. Dex4D also generalizes to unseen geometries, variable layouts, backgrounds, camera viewpoints, and external perturbations.

6. Limitations and Prospects for Future Work

Primary limitations include point tracker (CoTracker3) failures under severe motion or ambiguous texture, which are a major source of deployment errors. The policy can occasionally exhibit over-grasping “pushing” behavior, leading to undesired knock-off of small objects. Current AP2AP formulation handles only single rigid objects, with no support for articulated or multi-object tasks. The system does not use tactile sensing or human grasp priors, and LEAP hand embodiment introduces additional gaps. Future research directions include improved 3D tracking, tactile feedback integration, utilization of human-demonstration grasp priors, and extension to articulated, stacked, or multi-object planning domains.

Dex4D demonstrates that a task-agnostic, point-track-conditioned policy trained entirely in simulation with AP2AP objectives can be flexibly recomposed at deployment using video-generated 3D point tracks, providing robust, generalizable, zero-shot dexterous manipulation on real robotic hardware (Kuang et al., 17 Feb 2026).