---
title: 'NestDex: Nested Policy Learning for Dexterous Manipulation'
url: https://www.emergentmind.com/topics/nestdex
type: topic
---

# NestDex: Nested Policy Learning for Dexterous Manipulation

NestDex is a nested policy-learning framework for dexterous robotic manipulation that uses learned hand skills as copilots during teleoperated demonstration collection and then trains an autonomous outer visuomotor policy from the resulting complete-task demonstrations. The operator controls task-level arm motion, while reusable proprioceptive hand policies generate coordinated finger behavior; a single-degree-of-freedom clutch regulates skill progress, and a vision-language selector activates skills at task transitions. At deployment, the inner hand policies, clutch, and selector are removed: the outer policy directly predicts arm commands and compact hand-action representations from visual and proprioceptive observations [2608.13362].

## 1. Concept and system architecture

NestDex addresses the difficulty of collecting demonstrations for dexterous manipulation. Unlike parallel-jaw manipulation, dexterous tasks require simultaneous coordination of arm motion and contact-rich, multi-finger behavior throughout a complete task. Demonstrations must be complete, successful, temporally consistent, and contact-aware. Direct teleoperation of all finger joints can impose a substantial motor and cognitive burden on the operator.

The framework separates demonstration collection from autonomous deployment through two nested control and learning loops:

1. **Inner loop**: learned hand policies assist the operator during teleoperation.
2. **Outer loop**: an autonomous visuomotor policy is trained from the assisted demonstrations.

The inner loop comprises reusable hand skills. Human demonstrations are retargeted to the robot hand, and a proprioceptive action-chunk policy is trained for each skill. During complete-task teleoperation, the operator moves the arm directly, while the active hand policy generates fine-grained finger commands. A one-dimensional clutch advances, pauses, or reverses the active skill.

The outer loop records images, robot states, arm commands, and hand commands from the assisted demonstrations. Hand commands are compressed with a hand-action variational autoencoder (H-VAE), and an outer visuomotor policy is trained by behavior cloning. At deployment, the outer policy predicts arm commands and latent hand actions; the H-VAE decoder converts the latter into 20-degree-of-freedom hand joint-position commands.

The nesting is therefore principally a property of the training and data-collection pipeline. The final deployed controller is not a hierarchy that invokes the inner policies sequentially. Instead, the inner policies function as data-collection copilots whose behavior is distilled into a single outer policy.

## 2. Hardware and operator interface

The experiments use a leader–follower platform. Each arm–hand pair consists of a 7-DoF Piper Nero leader arm, a 1-DoF leader-side clutch, a matching 7-DoF Piper Nero follower arm, a 20-DoF five-finger WujiHand I, and a wrist-mounted camera.

The follower arm state is

$$
\mathbf{s}^{\mathrm{arm}}_t =
\left[
\mathbf{q}^{\mathrm{arm}}_t,
\mathbf{e}^{\mathrm{arm}}_t
\right],
$$

where $\mathbf{q}^{\mathrm{arm}}_t\in\mathbb{R}^{7}$ is the arm joint-position vector and $\mathbf{e}^{\mathrm{arm}}_t\in\mathbb{R}^{7}$ is the arm joint-effort vector. The hand state is

$$
\mathbf{x}_t =
\left[
\mathbf{q}_t,
\mathbf{e}_t
\right],
$$

where $\mathbf{q}_t\in\mathbb{R}^{20}$ and $\mathbf{e}_t\in\mathbb{R}^{20}$ denote hand joint positions and joint efforts. Inner policies execute at 100 Hz.

Leader-arm joint positions are mapped directly to follower-arm joint-position commands:

$$
\mathbf{a}^{\mathrm{arm}}_t =
\mathbf{q}^{\mathrm{leader}}_t.
$$

The operator consequently remains responsible for gross arm motion, including approach, transport, tool manipulation, and placement. Finger motion is generated by the active hand policy rather than specified directly through the leader arm.

### Clutch-mediated skill progression

Let $c_t$ denote the current clutch value, and let $c_{\mathrm{start}}$ and $c_{\mathrm{end}}$ denote the values associated with the beginning and end of a skill. Normalized skill progress is

$$
p_t =
\operatorname{clip}
\left(
\frac{c_t-c_{\mathrm{start}}}
{c_{\mathrm{end}}-c_{\mathrm{start}}},
0,1
\right).
$$

If $T_{\max}$ is the maximum trajectory length among successful demonstrations of the skill, the clutch-requested execution index is

$$
s_t^{\mathrm{in}} =
\left\lfloor
p_t(T_{\max}-1)
\right\rfloor.
$$

The actual generated trajectory index, $r_t^{\mathrm{in}}$, changes by at most one index per control cycle:

$$
r_{t+1}^{\mathrm{in}}
=
r_t^{\mathrm{in}}
+
\operatorname{clip}
\left(
s_t^{\mathrm{in}}-r_t^{\mathrm{in}},
-1,1
\right),
\qquad
r_0^{\mathrm{in}}=0.
$$

This mechanism makes skill execution reversible. Positive clutch motion advances the skill, releasing or reversing the clutch moves backward through generated commands, and a zero index change holds the current posture. The operator cannot skip a large portion of the learned trajectory. On reversal, the ensemble-prediction buffer is cleared while the generated trajectory buffer is retained; when forward execution resumes, the policy receives the latest proprioceptive state rather than replaying an abandoned future command sequence.

## 3. Inner hand-skill policies

NestDex trains separate inner policies for reusable hand skills, including Tongs Grasp, Bottle Grasp, Dual-Object Grasp, Scallion Pinch, Pot Grasp, Button Press, Plate Grasp, Paper Pinch, Hole-Punch Press, and Binder Closure.

### Demonstration retargeting

Human hand demonstrations are observed using multiple calibrated cameras. Synchronized hand keypoints are triangulated across views to reduce single-view depth ambiguity, finger occlusion, and unreliable three-dimensional reconstruction.

The reconstructed human motion is retargeted to the robot hand through vector correspondences. At each time step, the robot-hand configuration is obtained by solving a constrained optimization problem:

$$
\mathbf{q}^{*}_t =
\underset{
\mathbf{q}_{\min}\leq \mathbf{q}\leq \mathbf{q}_{\max}
}{\arg\min}
\left[
\sum_{i=1}^{M}
\rho_{\delta}
\left(
\left\|
\alpha\mathbf{v}^{h}_{i,t}
-
\mathbf{v}^{r}_{i}(\mathbf{q})
\right\|_2
\right)
+
\beta
\left\|
\mathbf{q}-\mathbf{q}^{*}_{t-1}
\right\|_2^2
\right].
$$

Here, $\mathbf{v}^{h}_{i,t}$ and $\mathbf{v}^{r}_{i}(\mathbf{q})$ are corresponding human- and robot-hand vectors, $\alpha$ is a hand-size scale factor, $\beta$ controls temporal smoothness, and $\rho_\delta$ is a Huber penalty. The joint-limit constraints restrict $\mathbf{q}$ to the robot’s allowable configuration range.

Each retargeted trajectory is represented as

$$
\tau_n =
\left\{
(\mathbf{q}_{n,t},\mathbf{e}_{n,t})
\right\}_{t=0}^{T_n-1}.
$$

The paper uses 10 trajectories to train each required inner policy. The main grasp-policy example uses a green scallion toy, a water bottle, a rounded or radish-like object, and a paper cup. One policy produces different hand configurations for these objects despite receiving no images or object labels during execution. The variation is conditioned on proprioceptive state and contact-dependent changes, although the objects were represented during training; unseen-object generalization is not established.

### Proprioceptive action-chunk prediction

The inner policy receives a history of hand positions and efforts rather than images or object identity. With

$$
\mathbf{x}_{n,t}
=
[
\mathbf{q}_{n,t},
\mathbf{e}_{n,t}
],
$$

and history length $h$, the observation is

$$
\mathbf{o}_{n,t}
=
[
\mathbf{x}_{n,t-h+1},
\ldots,
\mathbf{x}_{n,t}
].
$$

The policy predicts a future hand joint-position chunk of horizon $H_{\mathrm{in}}$:

$$
\mathbf{A}_{n,t}
=
[
\mathbf{q}_{n,t+1},
\ldots,
\mathbf{q}_{n,t+H_{\mathrm{in}}}
].
$$

The policy mapping is

$$
\widehat{\mathbf{A}}_{n,t}
=
\pi_{\mathrm{in}}
\left(
\mathbf{o}_{n,t}
\right).
$$

The policy is repeatedly queried during execution, allowing its output to adapt to the latest proprioceptive state. This is relevant when contact geometry, resistance, or the current hand posture differs from the nominal trajectory. The supplied description specifies supervised prediction of future joint-position chunks but does not provide a separate explicit scalar inner-policy loss or complete network architecture.

## 4. Skill selection, action chunking, and temporal ensembling

During complete-task teleoperation, a vision-language selector determines which reusable hand skill should be active at the current task stage. The selector activates skills such as:

- **Toast Preparation**: Tongs Grasp, Button Press, Plate Grasp, and Tongs Grasp again.
- **Binder Filing**: Paper Pinch, Hole-Punch Press, Paper Pinch again, and Binder Closure.

The selector enables skill reuse across separate stages. Once a skill is activated, it moves the hand toward its initial posture, while the operator regulates progress through the clutch. The exact vision-language architecture, prompt format, operating frequency, selection accuracy, and training loss are not specified. The evidence for this component is qualitative.

Both inner and outer policies predict action chunks. For a generic policy,

$$
\widehat{\mathbf{A}}_t
=
\pi_\theta
(\mathbf{v}_t,\mathbf{s}_t)
=
[
\widehat{\mathbf{a}}_{t+1\mid t},
\ldots,
\widehat{\mathbf{a}}_{t+H\mid t}
].
$$

The policy is queried at every control step, producing overlapping predictions for future execution times. If $\mathcal{P}_t$ is the set of predictions available for execution time $t$, the executed action is an exponentially weighted average:

$$
\mathbf{a}^{\mathrm{exec}}_t
=
\frac{
\sum_{i=0}^{|\mathcal{P}_t|-1}
w_i\mathcal{P}_t[i]
}{
\sum_{i=0}^{|\mathcal{P}_t|-1}w_i
},
\qquad
w_i=\exp(-mi),
$$

where $m>0$ controls the weighting of newer predictions.

This procedure combines two forms of adaptation. Closed-loop prediction conditions future commands on current proprioceptive state and contact, while temporal ensembling smooths transitions between overlapping action chunks. During reversal of a skill, the ensemble-prediction buffer is cleared, preventing stale predictions from influencing subsequent forward execution.

## 5. H-VAE and outer visuomotor policy

### Hand-action compression

The robot hand produces 20-dimensional joint-position commands. NestDex trains a hand-action variational autoencoder to compress these commands while leaving arm actions in joint space.

For a hand command $\mathbf{a}^{\mathrm{hand}}_k$, the encoder produces

$$
q_\phi
\left(
\mathbf{z}_k
\mid
\mathbf{a}^{\mathrm{hand}}_k
\right)
=
\mathcal{N}
\left(
\boldsymbol{\mu}_\phi
\left(
\mathbf{a}^{\mathrm{hand}}_k
\right),
\operatorname{diag}
\boldsymbol{\sigma}^2_\phi
\left(
\mathbf{a}^{\mathrm{hand}}_k
\right)
\right),
$$

where $\mathbf{z}_k\in\mathbb{R}^{d_z}$ and $d_z<20$. Using the reparameterization trick,

$$
\mathbf{z}_k
=
\boldsymbol{\mu}_\phi
\left(
\mathbf{a}^{\mathrm{hand}}_k
\right)
+
\boldsymbol{\sigma}_\phi
\left(
\mathbf{a}^{\mathrm{hand}}_k
\right)
\odot
\boldsymbol{\epsilon},
\qquad
\boldsymbol{\epsilon}
\sim
\mathcal{N}(\mathbf{0},\mathbf{I}).
$$

The decoder reconstructs the hand command:

$$
\widehat{\mathbf{a}}^{\mathrm{hand}}_k
=
D_\psi(\mathbf{z}_k).
$$

The H-VAE uses reconstruction and Kullback–Leibler regularization. After training, NestDex uses the posterior mean,

$$
\overline{\mathbf{z}}_k
=
\boldsymbol{\mu}_\phi
\left(
\mathbf{a}^{\mathrm{hand}}_k
\right),
$$

rather than sampling a latent code. The outer-policy action is therefore

$$
\mathbf{a}_k
=
[
\mathbf{a}^{\mathrm{arm}}_k,
\overline{\mathbf{z}}_k
].
$$

The supplied description does not specify the H-VAE architecture, latent dimensionality, or numerical weighting coefficient for the VAE objective.

### Outer policy

Each complete-task demonstration records a wrist-camera image, robot state, arm commands, and hand commands. The outer state is

$$
\mathbf{s}_t
=
[
\mathbf{q}^{\mathrm{arm}}_t,
\mathbf{e}^{\mathrm{arm}}_t,
\mathbf{q}^{\mathrm{hand}}_t,
\mathbf{e}^{\mathrm{hand}}_t
].
$$

The outer policy receives $(\mathbf{v}_t,\mathbf{s}_t)$ and predicts an action chunk:

$$
\widehat{\mathbf{A}}_t
=
\pi_\theta
(\mathbf{v}_t,\mathbf{s}_t).
$$

The chunk contains explicit arm joint-position commands and H-VAE latent hand commands. Behavior cloning optimizes a supervised action-prediction objective over the complete-task dataset.

At deployment, the outer policy predicts arm commands and hand latents, temporal ensembling combines overlapping predictions, the arm commands are sent directly to the follower arm, and the hand latents are decoded into 20-DoF hand commands. The inner policies and vision-language selector are absent from this execution loop.

## 6. Experimental evaluation and limitations

NestDex is evaluated on six real-world tasks:

| Task | Mode | Description |
|---|---|---|
| Tongs Transfer | Single-arm | Extract tongs, grasp a wooden carrot, and place it in a pan |
| Bottle Disposal | Single-arm | Open a bin lid, grasp an empty plastic bottle, and place it inside |
| Dual-Object Transfer | Single-arm | Grasp two wooden number blocks simultaneously and place them in a bin |
| Ingredient and Pot Transfer | Single-arm | Place a wooden scallion in a pot, then grasp and move the pot |
| Toast Preparation | Dual-arm | Use tongs, press a toaster button, position a plate, and return toast |
| Binder Filing | Dual-arm | Remove paper, punch it, insert it into a binder, and close the rings |

The principal teleoperation baseline is AnyTeleop, evaluated on the same leader–follower hardware. Demonstration-collection performance is measured by complete-task success rate and time per successful demonstration:

$$
\text{time per successful demonstration}
=
\frac{
\text{total duration of successful and failed attempts}
}{
\text{number of successful demonstrations}
}.
$$

Across 20 attempts per method and task, copilot-assisted collection achieved a 100% success rate on all six tasks. AnyTeleop achieved success rates of 0% for Tongs Transfer, 50% for Bottle Disposal, 30% for Dual-Object Transfer, 75% for Ingredient and Pot Transfer, and 0% for both Toast Preparation and Binder Filing. Copilot time per successful demonstration was 44.33 seconds for Tongs Transfer, 41.37 seconds for Bottle Disposal, 36.19 seconds for Dual-Object Transfer, 43.26 seconds for Ingredient and Pot Transfer, 327.46 seconds for Toast Preparation, and 221.80 seconds for Binder Filing. AnyTeleop’s time metric was undefined for tasks with no successful demonstrations.

Outer policies were trained from 20 successful complete-task demonstrations where available and evaluated over 20 rollouts. Success rates were:

| Demonstration source and representation | Tongs | Bottle | Dual-object | Ingredient/pot |
|---|---:|---:|---:|---:|
| Copilot, no H-VAE | 65% | 60% | 80% | 85% |
| Copilot, H-VAE | 100% | 75% | 90% | 100% |
| AnyTeleop, no H-VAE | N/A | 40% | 20% | 75% |

The results support improved autonomous performance when H-VAE hand-action compression is used. The reported interpretation is that coordinated hand commands occupy a more compact structure than their raw 20-joint representation, making them easier for the outer policy to predict.

### Closed-loop and temporal-ensemble ablation

A bottle-grasp ablation compares fixed-command replay, closed-loop execution without ensembling, and closed-loop execution with temporal ensembling. Each condition contributes 10 adjudicated trials. Success rates were 3/10 for fixed replay, 7/10 for closed-loop execution without ensembling, and 9/10 for closed-loop execution with temporal ensembling. The difference between fixed replay and closed-loop temporal ensembling was statistically significant, with $p=0.0198$.

The 95th percentile of absolute joint jerk during the active closing phase was $2.30\times$ higher for closed-loop execution without ensembling than for closed-loop temporal ensembling, with $p=1.8\times10^{-4}$. This trend held before and after command limiting and across Savitzky–Golay filter windows of 0.11, 0.21, and 0.31 seconds. Temporal ensembling did not increase closing duration.

### Limitations

The reported evidence is limited to one hardware platform, a small real-world task suite, and demonstrated object and task distributions. The grasp-policy experiment does not establish generalization to unseen objects. Each required inner skill also requires an initial dataset and training cost, so NestDex shifts part of the data-collection burden from complete-task teleoperation to reusable skill-library construction.

The vision-language selector is underspecified with respect to architecture, prompts, latency, training, and quantitative failure rates. The supplied description likewise does not provide complete inner-policy architecture or loss details, H-VAE architecture and latent dimension, force-control guarantees, or formal safety and failure-recovery guarantees. The principal execution ablation uses 10 trials per condition, and transfer to different arms, hands, sensing systems, clutch interfaces, or task distributions is not established.

NestDex’s principal contribution is consequently a collection-to-autonomy pipeline: reusable proprioceptive hand skills reduce the burden of dexterous teleoperation; a reversible clutch regulates their execution; a vision-language selector coordinates skills across task stages; H-VAE compression produces compact hand-action targets; and an outer visuomotor policy learns complete arm–hand behavior without requiring the copilot stack during deployment.

Source: https://www.emergentmind.com/topics/nestdex