---
title: 'SEED-UMI: Dexterous Imitation Learning'
url: https://www.emergentmind.com/topics/seed-umi
type: topic
---

# SEED-UMI: Dexterous Imitation Learning

SEED-UMI (“Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration”) is a dexterous imitation-learning framework that uses the same mechanically isomorphic exoskeleton on both a human demonstrator and a robot. The shared exoskeleton provides a common 20-joint measurement interface, matched outer contact geometry, and exoskeleton-mounted wrist cameras that observe comparable visual configurations during demonstration and execution. Human demonstrations are replayed on the robot to obtain paired cross-embodiment supervision, enabling refinement of the encoder-to-command mapping under contact. On five contact-rich tasks, SEED-UMI reports a $3.0\times$ improvement in successful demonstration-collection throughput over exoskeleton-based teleoperation and a $70.0\%$ average autonomous rollout success rate [2609.11753].

## 1. Motivation and design principles

Dexterous imitation learning is constrained by the difficulty of collecting demonstrations that preserve contact-rich human motion when transferred to a robot. Vision-based tracking is natural but vulnerable to occlusion, jitter, and loss of accurate finger motion during contact. Gloves provide more stable measurements but do not generally correspond one-to-one with robot joints. Human-worn exoskeletons constrain motion mechanically and produce repeatable measurements, yet conventional systems typically record only the human side and apply an open-loop mapping calibrated primarily in free space.

Such mappings may degrade under friction, compliance, external loads, hysteresis, and joint coupling. A mapping that is accurate when the fingers move without contact need not remain accurate when the robot grasps, presses, wipes, inserts, or drives an object. Teleoperation exposes the operator to robot-side behavior, but requires real-time control of the robot and can introduce delay, reduced contact awareness, and lower collection throughput.

SEED-UMI changes the physical interface rather than relying solely on a more complex post-hoc retargeting model. Its principal design principles are:

- **Mechanical isomorphism**: the human and robot use the same 20-joint exoskeleton structure.
- **Shared measurement**: joint encoders measure the same mechanically corresponding degrees of freedom on both embodiments.
- **Matched observation**: wrist cameras are mounted to the exoskeleton and observe the same outer mechanism and workspace arrangement.
- **Paired supervision**: human demonstrations are replayed on the robot, allowing discrepancies between human-side and robot-side measurements to refine the mapping.
- **Raw visual learning**: policies operate on exoskeleton-centric wrist images without hand segmentation, inpainting, compositing, or generative image translation.

The framework is therefore intended for one-to-one human–robot transfer rather than arbitrary embodiment-agnostic retargeting. Its shared geometry improves correspondence for the target robot, but also creates hardware-specific engineering requirements [2609.11753].

## 2. Hardware and shared exoskeleton interface

The target platform is a fully actuated Wuji dexterous hand mounted on a RealMan RX75 robot arm. The hand has five fingers, four actuated degrees of freedom per finger, and 20 total joint degrees of freedom. The exoskeleton is co-designed for this hand: link lengths, joint axes, motion limits, outer contact surfaces, palmar fingertip pads, and encoder frames are matched to the robot hand.

The same mechanically corresponding structure is worn by the human demonstrator and the robot. If $\mathbf{e}_t\in\mathbb{R}^{20}$ denotes the encoder vector at time $t$, it is intended to represent the same physical joint interface in both conditions. The human and robot versions have different cuffs and mounting adaptations, but the contact-facing skeleton remains the same.

Direct encoder placement at lateral finger joints would protrude into the clearance envelope and interfere with palmar contact. SEED-UMI therefore uses a parallel four-bar linkage to route joint motion to dorsal, contactless magnetic rotary encoders. The linkage is designed to preserve a one-to-one angular relationship while keeping the palm and lateral finger surfaces unobstructed and maintaining the robot’s range of motion.

The sensing configuration consists of:

- **Joint sensing**: one contactless rotary encoder per active joint, producing a 20-dimensional vector.
- **Wrist tracking**: a dorsal Intel RealSense T265 providing six-degree-of-freedom wrist tracking.
- **Visual observation**: a ventral fisheye camera with a $150^\circ$ field of view, observing the workspace and the exoskeleton’s contact-facing region.

The cameras are rigidly attached to the exoskeleton. Their intrinsics and exoskeleton-frame extrinsics are consequently intended to remain fixed across human data collection and robot rollout. The policy receives the raw image rather than a synthesized robot-hand view. Ordinary photometric and crop augmentation are used, but the paper does not introduce hand segmentation, object–hand compositing, generative inpainting, or cross-painting [2609.11753].

## 3. Encoder-to-command mapping and paired refinement

SEED-UMI retains an encoder-to-robot-command mapping, but replaces the assumption of a permanently fixed free-space mapping with a paired cross-embodiment refinement procedure. Let $\mathbf{e}_t$ be the exoskeleton encoder vector and $\mathbf{q}_t$ the robot hand joint-command vector. The learned inverse mapping is represented as

$$
\mathbf{q}_t=f_\theta(\mathbf{e}_t,\mathbf{h}_t),
$$

where $\mathbf{h}_t$ contains temporal information: previous encoder readings, finite-difference encoder velocities, and a binary opening/closing direction indicator for each joint. The history allows the mapping to represent motion-direction dependence, velocity effects, hysteresis, linkage behavior, and other temporal phenomena.

### Robot motor babbling

The robot first wears the exoskeleton and performs structured exploration, including single-joint sweeps, coupled multi-joint motions, varied speeds, and self-contact poses. Robot joint positions and robot-side exoskeleton encoder measurements are recorded as $\{\mathbf{q}_t,\mathbf{e}^{(r)}_t\}$. These data initialize a mapping

$$
f_{\theta^0}:(\mathbf{e}_t,\mathbf{h}_t)\mapsto\mathbf{q}_t.
$$

This initial model captures basic kinematic correspondence and history-dependent behavior, but can remain inaccurate under contact loads.

### Human demonstration

The human wears the exoskeleton and performs representative manipulation motions, including grasping and contact-rich actions. Human-side encoder traces $\{\mathbf{e}^{(h)}_t,\mathbf{h}^{(h)}_t\}$ and exoskeleton-centric wrist observations are recorded.

### Paired robot replay

Each human demonstration is replayed on the robot using $f_{\theta^0}$. The robot-side replay records $\{\mathbf{e}^{(r)}_t,\mathbf{h}^{(r)}_t,\mathbf{q}^{(r)}_t\}$. The human trajectory and robot replay are not expected to coincide exactly; their discrepancy is used as supervision.

SEED-UMI introduces a learned forward model,

$$
g_\phi(\mathbf{q}_t,\mathbf{h}_t)\approx\mathbf{e}_t,
$$

which predicts robot-side encoder behavior from robot commands and history. The mapping objective is

$$
\mathcal{L}_{\mathrm{map}} =
\left\|
g_\phi\!\left(
f_\theta(\mathbf{e}^{(h)}_t,\mathbf{h}^{(h)}_t),
\mathbf{h}^{(h)}_t
\right)
-\mathbf{e}^{(h)}_t
\right\|_2^2
+
\beta
\left\|
f_\theta(\mathbf{e}^{(h)}_t,\mathbf{h}^{(h)}_t)
-\mathbf{q}^{(r)}_t
\right\|_2^2
+
\lambda\left\|\Delta f_\theta\right\|_2^2.
$$

The terms have distinct roles. The encoder-reconstruction term requires the predicted robot command to produce robot-side encoder behavior compatible with the human trajectory. The paired-replay term keeps the refined mapping near the command actually used during replay. The temporal smoothness term penalizes abrupt command changes. The paper states that replay and refinement can be repeated once.

This procedure converts retargeting into a supervised correction problem. The mechanically shared interface makes residual discrepancies informative about contact loads, friction, compliance, hysteresis, and coupled motion rather than leaving them entirely implicit in an open-loop mapping [2609.11753].

## 4. Policy representation and execution

SEED-UMI evaluates three policy backbones: ACT, Diffusion Policy, and $\pi_{0.5}$. Policies are trained on human-collected real-world demonstrations. The policy observation is represented as

$$
\mathbf{o}_t=(I_t,\mathbf{e}_t),
$$

where $I_t$ is the raw image from the exoskeleton-mounted ventral fisheye camera and $\mathbf{e}_t\in\mathbb{R}^{20}$ is the current exoskeleton encoder vector. Wrist-pose information supplied by the T265 can form part of the wrist observation.

Each predicted action contains a six-degree-of-freedom wrist action and a 20-dimensional hand command:

$$
\mathbf{a}_t=
\left[
\mathbf{u}^{\mathrm{wrist}}_t;
\mathbf{q}^{\mathrm{hand}}_t
\right]
\in\mathbb{R}^{26}.
$$

The policy predicts action chunks,

$$
\mathbf{a}_{t:t+L-1}
=
\{\mathbf{a}_{t+\ell}\}_{\ell=0}^{L-1},
$$

where $L$ is the action-chunk horizon. The paper does not specify the exact chunk length, network architectures, optimizer, learning-rate schedule, batch size, or backbone-specific loss functions for ACT, Diffusion Policy, and $\pi_{0.5}$.

At rollout, predicted hand commands are executed through the learned robot-side control interface. The policy does not need to observe a segmented or synthesized robot hand image because the exoskeleton and camera arrangement are intended to produce a shared visual domain.

### Wrist calibration

The T265 provides six-degree-of-freedom wrist pose. Wrist trajectories use a fixed T265-to-tool-center-point alignment, and no trajectory correction is applied. If ${}^{W}\mathbf{T}_{C}(t)$ is the camera pose in a world frame and ${}^{C}\mathbf{T}_{T}$ is the fixed camera-to-tool transform, the implied rigid-body composition is

$$
{}^{W}\mathbf{T}_{T}(t)
=
{}^{W}\mathbf{T}_{C}(t)\,{}^{C}\mathbf{T}_{T}.
$$

The paper does not report numerical calibration matrices or an additional trajectory-registration procedure. The fixed alignment assumes that the camera-to-tool relationship remains sufficiently stable during both collection and rollout [2609.11753].

## 5. Contact-rich manipulation and experimental protocol

SEED-UMI evaluates five tasks:

1. **Screw Driving**: grasp an electric screwdriver, align it with a screw head, maintain axial pressure, and drive the screw.
2. **AirPods Case Insertion**: pick up an AirPod and insert it into its charging case using precise index–middle-finger coordination.
3. **Ball Basket Throwing**: pick up a ball, lift it, and release it into a basket.
4. **Table Cleaning**: extract a tissue, wipe the tabletop under sustained contact, and discard the tissue.
5. **Air Freshener Spray**: grasp a spray can and initiate a downward actuator press with coordinated thumb and multi-finger support.

For Air Freshener Spray, success requires a secure grasp and a clearly initiated downward press. Complete actuator depression and actual spray emission are not required because the robot hand cannot reliably provide the required actuation force.

Three conditions are compared:

- **Teleoperation ($T$)**: the operator wears the exoskeleton, whose encoder readings are mapped directly to robot hand motor commands in real time. Policies are trained on robot-side teleoperation demonstrations.
- **SEED-UMI without paired fine-tuning ($P^{-}$)**: policies are trained on human-side demonstrations, while rollout uses only the motor-babbling mapping $f_{\theta^0}$.
- **SEED-UMI with paired fine-tuning ($P^{+}$)**: policies are trained on human-side demonstrations, while rollout uses the paired-replay-refined mapping $f_\theta$.

For every task and condition, 100 demonstration episodes per task are used for policy training. Paired fine-tuning uses 250 calibration episodes: 50 episodes for each of five objects, with variation in size, shape, stiffness, and grasp type. Each calibration episode fixes one object at one tabletop pose, records the human grasp with the exoskeleton, and replays it on the robot using $f_{\theta^0}$. Every task–backbone–condition combination is evaluated over 20 autonomous robot rollouts with randomized initial object positions [2609.11753].

## 6. Results, limitations, and significance

The reported success percentages over 20 rollouts per entry are:

| Policy | Condition | Screw | AirPods | Throwing | Cleaning | Spray |
|---|---:|---:|---:|---:|---:|---:|
| ACT | $T$ | 80 | 65 | 60 | 75 | 90 |
| ACT | $P^{-}$ | 55 | 65 | 60 | 50 | 70 |
| ACT | $P^{+}$ | 70 | 75 | 70 | 65 | 85 |
| Diffusion Policy | $T$ | 65 | 55 | 50 | 60 | 70 |
| Diffusion Policy | $P^{-}$ | 40 | 55 | 50 | 40 | 45 |
| Diffusion Policy | $P^{+}$ | 55 | 60 | 55 | 55 | 60 |
| $\pi_{0.5}$ | $T$ | 90 | 75 | 70 | 85 | 85 |
| $\pi_{0.5}$ | $P^{-}$ | 60 | 75 | 70 | 60 | 65 |
| $\pi_{0.5}$ | $P^{+}$ | 80 | 85 | 80 | 75 | 80 |

Averaged over tasks and policy backbones, the success rates are:

$$
\mathrm{Success}(T)=71.7\%,
$$

$$
\mathrm{Success}(P^{-})=57.3\%,
$$

$$
\mathrm{Success}(P^{+})=70.0\%.
$$

The task success rate is

$$
\mathrm{SuccessRate}
=
\frac{N_{\mathrm{successful}}}{N_{\mathrm{rollouts}}}
\times 100\%.
$$

The paired refinement therefore improves the average result by 12.7 percentage points from $P^{-}$ to $P^{+}$ and brings SEED-UMI close to robot-side teleoperation. Gains are especially pronounced for Screw Driving, Table Cleaning, and Air Freshener Spray, which involve sustained pressure, friction, or force-sensitive interaction. The reported paired-fine-tuning improvements are $16.7$ percentage points for Screw Driving, $15.0$ for Table Cleaning, $15.0$ for Air Freshener Spray, and $12.7$ overall.

For the collection-efficiency comparison on AirPods Case Insertion, a 30-minute window produced 18 successful teleoperation demonstrations and 52 successful SEED-UMI demonstrations. The ratio $52/18$ is approximately $2.9$, reported as a $3.0\times$ throughput improvement. After success normalization, the paper reports approximately $2.9\times$ more useful training data per minute.

SEED-UMI’s principal limitations follow from its one-to-one design. Adapting the system to another dexterous hand requires redesign of link geometry, joint limits, actuator ranges, encoder placement, cuffs, mounts, and calibration motions. The human must wear the exoskeleton, creating potential constraints related to weight, comfort, donning time, and long-horizon use. The system does not include tactile sensing, and the spray task exposes a hand actuation-force limitation. Fixed T265-to-tool calibration can be affected by calibration error or sensor drift. The paper also does not report model sizes, training time, compute hardware, inference latency, optimizer settings, or detailed implementation parameters for the evaluated policy backbones.

The framework should therefore be distinguished from general embodiment-agnostic systems. UMI-on-Air, for example, retains a high-level UMI policy and injects embodiment-specific controller-tracking gradients into diffusion sampling at inference time, enabling adaptation to constrained aerial embodiments [2510.02614]. UMI-Bench 1.0 instead provides a reproducible real-robot evaluation protocol for UMI-style policies and evaluates $\pi_0$, $\pi_{0.5}$, and DreamZero rather than SEED-UMI [2606.10382]. SEED-UMI’s distinctive contribution is earlier in the pipeline: it physically shares the measurement and observation interface between human and robot, then uses paired replay to correct the contact-dependent mapping.

The central significance of SEED-UMI is consequently an interface-design principle: human demonstration and robot execution should share the same physical measurement and observation structure whenever possible. The exoskeleton supplies one-to-one joint correspondence, matched contact geometry, fixed camera placement, paired replay data, and a common raw-image domain. This reduces the need for visual domain translation and converts contact-induced retargeting error into an observable supervised signal. The reported results indicate that this arrangement can approach robot-side teleoperation quality while substantially increasing the efficiency of dexterous demonstration collection.

Source: https://www.emergentmind.com/topics/seed-umi