---
title: Glove2Robot Policy Pipeline
url: https://www.emergentmind.com/topics/glove2robot-policy-pipeline
type: topic
---

# Glove2Robot Policy Pipeline

The Glove2Robot policy pipeline refers to a family of end-to-end methodologies for transferring fine-grained human manipulation skills to autonomous robot systems via human-hand demonstration, with or without instrumented tactile gloves. These pipelines are characterized by: (i) direct sensing of human hand motion (via vision, IMU, or tactile glove), (ii) systematic pre-processing and alignment to the robot embodiment, (iii) generative or kinematic translation of demonstration data to robot-observable space, (iv) SE(3) and/or joint-state action extraction, and (v) final integration into policy learning frameworks such as conditional diffusion models or behavior cloning. Across recent instantiations, Glove2Robot approaches enable scalable collection of high-fidelity demonstrations suitable for closed-loop policy deployment that matches or approaches performance from direct teleoperation, at reduced hardware cost and complexity [2507.03930][2512.08920].

## 1. Human Demonstration Capture Modalities

Glove2Robot pipelines support a range of demonstration capture modalities. The approach in "RwoR" [2507.03930] utilizes a GoPro Hero 9 camera with a Max Lens Mod 1.0 fisheye module, wrist-mounted via a custom 3D-printed adapter, optimized for a 160° field-of-view that reliably frames both hand and object. The wrist adapter allows tuning of both wrist-axis rotation and translation (12–15 cm from the fingertips), visually mimicking the robot’s onboard wrist camera. Pose information is inferred indirectly from camera and IMU, dispensing with the need for an instrumented glove; each demonstration frame is 4K at 30 Hz, with a synchronized 200 Hz IMU stream.

In contrast, the OSMO pipeline [2512.08920] employs a custom open-source tactile glove featuring 12 3-axis magnetic-tactile taxels (five on the fingertips and three on the palm), each comprising paired Bosch BMM350 magnetometers and an integrated 6-axis BHI360 IMU, supporting a force range of 0.3 N to 80 N. Sensor boards communicate via I²C to an STM32 microcontroller, sampled at 100 Hz and logged, along with Intel RealSense RGB + stereo IR frames, via ROS 2 at 25 Hz. The tactile glove facilitates direct measurement of continuous shear and normal force, minimizing the embodiment gap and eliminating reliance on force inference through image processing.

## 2. Data Pre-processing and Alignment

Both vision- and glove-based pipelines necessitate rigorous pre-processing to resolve the temporal and spatial misalignments between human demonstrations and the robot's sensory-modalities.

For vision-based approaches [2507.03930], raw human video $\mathcal{H} = \{h_i\}_{i=1}^{T_1}$ and paired robot gripper demonstration $\mathcal{R} = \{r_j\}_{j=1}^{T_2}$ are recorded separately ($T_1 \neq T_2$). Temporal alignment leverages Temporal Cycle-Consistency (TCC) to learn a shared embedding function $f:\text{Image}\rightarrow \mathbb{R}^d$, aligning each human frame to its nearest robot frame in this latent space:
$$
t(i) = \arg\min_{j} \|f(h_i) - f(r_j)\|_2
$$
Remaining spatial/viewpoint discrepancies are addressed by compositing: foreground masks $M_t$ for the gripper and object (from SAM2) are used to inpaint human background into aligned robot frames, yielding synthesized ground-truth robot images:
$$
\hat r_t = (1 - M_t)\odot h_t + M_t\odot r_t
$$

The tactile-glove pipeline [2512.08920] processes raw magnetic field vectors by channel-wise differential subtraction (to mitigate cross-talk) and robust percentile-based normalization (Eq. 3). Hand pose alignment is solved by first segmenting hands via SAM2, then extracting mesh and wrist pose with HaMeR, and further refining wrist height via back-projected IR stereo; trajectories are smoothed with a Savitzky-Golay filter.

## 3. Human-to-Robot Demonstration Translation

Vision-based Glove2Robot pipelines employ generative models to transform human hand frames to robot-equivalent observations. The RwoR approach [2507.03930] fine-tunes InstructPix2Pix, a conditional diffusion model (based on Stable Diffusion), to map each human observation $h_t$ and text prompt $l$ to a robot image $\hat r_t$. Training uses $15,000$ aligned pairs over $3$ epochs, with batch size $4$, and a linear noise schedule ($1,000$ steps, $\beta_1 \ldots \beta_{1000}$). The generative loss is:
$$
L_{\mathrm{gen}} = \mathbb{E}_{t,\epsilon} \Bigl[\|\epsilon - \epsilon_\theta(z_t, t, \mathcal{E}(h_t), l)\|_2^2\Bigr]
$$
At inference, each $h_t$ is mapped to a predicted robot frame $\tilde r_t$.

Tactile-glove pipelines [2512.08920] bypass this vision-to-robot transfer. By equipping the human demonstrator and the robot with identical sensorized gloves, all tactile and kinematic signals are natively matched across domains, negating the need for generative visual translation or force inference.

## 4. Action Extraction and Retargeting

A central component of Glove2Robot is extracting SE(3) or joint-state actions corresponding to demonstrator intent.

In wrist-camera pipelines [2507.03930], ORB-SLAM3 recovers camera poses $\{T_{c_t}^w\}$ from the human video, refined via GoPro IMU. Fixed extrinsics $T_f^c\in SE(3)$ from camera to “fingertip” yield:
$$
T_{f_t}^w = T_{c_t}^w T_f^c
$$
Actions are then incremental SE(3) transforms:
$$
\Delta T_t = (T_{f_t}^w)^{-1} T_{f_{t+1}}^w
$$
Gripper open/closed states $g_t$ are binary, detected via foreground mask transitions.

For glove-based pipelines [2512.08920], Cartesian positions from hand tracking and tactile signals are retargeted through the Mink IK solver (MuJoCo backend) to generate 7-DoF arm + 6-DoF hand joint sequences. Unsafe kinematic states are handled by frame skipping or pose repetition. Both proprioceptive and tactile information are synchronized across the trajectory.

## 5. Policy Learning and Integration

Resulting demonstration sequences are structured for integration into policy learning frameworks. The vision-based pipeline [2507.03930] constructs tuples:
$$
(o_t, a_t)_{t=1..T}, \quad o_t = \{\tilde r_{t-k+1},..., \tilde r_t, T_{f_t}^w, g_t\}, \quad a_t = (\Delta T_t, g_{t+1})
$$
A conditional diffusion policy $\pi_\phi$ is then trained per [Chi et al., 2023], with 1,000 noise schedule steps, batch size 32, Adam optimizer ($\eta=10^{-4}$), across 100 epochs. Policy inference is conducted in closed-loop with online wrist-camera input, generating SE(3) actions and gripper commands for the robot.

The tactile-glove pipeline [2512.08920] adopts a conditional diffusion-policy learning framework: image features (DINOv2), proprioceptive vectors, and normalized tactile features are embedded and concatenated, then conditioned into a FiLM-adapted U-Net denoiser across 100 DDPM steps. The policy predicts future joint states over a 16-step horizon (training), and executes 4 steps per chunk at test time. Tactile and visual signals are incorporated directly into input features, with no separate behavioral cloning or force loss.

## 6. Empirical Evaluation and Performance

Performance evaluations highlight the capacity of Glove2Robot pipelines to approach or match direct teleoperation performance on real-world manipulation tasks.

- In RwoR [2507.03930], nine tasks on a Franka Emika R3 with UMI gripper yield a mean success rate of 0.78, near the upper bound of 0.82 (direct-UMI demonstrations), substantially exceeding a rule-based baseline (0.37). Model ablations demonstrate incremental value from composited inpainting and the full generative pipeline, with PSNR/SSIM progressing from (31.5/0.77) (raw), to (33.4/0.84) (inpaint-only), to (33.8/0.86) (full). Generalization to unseen actions and object instances yields success rates of 0.80–0.87.
- The OSMO pipeline [2512.08920] reports a 72% ± 27.4% success rate on a contact-rich whiteboard-wiping task, outperforming vision-only (55.8% ± 30.0%) and proprio-only (27.1% ± 32.4%) policies. Qualitative analyses indicate vision-based policies suffer contact failure modes (over/under-pressing, slip), while the tactile-inclusive policy maintains robust contact and pressure.

These results collectively indicate that Glove2Robot pipelines, leveraging either wrist-mounted vision with generative translation or high-fidelity tactile gloves with explicit force conditioning, enable robot policy imitation with minimal hardware overhead and improved embodiment-robustness.

## 7. Embodiment Gap, Limitations, and Extensions

A fundamental challenge addressed by Glove2Robot pipelines is the visual and sensory “embodiment gap” between human demonstrations and robot execution. The vision-only pipelines require sophisticated timestamp alignment, image compositing, and hand-to-gripper generative translation to reconcile differences, whereas the tactile-glove approach in OSMO [2512.08920] sidesteps these issues by standardizing hardware/sensing across both domains.

Implications include:
- Settings devoid of tactile instrumentation require careful generative and pre-processing components to align human and robot domains; in contrast, demonstration collection with matched tactile gloves simplifies policy transfer and improves contact-relevant task performance.
- The selection of policy learning paradigms (diffusion policies, conditional imitation learning) is directly informed by the structure and alignment of the demonstration data.
- Tasks that critically depend on force feedback or contact are particularly sensitive to embodiment-gap minimization; OSMO data demonstrates performance improvements in such domains.

A plausible implication is that further generalization and scalability of the Glove2Robot paradigm may depend on improvements in hand tracking accuracy, tactile-glove cost reduction, and policy architectures robust to diverse demonstration domains.

---

**References**:  
- "RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot" [2507.03930].  
- "OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer" [2512.08920].

Source: https://www.emergentmind.com/topics/glove2robot-policy-pipeline