---
title: 'exUMI: Tactile-Aware Robot Teaching System'
url: https://www.emergentmind.com/topics/exumi
type: topic
---

# exUMI: Tactile-Aware Robot Teaching System

exUMI is a tactile-aware robot teaching system that couples a portable demonstration device with an action-conditioned tactile representation learning pipeline for contact-rich manipulation. Introduced as an extensible upgrade to the original UMI, it is designed to address three intertwined limitations in tactile robot learning: data scarcity, tactile sparsity, and the lack of force-feedback capture in existing teaching systems. The system combines hardware modifications—robust proprioception, modular visuo-tactile sensing, centralized logging, calibration, and synchronization—with Tactile Prediction Pretraining (TPP), a task-agnostic method for learning tactile representations from aligned tactile, action, and visual streams. The core claim is that progress in tactile manipulation depends on co-designing collection infrastructure and representation learning rather than treating policy learning alone as the bottleneck [2509.14688].

## 1. Definition and problem setting

The exUMI paper describes the platform as an **extensible upgrade to UMI** for tactile-aware robot teaching and identifies it as a **portable handheld physical twin of a robot gripper** with accurate end-effector tracking, robust gripper-width sensing, integrated visuo-tactile fingertips, centralized multimodal logging, and automated calibration and temporal alignment [2509.14688]. In the detailed description, exUMI is expanded as the **extensible Universal Manipulation Interface**. Its purpose is to bridge the gap between the human demonstrator’s tactile intuition and the sensory information made available to the robot learner.

The motivating problem is specific to contact-rich manipulation. Humans use force and touch to regulate grasping, insertion, twisting, sliding, and compliance-sensitive interaction, but many robot teaching systems record only vision and kinematics, or depend on fragile tracking pipelines that degrade multimodal data quality. The paper states that valid tactile contact often occupies **less than 10% of manipulation trajectories** in regular robot data collection, which makes direct end-to-end tactile imitation learning statistically inefficient [2509.14688]. exUMI is therefore positioned not merely as a sensing add-on, but as an infrastructure for collecting **high-quality, contact-rich, action-aligned tactile data**.

A central historical comparison in the paper is with the original or “vanilla” UMI system. Vanilla UMI relied on **visual SLAM** for pose tracking and **ArUco markers** for gripper state estimation. The exUMI paper argues that these components are vulnerable to occlusion, difficult backgrounds, fisheye distortion, and preprocessing failure. It reports that exUMI improves the effective data ratio to **nearly 100%**, compared to **less than 60%** for vanilla UMI in the authors’ pipeline, and that ArUco-based gripper estimation alone caused about **10% of preprocessing failures** [2509.14688]. This framing makes data usability—not only policy accuracy—a first-class systems objective.

## 2. Hardware architecture and multimodal sensing

The exUMI device is organized around three explicit design principles: **precise robot proprioception**, **extensibility**, and **portability** [2509.14688]. Its hardware stack includes a **Meta Quest 3 AR motion capture subsystem**, an **AS5600 magnetic rotary encoder** for gripper width, an **Orange Pi controller** as a central sensor hub, a **GoPro fisheye RGB camera**, **two fingertip visuo-tactile sensors**, and optional compatibility with non-parallel grippers through an additional mechanical design.

A key design choice is the paper’s decision to **disentangle proprioception**. exUMI assigns **6D end-effector pose** estimation to AR motion capture and **gripper opening/width** estimation to a dedicated encoder, rather than inferring both visually. For 6D tracking, the system uses a **Meta Quest 3 headset** that tracks a left VR controller attached to the handheld device by a custom mount. The paper presents this as a replacement for vulnerable SLAM-based tracking and emphasizes its portability, real-time 6D pose streaming, and robustness to occlusion. The appendix reports mean position errors of **5.4 mm**, **2.3 mm**, and **1.7 mm** on the three axes, with rotation errors below **1 degree**, and describes the average error as **less than 10 mm** compared to Fast-UMI. The tracking range is reported as **3 meters**, which the paper characterizes as adequate for typical robot workspaces [2509.14688].

For gripper state sensing, exUMI replaces ArUco-based estimation with a **low-cost AS5600 magnetic rotary encoder** mounted above a joint with a radial magnet. The sensor provides **12-bit position readings**, **4096 positions per revolution**, and **I\(^2\)C communication**, while being immune to visual occlusion and incurring negligible compute overhead [2509.14688]. This substitution directly targets one of the previously identified failure modes of vanilla UMI.

The tactile subsystem consists of **two fingertip visuo-tactile sensors** based on **9DTact**, redesigned for practical deployment. The paper lists four modifications: a contact protection or beveled structure to secure the silicone gel under large tangential force, improved cable and power connections for durability, a custom mold for consistent silicone thickness across sensors, and LED board modifications to improve robustness and reduce power draw. The exUMI paper does not frame the sensor as a precision geometry sensor like GelSight; rather, it emphasizes low-cost capture of **normal force**, **tangential force**, **deformation patterns**, and **fast recovery** [2509.14688]. This distinction matters because the downstream representation learning objective depends on temporal contact dynamics rather than fine surface reconstruction.

The **Orange Pi single-board computer** acts as a universal sensor hub. It synchronously logs AR headset and controller data, encoder data, tactile streams, and potentially other modalities. The paper explicitly states that the design supports adding tactile, audio, or other custom sensors, making extensibility an architectural property rather than an afterthought [2509.14688].

## 3. Calibration, synchronization, and robot embodiment

A major systems contribution of exUMI is its calibration and synchronization pipeline. The paper argues that tactile learning is fundamentally about **contact dynamics**, so temporal misalignment between action, touch, and vision corrupts supervision even when each modality is individually high quality [2509.14688].

The system uses two one-time calibrations. In **AR controller calibration**, the operator aligns exUMI with the base coordinate frame in AR space and records the controller transform, which is then used to correct future pose tracking. In **gripper state calibration**, the user places the gripper at **1 cm intervals**, records AS5600 readings, and interpolates a mapping from encoder readings to absolute gripper width [2509.14688]. These procedures are intentionally lightweight, reducing the burden of deployment while preserving reliable proprioception.

For cross-modal alignment, the paper introduces a **latency calibration protocol**. At the start of data collection, the operator sweeps exUMI horizontally above an ArUco marker; the system extracts the AR system’s x-axis motion and the marker trajectory in video, then searches for the offset minimizing the mean-squared discrepancy. The appendix gives the objective informally as finding $\delta^*$ such that
$$
f(t) \approx g(t+\delta^*)
$$
and minimizing
$$
\delta^* = \arg\min_{\delta_k} \sum_{i=1}^{T}\left\| f(t_i)-g(t_i+\delta_k)\right\|_2^2 .
$$
The main paper states that synchronization error is **under 50 ms**, while the appendix reports **less than 5 ms latency error** with proper sweep frequency for the AR/vision calibration [2509.14688]. The difference reflects separate reporting contexts rather than a single reconciled number.

Embodiment consistency is another design axis. exUMI is a physical twin of the target robot gripper, and the paper evaluates on a **Flexiv Rizon 4 arm** with a **Flexiv Grav adaptive gripper**. For robot deployment, the authors replicate the exUMI sensor placement on the robot using a **pipe-clamp-style mount** for the GoPro and tactile sensors [2509.14688]. This reduces train–test mismatch between human demonstration and robot execution. The robot is controlled at **10 Hz** from a master computer with an RTX 4070 GPU.

The bill of materials totals **\$698**: **\$298** for a GoPro 11 plus accessories, **\$299** for a Meta Quest headset, **\$35** for an Orange Pi 3B, **\$1** for the AS5600, **\$15** for 3D-printed parts, **\$30** for visuo-tactile sensors, and **\$20** for miscellaneous components [2509.14688]. The paper explicitly characterizes the system as low-cost, DIY-friendly, portable, and open-source.

## 4. Data collection regimes and tactile dataset design

The paper uses exUMI in two distinct data-collection regimes: **task demonstrations** for downstream imitation learning and **human play data** for tactile representation pretraining [2509.14688]. This bifurcation is central to the system’s design philosophy. Task demonstrations provide behaviorally relevant supervision, while play data provide contact diversity and density.

For downstream tasks, users collect about **100 to 200 demonstrations per task**. The paper reports examples such as **204 demos in 42 min** for Pick Cube, **135 demos in 31 min** for Pick Carrot, **170 demos in 47 min** for Pick Broccoli, **201 demos in 26 min** for Stack Cube, **139 demos in 60 min** for Insert Pen, **181 demos in 66 min** for Put Ball, **270 demos in 79 min** for Open Bottle, **202 demos in 70 min** for Pull Drawer, and **163 demos in 56 min** for Peg in Hole [2509.14688]. It also states that for a simple pick-and-place task, a user can collect **100 demonstrations in 20 minutes** and train a behavioral cloning policy that exceeds **70% success rate**.

The pretraining corpus is more distinctive. The paper reports interaction in **10 real-world environments** with **300+ objects**, spanning **rigid tools**, **deformable fabrics**, and **granular materials**, yielding **over 1 million frames** of aligned image–tactile–action data [2509.14688]. Other sections report **480K tactile frames in 5 hours of human interaction** and **480.9K raw tactile frames** in an appendix comparison. The paper explicitly notes that the exact accounting varies across sections, but consistently presents the corpus as substantially larger than prior tactile datasets.

This play-based collection strategy is designed to overcome tactile sparsity. Whereas ordinary demonstrations contain **less than 10%** meaningful tactile contact, the paper reports that in the play dataset **over 60% of frames are active tactile frames**, yielding more than **10× efficiency** relative to teleoperation for collecting tactile frames [2509.14688]. A plausible implication is that exUMI’s main statistical advantage lies not only in raw scale but in raising the density of informative contact transitions.

The recorded modalities include aligned RGB image data, tactile images from both fingertips, action or proprioception, and gripper state. Action is represented as **relative pose** and **gripper state**. For tactile processing, the left and right tactile images are concatenated, converted to calibrated grayscale, transformed into a 3-channel representation consisting of **grayscale + convex map + concave map**, and downsampled to **224 × 224** [2509.14688]. The training/validation split is **15:1**.

Preprocessing includes temporal synchronization via latency calibration, alignment of AR and video streams by interpolation, generation of synchronized packets, and a **data rejection strategy** that filters out trivial segments during pretraining: if all tactile frames in a chunk have active pixel proportions below a threshold, the chunk is discarded and resampled [2509.14688]. This preprocessing strategy is explicitly motivated by tactile sparsity rather than by conventional dataset cleanliness alone.

## 5. Tactile Prediction Pretraining

The algorithmic counterpart to exUMI is **Tactile Prediction Pretraining (TPP)**, which the paper defines as a representation learning framework based on **action-aware temporal tactile prediction** [2509.14688]. The stated motivation is that tactile perception should not be treated as a static image understanding problem. Instead, touch is modeled as a temporally evolving signal whose semantics depend on action.

The downstream tactile-aware policy is written as
$$
\pi(\mathbf{a}_{t} \mid \mathcal{E}_S(\mathbf{s}_{t}), \mathcal{E}_T(\mathbf{T}_t), \mathcal{E}_V(\mathbf{V}_t)),
$$
with an appendix variant
$$
\pi(\mathbf{a}_{t+1} \mid \mathcal{E}_S(\mathbf{s}_{t}), \mathcal{E}_T(\mathbf{T}_t), \mathcal{E}_V(\mathbf{V}_t)).
$$
Here, $\mathbf{s}_t$ denotes robot state or proprioception, $\mathbf{T}_t$ tactile input, $\mathbf{V}_t$ visual input, and $\mathcal{E}_S$, $\mathcal{E}_T$, and $\mathcal{E}_V$ the corresponding encoders [2509.14688]. The learned object of interest is the tactile encoder $\mathcal{E}_T$.

The core predictive objective is
$$
p_\theta\left(\mathbf{T}_{t+1:t+n} \mid \mathcal{E}_T(\mathbf{T}_{t-n+1:t}), \mathcal{E}_V(\mathbf{V}_t), \mathcal{E}_A(\mathbf{A}_{t-n+1:t+n})\right),
$$
which conditions future tactile frames on encoded tactile history, current visual observation, and an encoded action sequence [2509.14688]. This formulation is described as **task-agnostic** because it does not require task labels and **action-aware** because action is part of the conditioning set. The paper’s critique of prior tactile learning falls into three categories: direct multimodal imitation learning suffers from small task datasets and sparse contact; spatial self-supervised learning imports visual inductive biases such as translation invariance that may not hold for tactile images; and visual–tactile alignment assumes an overly simple one-to-one correspondence even though vision-to-touch is often one-to-many [2509.14688].

Architecturally, the method follows the general structure of the **Unified Video Action Model (UVA)** but adapts it to tactile prediction. A **VAE** serves as tactile encoder and decoder, each tactile image is patchified into embeddings, and prediction is implemented over **8 temporal frames**, using **4 random frames in the first half** as input and **4 frames in the second half** as prediction targets [2509.14688]. Random masking is applied to history tactile patch embeddings and action features; these are fused with a transformer; and the fused history latents are passed to a **latent diffusion model** that predicts future tactile latents. The denoising process is conditioned on future action embeddings and the current RGB image.

The training objective combines a diffusion loss and a reconstruction loss:
$$
\mathcal{L} = \mathcal{L}_{diff} + \mathcal{L}_{recon}.
$$
The paper defines $\mathcal{L}_{diff}$ as “the regular diffusion loss between the predicted and actual noise perturbations” and $\mathcal{L}_{recon}$ as the reconstruction MSE between reconstructed and original tactile images [2509.14688]. It does not provide a full closed-form diffusion parameterization in the main description.

The pretraining and downstream stages are separated. First, the tactile encoder $\mathcal{E}_T$ is learned inside the tactile prediction model on the large play dataset. Then the encoder is **frozen**, and a multimodal imitation policy is trained using **Diffusion Policy** with a ViT image backbone and direct feature concatenation [2509.14688]. The tactile model is explicitly **not finetuned** downstream. This suggests that the paper treats tactile representation learning as a reusable pretraining problem rather than a task-specific auxiliary module.

## 6. Empirical results, limitations, and research significance

The paper evaluates exUMI in both non-tactile and tactile-aware settings. On non-tactile imitation tasks, a vision-only policy trained on exUMI demonstrations achieves **85%** on Pick Cube, **80%** on Pick Carrot, **60%** on Pick Broccoli, **60%** on Stack Cube, **65%** on Insert Pen, **70%** on Put Ball, **20%** on Open Bottle, **40%** on Pull Drawer, and **50%** on Peg in Hole [2509.14688]. These results are presented mainly as evidence that the device’s proprioception and demonstration quality are sufficient for robot policy learning even before tactile modeling is added.

The more consequential results concern tactile-aware tasks evaluated over **20 trials**. For Put Ball, the reported success rates are **70%** for Vision Only, **70%** for Vision + Tactile, and **85%** for Vision + Tactile + TPP. For Open Bottle, the rates are **20%**, **50%**, and **60%**. For Pull Drawer Empty they are **100%**, **100%**, and **100%**. For Pull Drawer Random they are **40%**, **50%**, and **95%**. For Peg Grasp they are **100%**, **100%**, and **100%**. For Peg Insert they are **50%**, **60%**, and **80%** [2509.14688]. The paper summarizes these as **over 20% gains over tactile learning baselines**, with especially strong improvements in force-sensitive stages such as random-weight drawer pulling and peg insertion.

The pretraining ablations support the paper’s action-aware design. For tactile prediction MSE under different modality combinations, the reported values are **0.0298**, **0.0132**, **0.0125**, **0.0117**, and **0.0099**, with the written conclusion that multimodal conditioning reduces prediction error and that action-aware prediction is best [2509.14688]. A separate representation-learning ablation in the appendix reports, for Put Ball and Peg in Hole respectively, **70%/50%** for V only, **70%/60%** for V+T Direct, **80%/50%** for V+T BYOL, and **85%/80%** for V+T TPP.

The paper also reports qualitative attention analysis indicating that the TPP tactile encoder focuses on regions corresponding to **force magnitude** and **tangential force direction** [2509.14688]. This is used to support the interpretation that the learned features encode physically meaningful contact cues rather than merely appearance regularities in tactile frames.

Several limitations are explicitly acknowledged. On the hardware side, users reported **thermal discomfort and neck strain** from the AR headset; more accurate alternatives such as Vive trackers often require external base stations and therefore conflict with the portability objective; and the tactile sensors, though improved over 9DTact, still require better durability and consistency [2509.14688]. On the algorithmic side, the paper notes **limited action information**, **limited visual viewpoint**, and **imperfect prediction performance**. Proposed future directions include adding **force-torque measurements**, incorporating **multi-view vision**, improving ergonomics, and broadening open-source deployment and crowd-sourced data collection.

The paper states that exUMI offers open-source resources through its project page, and that **CAD files**, **code**, **hardware**, and **datasets** are intended for release [2509.14688]. More broadly, exUMI situates tactile robot learning as a joint systems-and-representation problem. The paper’s central implication is not merely that tactile sensing helps, but that useful tactile learning requires synchronized action-touch-vision data at sufficient scale and with sufficient contact density to support predictive pretraining.

Source: https://www.emergentmind.com/topics/exumi