exUMI: Tactile-Aware Robot Teaching System

• exUMI is a tactile-aware robot teaching system that integrates precise proprioception, modular visuo-tactile sensors, and synchronized data collection for enhanced contact-rich manipulation.
• It combines hardware innovations like AR-based 6D tracking and low-cost rotary encoder sensing with Tactile Prediction Pretraining to accurately forecast tactile dynamics.
• The system significantly increases effective tactile data density, leading to over 20% improvement in force-sensitive manipulation tasks in empirical evaluations.

exUMI is a tactile-aware robot teaching system that couples a portable demonstration device with an action-conditioned tactile representation learning pipeline for contact-rich manipulation. Introduced as an extensible upgrade to the original UMI, it is designed to address three intertwined limitations in tactile robot learning: data scarcity, tactile sparsity, and the lack of force-feedback capture in existing teaching systems. The system combines hardware modifications—robust proprioception, modular visuo-tactile sensing, centralized logging, calibration, and synchronization—with Tactile Prediction Pretraining (TPP), a task-agnostic method for learning tactile representations from aligned tactile, action, and visual streams. The core claim is that progress in tactile manipulation depends on co-designing collection infrastructure and representation learning rather than treating policy learning alone as the bottleneck (Xu et al., 18 Sep 2025).

1. Definition and problem setting

The exUMI paper describes the platform as an extensible upgrade to UMI for tactile-aware robot teaching and identifies it as a portable handheld physical twin of a robot gripper with accurate end-effector tracking, robust gripper-width sensing, integrated visuo-tactile fingertips, centralized multimodal logging, and automated calibration and temporal alignment (Xu et al., 18 Sep 2025). In the detailed description, exUMI is expanded as the extensible Universal Manipulation Interface. Its purpose is to bridge the gap between the human demonstrator’s tactile intuition and the sensory information made available to the robot learner.

The motivating problem is specific to contact-rich manipulation. Humans use force and touch to regulate grasping, insertion, twisting, sliding, and compliance-sensitive interaction, but many robot teaching systems record only vision and kinematics, or depend on fragile tracking pipelines that degrade multimodal data quality. The paper states that valid tactile contact often occupies less than 10% of manipulation trajectories in regular robot data collection, which makes direct end-to-end tactile imitation learning statistically inefficient (Xu et al., 18 Sep 2025). exUMI is therefore positioned not merely as a sensing add-on, but as an infrastructure for collecting high-quality, contact-rich, action-aligned tactile data.

A central historical comparison in the paper is with the original or “vanilla” UMI system. Vanilla UMI relied on visual SLAM for pose tracking and ArUco markers for gripper state estimation. The exUMI paper argues that these components are vulnerable to occlusion, difficult backgrounds, fisheye distortion, and preprocessing failure. It reports that exUMI improves the effective data ratio to nearly 100%, compared to less than 60% for vanilla UMI in the authors’ pipeline, and that ArUco-based gripper estimation alone caused about 10% of preprocessing failures (Xu et al., 18 Sep 2025). This framing makes data usability—not only policy accuracy—a first-class systems objective.

2. Hardware architecture and multimodal sensing

The exUMI device is organized around three explicit design principles: precise robot proprioception, extensibility, and portability (Xu et al., 18 Sep 2025). Its hardware stack includes a Meta Quest 3 AR motion capture subsystem, an AS5600 magnetic rotary encoder for gripper width, an Orange Pi controller as a central sensor hub, a GoPro fisheye RGB camera, two fingertip visuo-tactile sensors, and optional compatibility with non-parallel grippers through an additional mechanical design.

A key design choice is the paper’s decision to disentangle proprioception. exUMI assigns 6D end-effector pose estimation to AR motion capture and gripper opening/width estimation to a dedicated encoder, rather than inferring both visually. For 6D tracking, the system uses a Meta Quest 3 headset that tracks a left VR controller attached to the handheld device by a custom mount. The paper presents this as a replacement for vulnerable SLAM-based tracking and emphasizes its portability, real-time 6D pose streaming, and robustness to occlusion. The appendix reports mean position errors of 5.4 mm, 2.3 mm, and 1.7 mm on the three axes, with rotation errors below 1 degree, and describes the average error as less than 10 mm compared to Fast-UMI. The tracking range is reported as 3 meters, which the paper characterizes as adequate for typical robot workspaces (Xu et al., 18 Sep 2025).

For gripper state sensing, exUMI replaces ArUco-based estimation with a low-cost AS5600 magnetic rotary encoder mounted above a joint with a radial magnet. The sensor provides 12-bit position readings, 4096 positions per revolution, and I$^2$C communication, while being immune to visual occlusion and incurring negligible compute overhead (Xu et al., 18 Sep 2025). This substitution directly targets one of the previously identified failure modes of vanilla UMI.

The tactile subsystem consists of two fingertip visuo-tactile sensors based on 9DTact, redesigned for practical deployment. The paper lists four modifications: a contact protection or beveled structure to secure the silicone gel under large tangential force, improved cable and power connections for durability, a custom mold for consistent silicone thickness across sensors, and LED board modifications to improve robustness and reduce power draw. The exUMI paper does not frame the sensor as a precision geometry sensor like GelSight; rather, it emphasizes low-cost capture of normal force, tangential force, deformation patterns, and fast recovery (Xu et al., 18 Sep 2025). This distinction matters because the downstream representation learning objective depends on temporal contact dynamics rather than fine surface reconstruction.

The Orange Pi single-board computer acts as a universal sensor hub. It synchronously logs AR headset and controller data, encoder data, tactile streams, and potentially other modalities. The paper explicitly states that the design supports adding tactile, audio, or other custom sensors, making extensibility an architectural property rather than an afterthought (Xu et al., 18 Sep 2025).

3. Calibration, synchronization, and robot embodiment

A major systems contribution of exUMI is its calibration and synchronization pipeline. The paper argues that tactile learning is fundamentally about contact dynamics, so temporal misalignment between action, touch, and vision corrupts supervision even when each modality is individually high quality (Xu et al., 18 Sep 2025).

The system uses two one-time calibrations. In AR controller calibration, the operator aligns exUMI with the base coordinate frame in AR space and records the controller transform, which is then used to correct future pose tracking. In gripper state calibration, the user places the gripper at 1 cm intervals, records AS5600 readings, and interpolates a mapping from encoder readings to absolute gripper width (Xu et al., 18 Sep 2025). These procedures are intentionally lightweight, reducing the burden of deployment while preserving reliable proprioception.

For cross-modal alignment, the paper introduces a latency calibration protocol. At the start of data collection, the operator sweeps exUMI horizontally above an ArUco marker; the system extracts the AR system’s x-axis motion and the marker trajectory in video, then searches for the offset minimizing the mean-squared discrepancy. The appendix gives the objective informally as finding $\delta^*$ such that

$f(t) \approx g(t+\delta^*)$

and minimizing

$$\delta^* = \arg\min_{\delta_k} \sum_{i=1}^{T}\left\| f(t_i)-g(t_i+\delta_k)\right\|_2^2 .$$

The main paper states that synchronization error is under 50 ms, while the appendix reports less than 5 ms latency error with proper sweep frequency for the AR/vision calibration (Xu et al., 18 Sep 2025). The difference reflects separate reporting contexts rather than a single reconciled number.

Embodiment consistency is another design axis. exUMI is a physical twin of the target robot gripper, and the paper evaluates on a Flexiv Rizon 4 arm with a Flexiv Grav adaptive gripper. For robot deployment, the authors replicate the exUMI sensor placement on the robot using a pipe-clamp-style mount for the GoPro and tactile sensors (Xu et al., 18 Sep 2025). This reduces train–test mismatch between human demonstration and robot execution. The robot is controlled at 10 Hz from a master computer with an RTX 4070 GPU.

The bill of materials totals \$698**: **\$298 for a GoPro 11 plus accessories, \$299** for a Meta Quest headset, **\$35 for an Orange Pi 3B, \$1** for the AS5600, **\$15 for 3D-printed parts, \$30** for visuo-tactile sensors, and **\$20 for miscellaneous components (Xu et al., 18 Sep 2025). The paper explicitly characterizes the system as low-cost, DIY-friendly, portable, and open-source.

4. Data collection regimes and tactile dataset design

The paper uses exUMI in two distinct data-collection regimes: task demonstrations for downstream imitation learning and human play data for tactile representation pretraining (Xu et al., 18 Sep 2025). This bifurcation is central to the system’s design philosophy. Task demonstrations provide behaviorally relevant supervision, while play data provide contact diversity and density.

For downstream tasks, users collect about 100 to 200 demonstrations per task. The paper reports examples such as 204 demos in 42 min for Pick Cube, 135 demos in 31 min for Pick Carrot, 170 demos in 47 min for Pick Broccoli, 201 demos in 26 min for Stack Cube, 139 demos in 60 min for Insert Pen, 181 demos in 66 min for Put Ball, 270 demos in 79 min for Open Bottle, 202 demos in 70 min for Pull Drawer, and 163 demos in 56 min for Peg in Hole (Xu et al., 18 Sep 2025). It also states that for a simple pick-and-place task, a user can collect 100 demonstrations in 20 minutes and train a behavioral cloning policy that exceeds 70% success rate.

The pretraining corpus is more distinctive. The paper reports interaction in 10 real-world environments with 300+ objects, spanning rigid tools, deformable fabrics, and granular materials, yielding over 1 million frames of aligned image–tactile–action data (Xu et al., 18 Sep 2025). Other sections report 480K tactile frames in 5 hours of human interaction and 480.9K raw tactile frames in an appendix comparison. The paper explicitly notes that the exact accounting varies across sections, but consistently presents the corpus as substantially larger than prior tactile datasets.

This play-based collection strategy is designed to overcome tactile sparsity. Whereas ordinary demonstrations contain less than 10% meaningful tactile contact, the paper reports that in the play dataset over 60% of frames are active tactile frames, yielding more than 10× efficiency relative to teleoperation for collecting tactile frames (Xu et al., 18 Sep 2025). A plausible implication is that exUMI’s main statistical advantage lies not only in raw scale but in raising the density of informative contact transitions.

The recorded modalities include aligned RGB image data, tactile images from both fingertips, action or proprioception, and gripper state. Action is represented as relative pose and gripper state. For tactile processing, the left and right tactile images are concatenated, converted to calibrated grayscale, transformed into a 3-channel representation consisting of grayscale + convex map + concave map, and downsampled to 224 × 224 (Xu et al., 18 Sep 2025). The training/validation split is 15:1.

Preprocessing includes temporal synchronization via latency calibration, alignment of AR and video streams by interpolation, generation of synchronized packets, and a data rejection strategy that filters out trivial segments during pretraining: if all tactile frames in a chunk have active pixel proportions below a threshold, the chunk is discarded and resampled (Xu et al., 18 Sep 2025). This preprocessing strategy is explicitly motivated by tactile sparsity rather than by conventional dataset cleanliness alone.

5. Tactile Prediction Pretraining

The algorithmic counterpart to exUMI is Tactile Prediction Pretraining (TPP), which the paper defines as a representation learning framework based on action-aware temporal tactile prediction (Xu et al., 18 Sep 2025). The stated motivation is that tactile perception should not be treated as a static image understanding problem. Instead, touch is modeled as a temporally evolving signal whose semantics depend on action.

The downstream tactile-aware policy is written as

$$\pi(\mathbf{a}_{t} \mid \mathcal{E}_S(\mathbf{s}_{t}), \mathcal{E}_T(\mathbf{T}_t), \mathcal{E}_V(\mathbf{V}_t)),$$

with an appendix variant

$$\pi(\mathbf{a}_{t+1} \mid \mathcal{E}_S(\mathbf{s}_{t}), \mathcal{E}_T(\mathbf{T}_t), \mathcal{E}_V(\mathbf{V}_t)).$$

Here, $\delta^*$0 denotes robot state or proprioception, $\delta^*$1 tactile input, $\delta^*$2 visual input, and $\delta^*$3, $\delta^*$4, and $\delta^*$5 the corresponding encoders (Xu et al., 18 Sep 2025). The learned object of interest is the tactile encoder $\delta^*$6.

The core predictive objective is

$\delta^*$7

which conditions future tactile frames on encoded tactile history, current visual observation, and an encoded action sequence (Xu et al., 18 Sep 2025). This formulation is described as task-agnostic because it does not require task labels and action-aware because action is part of the conditioning set. The paper’s critique of prior tactile learning falls into three categories: direct multimodal imitation learning suffers from small task datasets and sparse contact; spatial self-supervised learning imports visual inductive biases such as translation invariance that may not hold for tactile images; and visual–tactile alignment assumes an overly simple one-to-one correspondence even though vision-to-touch is often one-to-many (Xu et al., 18 Sep 2025).

Architecturally, the method follows the general structure of the Unified Video Action Model (UVA) but adapts it to tactile prediction. A VAE serves as tactile encoder and decoder, each tactile image is patchified into embeddings, and prediction is implemented over 8 temporal frames, using 4 random frames in the first half as input and 4 frames in the second half as prediction targets (Xu et al., 18 Sep 2025). Random masking is applied to history tactile patch embeddings and action features; these are fused with a transformer; and the fused history latents are passed to a latent diffusion model that predicts future tactile latents. The denoising process is conditioned on future action embeddings and the current RGB image.

The training objective combines a diffusion loss and a reconstruction loss:

$\delta^*$8

The paper defines $\delta^*$9 as “the regular diffusion loss between the predicted and actual noise perturbations” and $f(t) \approx g(t+\delta^*)$0 as the reconstruction MSE between reconstructed and original tactile images (Xu et al., 18 Sep 2025). It does not provide a full closed-form diffusion parameterization in the main description.

The pretraining and downstream stages are separated. First, the tactile encoder $f(t) \approx g(t+\delta^*)$1 is learned inside the tactile prediction model on the large play dataset. Then the encoder is frozen, and a multimodal imitation policy is trained using Diffusion Policy with a ViT image backbone and direct feature concatenation (Xu et al., 18 Sep 2025). The tactile model is explicitly not finetuned downstream. This suggests that the paper treats tactile representation learning as a reusable pretraining problem rather than a task-specific auxiliary module.

6. Empirical results, limitations, and research significance

The paper evaluates exUMI in both non-tactile and tactile-aware settings. On non-tactile imitation tasks, a vision-only policy trained on exUMI demonstrations achieves 85% on Pick Cube, 80% on Pick Carrot, 60% on Pick Broccoli, 60% on Stack Cube, 65% on Insert Pen, 70% on Put Ball, 20% on Open Bottle, 40% on Pull Drawer, and 50% on Peg in Hole (Xu et al., 18 Sep 2025). These results are presented mainly as evidence that the device’s proprioception and demonstration quality are sufficient for robot policy learning even before tactile modeling is added.

The more consequential results concern tactile-aware tasks evaluated over 20 trials. For Put Ball, the reported success rates are 70% for Vision Only, 70% for Vision + Tactile, and 85% for Vision + Tactile + TPP. For Open Bottle, the rates are 20%, 50%, and 60%. For Pull Drawer Empty they are 100%, 100%, and 100%. For Pull Drawer Random they are 40%, 50%, and 95%. For Peg Grasp they are 100%, 100%, and 100%. For Peg Insert they are 50%, 60%, and 80% (Xu et al., 18 Sep 2025). The paper summarizes these as over 20% gains over tactile learning baselines, with especially strong improvements in force-sensitive stages such as random-weight drawer pulling and peg insertion.

The pretraining ablations support the paper’s action-aware design. For tactile prediction MSE under different modality combinations, the reported values are 0.0298, 0.0132, 0.0125, 0.0117, and 0.0099, with the written conclusion that multimodal conditioning reduces prediction error and that action-aware prediction is best (Xu et al., 18 Sep 2025). A separate representation-learning ablation in the appendix reports, for Put Ball and Peg in Hole respectively, 70%/50% for V only, 70%/60% for V+T Direct, 80%/50% for V+T BYOL, and 85%/80% for V+T TPP.

The paper also reports qualitative attention analysis indicating that the TPP tactile encoder focuses on regions corresponding to force magnitude and tangential force direction (Xu et al., 18 Sep 2025). This is used to support the interpretation that the learned features encode physically meaningful contact cues rather than merely appearance regularities in tactile frames.

Several limitations are explicitly acknowledged. On the hardware side, users reported thermal discomfort and neck strain from the AR headset; more accurate alternatives such as Vive trackers often require external base stations and therefore conflict with the portability objective; and the tactile sensors, though improved over 9DTact, still require better durability and consistency (Xu et al., 18 Sep 2025). On the algorithmic side, the paper notes limited action information, limited visual viewpoint, and imperfect prediction performance. Proposed future directions include adding force-torque measurements, incorporating multi-view vision, improving ergonomics, and broadening open-source deployment and crowd-sourced data collection.

The paper states that exUMI offers open-source resources through its project page, and that CAD files, code, hardware, and datasets are intended for release (Xu et al., 18 Sep 2025). More broadly, exUMI situates tactile robot learning as a joint systems-and-representation problem. The paper’s central implication is not merely that tactile sensing helps, but that useful tactile learning requires synchronized action-touch-vision data at sufficient scale and with sufficient contact density to support predictive pretraining.