---
title: 'TacUMI: Multi-Modal Manipulation Segmentation'
url: https://www.emergentmind.com/topics/tacumi
type: topic
---

# TacUMI: Multi-Modal Manipulation Segmentation

TacUMI is a multi-modal data collection and segmentation system designed to facilitate understanding and learning of complex contact-rich manipulation tasks. Extending the Universal Manipulation Interface (UMI), TacUMI integrates high-resolution tactile sensors, a 6-axis force/torque (F/T) sensor, and drift-free 6-DoF pose tracking within a robot-compatible, handheld gripper platform. The system enables synchronized acquisition of tactile, force, pose, and RGB observations during human demonstrations, supporting robust task decomposition—an essential step for modular skill learning in long-horizon robotic manipulation [2601.14550].

## 1. Motivation and System Architecture

TacUMI addresses the challenges inherent in contact-rich, long-horizon manipulation tasks, where visual and proprioceptive data alone are insufficient to resolve subtle event transitions such as cable tensioning. The original UMI provided handheld, low-cost gripper hardware for recording vision and pose during demonstrations but lacked contact sensing and exhibited pose drift over extended trajectories. TacUMI augments this framework through:

- Integration of ViTac tactile sensors (GelSight Mini, 256×256 RGB at 16.7 Hz) into custom fingertip slots, providing deformation imaging.
- Rigid mounting of a 6-axis F/T sensor (Bota Systems SensONE, 1000 Hz) between gripper body and handle, designed to mimic a robot flange, which ensures recorded wrenches directly represent robot-environment interactions.
- Inclusion of a continuous rack-and-pinion trigger with mechanical self-locking, permitting operators to fix the gripper jaw width so that only external contact forces affect F/T readings.
- Use of an HTC Vive Tracker delivering drift-free 6-DoF pose data (60 Hz) registered to the tool center point (TCP).
- Acquisition of third-person RGB video using a RealSense D435i camera (60 Hz).
- All sensor streams are synchronized by resampling to the lowest common rate (16.7 Hz) via timestamp alignment, generating synchronized multi-modal observation vectors per frame.

## 2. Multi-Modal Segmentation Framework

TacUMI employs a multi-modal skill segmentation pipeline based on temporal neural architectures. At each time step $t$, the system extracts:

- 256-D embedding from tactile images via ResNet-50.
- 256-D embedding from third-person RGB images via ResNet-18 with GroupNorm and Spatial Softmax.
- 6-D vector from preprocessed F/T readings.
- 14-D vector from left/right TCP poses (each 7-D: position + quaternion).

These modalities are concatenated to form $x_t \in \mathbb{R}^{532}$ for temporal modeling. Three backbone architectures are evaluated: (1) three-layer bi-directional LSTM (128 hidden units/way), (2) Temporal Convolutional Network (TCN) with dilated 1D convolutions and residual blocks, and (3) a transformer encoder using sinusoidal position encoding and multi-head self-attention.

Training uses frame-wise cross-entropy loss over $C$ skill classes:
\[
\mathcal{L}(\theta) = -\sum_{t=1}^T \sum_{c=1}^C y_{t,c} \cdot \log p_{t,c}(\theta)
\]
Inference applies a sliding window of length 50 (stride 10) over $x_t$ sequences, extracting window-level logits. Full sequence labels are restored via soft voting:
\[
\hat y_t = \arg\max_c \frac{1}{K_t} \sum_{k=1}^{K_t} p_{t,c}^{(k)}
\]

## 3. Modality Contribution and Fusion Analysis

TacUMI exploits complementary sensing modalities for contact-rich skill segmentation:

- Visual data alone underperforms during contact-centric phases such as tensioning.
- Tactile signals disambiguate surface deformation states (e.g., cable stretching).
- F/T readings capture transitions between linear force and torque-dominated phases.
- Pose information primarily reflects tool motion but offers minimal discrimination during stationary phases.

Fusing all four modalities in the TacUMI segmentation framework yields robust boundary detection, achieving greater than 90% frame-wise accuracy, while vision-only ablation reduces accuracy to 76.1%. Modal ablations reveal that adding tactile or F/T data significantly increases segmentation performance.

| Modalities Included   | Frame-wise Accuracy (%) |
|-----------------------|------------------------|
| Camera only           | 76.1                   |
| Camera + Tactile      | 90.8                   |
| Camera + F/T          | 86.3                   |
| Camera + Pose         | 81.7                   |
| Camera + Tac + F/T    | 93.6                   |
| All (Cam+Tac+F/T+Pose)| ≈94.0                  |

As per-class F1 scores improve from 0.32 (transformer, vision only) to greater than 0.90 (bi-directional LSTM, full modalities), multi-modal fusion is essential for accurate and reliable event boundary detection [2601.14550].

## 4. Empirical Evaluation on Cable Mounting Tasks

TacUMI is validated on a physically and semantically complex cable mounting task involving sequential insertion of a cable into three U-clips with varying orientation. During data collection:

- The operator uses the TacUMI gripper with F/T and pose sensing in the right hand to apply tension and guide, and a second gripper with tactile and pose for gripping and inserting in the left hand.
- A third-person camera records RGB observations.

Task phases for segmentation include (C=5): idle, grasp cable, apply linear tension, apply torque (clip insertion), and release. Segmentation and boundary detection are evaluated using:

\[
\mathrm{Acc} = \frac{\#\mathrm{correct\_frames}}{\mathrm{total\_frames}} \times 100\%
\]
\[
\mathrm{Acc}_{\text{boundaries}} = \frac{\#\mathrm{correct\_transitions}}{\#\mathrm{true\_transitions}} \times 100\%
\]

Results demonstrate 94.0% frame-wise accuracy (full modality, bi-directional LSTM backbone) and boundary detection exceeding 90% across skill transitions. Data collection via TacUMI is efficient, requiring 1 min 10 s for demonstration compared to 4 min via teleoperation. Cross-platform generalization is shown on teleoperated Franka robots.

## 5. Core Contributions and Limitations

Key advances of TacUMI include:

1. Introduction of a handheld UMI-style device featuring synchronized tactile, F/T, and drift-free pose data, as well as a mechanically locked trigger for robot-compatible F/T data.
2. Development of a segmentation framework fusing vision, tactile, F/T, and pose with three temporal backbones for accurate event segmentation.
3. Achievement of over 90% segmentation accuracy and efficient demonstration collection, along with validated transfer to robotic execution.

Limitations:

- Evaluation is currently restricted to a single cable-mounting scenario, with generalizability to additional long-horizon tasks yet to be demonstrated.
- Rigorous sensor calibration, especially for pose-to-TCP and F/T frame transforms, is necessary; calibration errors can degrade segmentation performance.
- The "release" phase is very short (2–5 frames), causing boundary ambiguity, and the TCP pose offers limited discriminative information during this phase due to minimal tool motion.

## 6. Downstream Applications and Future Directions

TacUMI produces segmented, multi-modal demonstrations suitable for a variety of downstream robotics and manipulation tasks, including:

- Learning from Demonstration (LfD): construction of modular skill primitives for downstream policy learning or diffusion-based planning.
- Skill Library Development: creation of reusable, contact-rich manipulation primitives leveraging multi-modal embedding representations.
- Interactive Task Refinement: enabling human annotators to re-demonstrate or correct individual skill modules as opposed to entire trajectories.
- Industrial and Service Robotics: deployment in manufacturing pipelines (pneumatic/cable wiring, snap-fit insertion) and fine-grained in-hand manipulation.

A plausible implication is that extension and benchmarking of TacUMI on a broader spectrum of manipulation tasks could reveal its scalability and generalization properties, particularly in the context of modular and hierarchical skill learning for robots [2601.14550].

Source: https://www.emergentmind.com/topics/tacumi