Portable Human Demonstration (UMI)

• Portable Human Demonstration (UMI) is a modular framework that captures rich, high-fidelity robotic manipulation data using handheld grippers with integrated multimodal sensors.
• It combines egocentric vision, 6-DoF pose tracking, and proprioceptive feedback to decouple data collection from specific robot embodiments, enabling scalable policy learning.
• UMI supports diverse real-world tasks from industrial pick-and-place to surgical applications while addressing challenges like user ergonomics, SLAM robustness, and fine-grained skill segmentation.

Portable Human Demonstration (UMI) refers to a class of hardware and algorithmic frameworks that enable the capture of rich, high-fidelity robotic manipulation demonstrations by untrained humans using portable, robot-independent interfaces. The core paradigm exploits hand-held, instrumented grippers or surrogate devices with integrated vision, proprioception, and sometimes multimodal sensing (e.g., force/torque, tactile), operating independently of any physical robot during the demonstration phase. This approach decouples data collection from robot hardware, supporting large-scale, in-the-wild acquisition of diverse manipulation trajectories for scalable policy learning and cross-embodiment deployment.

1. Hardware Architectures and Sensing Modalities

Portable Human Demonstration devices are grounded in the principle of embodiment-agnostic, high-bandwidth measurement of human manipulation. Canonical implementations (e.g., UMI, FastUMI) consist of:

• Handheld Grippers: 3D-printed, lightweight fixtures (typically 250–1200 g) mimicking parallel jaw or custom robot gripper kinematics. Finger widths are monitored via fiducial markers or encoders (Chi et al., 2024, Gupta et al., 2 Oct 2025, Zhaxizhuoma et al., 2024).
• Egocentric Vision: Rigidly mounted wrist or fisheye cameras (GoPro, OAK-1W, Hikrobot, RealSense) providing the robot-aligned visual stream at 20–60 Hz (Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025, Zhaxizhuoma et al., 2024). Side mirrors are sometimes used to expand FoV and provide stereo or depth cues (Chi et al., 2024).
• 6-DoF Pose Tracking: Mechanical alignment is preserved via visual-inertial SLAM (e.g., ORB-SLAM3, ARKit), VSLAM modules (e.g., RealSense T265), or external fiducials (AprilTag, HTC Vive) to yield globally consistent wrist/EE trajectories at up to 200 Hz (Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025, Huang et al., 12 Nov 2025).
• Gripper Proprioception: Continuous recording of gripper opening via visual fiducials, encoders, or derived from tactile sensors (Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025).
• Advanced Sensing Extensions: Integration of depth cameras or LiDAR for direct point cloud capture (UMI-3D, UMIGen) (Wang, 15 Apr 2026, Huang et al., 12 Nov 2025), 6-axis F/T sensors and tactile arrays for contact-rich tasks (UMI-FT, OmniUMI, TacUMI) (Choi et al., 15 Jan 2026, Luo et al., 12 Apr 2026, Cheng et al., 21 Jan 2026).
• Portable Power and Data Streaming: Battery operation and Wi-Fi/USB interfaces enable untethered, field use (Gupta et al., 2 Oct 2025, Zhaxizhuoma et al., 2024).

Table 1: Representative Sensor Setups

Device	Vision	Pose Tracking	Force/Tactile	Other
UMI	GoPro fisheye	ORB-SLAM3/IMU	No	Side mirrors
FastUMI	GoPro fisheye	RealSense T265	No	Modular mount
UMI-3D	Fisheye	LiDAR-centric SLAM	No	LiDAR MID-360
UMI-FT	iPhone RGB-D	ARKit	CoinFT 6-axis	Fin-ray hands
TacUMI	Fisheye+3rd RGB	HTC Vive Tracker	Bota SensONE	Gelsight Mini
OmniUMI	Fisheye+depth	IMU/MoCap	F/T + tactile	Motor sensing

2. Data Acquisition, Calibration, and Synchronization

Portability is enforced by minimizing demands on the environment and facilitating rapid setup. Key protocol elements:

• Global Frame Alignment: Single-session mapping procedures (scene mapping, three-point calibration, hand–eye routines) tie all measurements to a reproducible world or task frame (Liu et al., 9 Oct 2025, Zou et al., 9 Apr 2026, Chi et al., 2024).
• Synchronized Logging: Raw streams—RGB-D frames, 6-DoF pose, gripper widths, and, if present, force/tactile vectors—are hardware timestamped and aligned via a unified ROS or custom software clock (Liu et al., 9 Oct 2025, Zhaxizhuoma et al., 2024, Huang et al., 12 Nov 2025).
• Automated Segmentation: Event-based heuristics (e.g., gripper release/open, proximity to waypoint) and chunking strategies extract discrete demonstrations or skill primitives from uninterrupted, high-throughput sessions (San-Miguel-Tello et al., 11 Jun 2025, Liu et al., 9 Oct 2025, Huang et al., 12 Nov 2025).
• Coordinate Transformation: Multi-sensor devices use fixed extrinsics to express all sensory readings in the end-effector frame, removing the need for per-platform calibration except for a one-time camera-to-tool registration (Liu et al., 9 Oct 2025, Huang et al., 12 Nov 2025, Chi et al., 2024).

3. Policy Interfaces, Learning Formulations, and Embodiment-Agnostic Representations

All UMI-style systems are architected to enable learned policies that transfer directly across robots:

• Relative Trajectory Representation: Policies predict relative SE(3) or Δ-action sequences (future horizon over position/orientation/gripper state) with respect to the current EE pose, naturally bridging hardware differences (Chi et al., 2024, Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025, Huang et al., 12 Nov 2025).
• Multimodal Observations: Input to the policy commonly consists of a sliding window over synchronized vision, proprioception, and—if available—force/tactile signals (Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025, Luo et al., 12 Apr 2026, Choi et al., 15 Jan 2026).
• Conditional Diffusion Models and Transformers: Diffusion policy architectures (U-Net/Transformer-based) are now the dominant approach for learning generative action-rollouts conditioned on rich observation states (Chi et al., 2024, Gupta et al., 2 Oct 2025, Huang et al., 12 Nov 2025, Wang, 15 Apr 2026).
• Vision-Language Pretraining: Integration of CLIP, ViT, or DINO backbones for vision, and sometimes vision-language-action (VLA) models for semantic grounding (Liu et al., 9 Oct 2025, Liu et al., 3 Feb 2026, Zeng et al., 2 Oct 2025).

The strict separation of the observation–action interface from any specific robot (action in EE or TCP space, gripper widths, and camera-aligned observations) ensures plug-and-play deployment, as any arm can mirror the camera-gripper geometry and use the raw policy output (Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025, Hou et al., 2024).

4. Embodiment-Aware and Embodiment-Agnostic Deployment

While the core strength of UMI is in “embodiment-agnostic” skill acquisition, deployment on physically constrained embodiments (e.g., aerial, mobile, or humanoid platforms) is addressed through hybrid control stacks:

• Low-Level Controllers: Reference trajectory from the UMI policy is mapped into robot joint space via standard inverse kinematics (damped pseudo-inverse) or model predictive control (MPC) for dynamics-limited platforms (Gupta et al., 2 Oct 2025).
• Controller-Guided Diffusion: The Embodiment-Aware Diffusion Policy (EADP) augments diffusion sampling with gradient guidance from control feasibility costs, producing dynamically valid, hardware-tailored trajectories at inference without retraining (Gupta et al., 2 Oct 2025).
• Cross-Embodiment, Plug-and-Play Transfer: Zero-shot deployment is realized by assembling the demonstration sensor suite (gripper plus camera) onto the target robot and mapping EE trajectories using a fixed hand–eye calibration; policy checkpoints are not retuned (Liu et al., 9 Oct 2025, Huang et al., 12 Nov 2025, Chi et al., 2024).

Table 2: Success Rate Improvement from EADP (DP=Standard Diffusion Policy, EADP=With Controller Guidance) (Gupta et al., 2 Oct 2025)

Platform	DP	EADP	ΔSuccess
UR10e	82%	89%	+7%
UAM (aerial)	63%	72%	+9%
UAM+disturbance	45%	66%	+21%
Peg-in-hole, real	0/5	5/5	+100%

5. Experimental Evaluation and Generalization

UMI-based systems have been subjected to extensive benchmarking across embodiments and domains:

• Manipulation Task Breadth: Evaluated on pick-and-place, peg-in-hole, valve-turning, dynamic tossing, cloth folding, pouring, surface wiping, surgical bandage opening, and agricultural harvesting (Gupta et al., 2 Oct 2025, Chi et al., 2024, San-Miguel-Tello et al., 11 Jun 2025, Choi et al., 15 Jan 2026, Liu et al., 9 Oct 2025, Georgadarellis et al., 17 Mar 2026).
• Performance Metrics: Quantified by success rate per sub-task (typically 50–100 trials per policy), trajectory-level metrics (completion time, contact forces), and generalization across novel objects, environments, and robots.
• Real-world and Cross-Embodiment Transfer: DP models trained on FastUMI-100K or similar UMI-style datasets were deployed without fine-tuning across multiple robots (Xarm6, Flexiv Rizon4), achieving ≥66% success on complex tasks and high robustness under perturbations (Liu et al., 9 Oct 2025, Zhaxizhuoma et al., 2024).
• Data Quality and Operator Throughput: Modern UMI rigs (FastUMI, UMI-3D) reduce setup time by >90% and enable one operator to collect 3–4× more data per unit time than with direct teleoperation, narrowing the user-fatigue gap relative to bare-hand demonstrations (Zhaxizhuoma et al., 2024, Wang, 15 Apr 2026).
• Multimodal and Contact-Rich Scenarios: Advanced UMI variants (UMI-FT, TacUMI, OmniUMI) demonstrate significant gains in force-sensitive and long-horizon tasks via the inclusion of force/tactile data—enabling >92% success on compliant insertion, 94% contact event segmentation, and robust performance on pick-and-place under hard-to-see conditions (Choi et al., 15 Jan 2026, Luo et al., 12 Apr 2026, Cheng et al., 21 Jan 2026).

6. Limitations, Design Trade-offs, and Future Directions

Several constraints and open design questions are prominent:

• User Ergonomics and Demonstration Fidelity: Even with lightweight construction and ergonomic redesign (e.g., concentrated load grippers), human demonstration is 4–15× slower and physically more demanding than bare-hand performance, especially for fine manipulations (Georgadarellis et al., 17 Mar 2026). Future refinements emphasize weight reduction (<400 g), modular fingers, and improved feedback.
• SLAM Robustness: Vision-based tracking can fail in textureless/outdoor settings; LiDAR or external marker fusion as in UMI-3D and UMIGen addresses this but increases sensor cost and complexity (Wang, 15 Apr 2026, Huang et al., 12 Nov 2025, San-Miguel-Tello et al., 11 Jun 2025).
• Embodiment Gap in Non-Rigid or Whole-Body Tasks: For highly dynamic, flexible, or mobile robot platforms, naïve transfer is limited. Solutions include hierarchical control architectures (HoMMI, BifrostUMI), explicit kinematic retargeting, and additional proprioceptive/context observation streams (Yu et al., 5 May 2026, Xu et al., 3 Mar 2026).
• Contact-Rich and Fine-Grained Segmentation: Tightly synchronized, multimodal data (vision, force, tactile, precise pose) allows for robust skill segmentation (TacUMI >94% framewise accuracy), supporting modular policy learning for complex behaviors (Cheng et al., 21 Jan 2026).
• Open Research Questions: Scalability to outdoor and high-speed applications, seamless haptic feedback for human operators, and joint vision-language-action policy pretraining remain active frontiers.

A plausible implication is that portable human demonstration with UMI-class interfaces, empowered by multimodal sensing and modular design, will become foundational for robotics at scale—enabling generalist, cross-platform, and contact-rich manipulation policy learning with strong real-world and embodiment robustness (Chi et al., 2024, Gupta et al., 2 Oct 2025, Liu et al., 9 Oct 2025).