H2R Aligner: Human-to-Robot Visual Alignment
- H2R Aligner is a data augmentation technique that converts egocentric human manipulation videos into robot-compatible visual representations.
- The method employs a pipeline of hand keypoint detection, hand removal with inpainting, motion synthesis, and accurate camera alignment for compositing.
- Empirical evaluations show that H2R consistently improves downstream task performance in both simulated and real-world robotic environments.
Searching arXiv for papers on H2R and related human-to-robot alignment methods. arxiv_search(query="H2R human-to-robot data augmentation robot pre-training videos", max_results=5, sort_by="relevance") arxiv_search.search({"query":"H2R human-to-robot data augmentation robot pre-training videos","max_results":5,"sort_by":"relevance"}) “H2R Aligner” (Editor’s term) denotes the use of H2R as a human-to-robot visual domain alignment method for robot pre-training from videos, where egocentric human manipulation footage is transformed into robot-centric visual training data by detecting human hand keypoints, synthesizing robot motions in simulation, and compositing rendered robots into the original scene (Li et al., 17 May 2025). In the cited literature, the core objective is to mitigate the visual embodiment gap between human hands and robot manipulators during representation learning. Related work extends the same alignment objective beyond augmentation into paired-data-free human-to-robot video translation, suggesting that “H2R alignment” is better understood as a methodological direction than as a single fixed pipeline (Ci et al., 10 Dec 2025).
1. Problem formulation and alignment objective
H2R is motivated by a mismatch between the scale of available egocentric human manipulation video and the cost of collecting equivalent robot demonstrations. The datasets named explicitly in this context are Ego4D and Something-Something V2 (SSv2), which provide abundant human video, whereas robot data remain comparatively expensive and labor-intensive to acquire (Li et al., 17 May 2025). The central claim is that direct pre-training on human footage is suboptimal for robot learning because human videos contain human hands and arms, while deployment involves robot arms / grippers / dexterous hands. Even when task semantics are shared, the acting agent’s appearance differs substantially.
Within this formulation, alignment means altering the pre-training distribution so that it remains semantically tied to the original manipulation while becoming visually closer to the downstream robotic setting. H2R therefore does not introduce a separate policy module; it is a data augmentation technique used before downstream policy learning. The intended effect is to reduce the amount of embodiment adaptation that downstream fine-tuning must absorb, so that visual encoders can emphasize manipulation-relevant content rather than spend capacity learning a human-to-robot appearance translation (Li et al., 17 May 2025).
A related but distinct formulation appears in H2R-Grounder, where the embodiment gap is expressed as a mismatch between human hand pose and robot gripper pose, and the alignment mechanism is a shared intermediate representation consisting of a manipulator pose trajectory plus an inpainted background video. This suggests a broader notion of H2R alignment in which the key operation is to factor out scene state from embodiment-specific appearance, then re-render or regenerate the scene under a robot embodiment (Ci et al., 10 Dec 2025).
2. Canonical H2R pipeline
The H2R pipeline in its original form has four stages: human hand keypoint detection and pose reconstruction, human hand/arm removal, motion synthesis in simulation, and camera alignment plus compositing (Li et al., 17 May 2025).
In the first stage, HaMeR is used to estimate 3D hand keypoints / hand pose together with camera intrinsics and extrinsics. This geometric estimate provides the retargeting signal from a human hand into a robot embodiment. In the second stage, SAM (Segment Anything Model) segments the human hand and arm regions automatically, after which the segmented region is removed and LaMa is used to inpaint the removed area. The result is a clean background prepared for robot insertion.
The third stage synthesizes robot motion in SAPIEN. For dexterous hands, finger joint angles are computed from the angle formed by three consecutive keypoints on a finger segment. For grippers, opening and closing are determined by the Euclidean distance between relevant fingertips. For arm joints without direct hand correspondence, values are manually assigned plausible values. The paper characterizes this as a pragmatic pose-driven reconstruction rather than full physics-based retargeting from action labels (Li et al., 17 May 2025).
The final stage aligns the simulated robot view to the original egocentric viewpoint and composites the rendering into the inpainted frame. The camera transformation is written as
The human and robot coordinate systems are built from finger keypoint directions:
$^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$
After rendering from the aligned camera, the robot mask and image are projected into pixel space and overlayed / composited onto the inpainted frame. The resulting output preserves the original manipulation context while replacing the visible human effector with a robot embodiment (Li et al., 17 May 2025).
3. Datasets, representations, and evaluation proxies
H2R is instantiated at scale through a family of H2R-1M datasets built from SSv2 and Ego4D (Li et al., 17 May 2025). From SSv2, the paper selects 62,500 videos from the official training split and samples 16 keyframes per video, yielding exactly 1,000,000 frames. The named products are H2R-UR5-SSv2-1M-Gripper, H2R-UR5-SSv2-1M-Leaphand, H2R-UR5-SSv2-1M-Mix, and H2R-Franka-SSv2-1M. From Ego4D, the construction uses 117,624 action clips from 2,486 Ego4D videos, with 3 keyframes per clip, producing H2R-UR5-Ego4D-1M-Gripper, H2R-UR5-Ego4D-1M-Leaphand, H2R-UR5-Ego4D-1M-Mix, and H2R-Franka-Ego4D-1M.
These datasets include more than frames alone. The listed metadata are the original label, 3D hand keypoints, camera intrinsics and extrinsics from HaMeR, inpainted background, robot joint configurations, and aligned simulator camera pose. This makes H2R a modular data resource as well as an augmentation pipeline.
To verify that augmentation preserves action semantics, the paper introduces a CLIP-based image-text similarity metric with CLIP ViT-B/32:
The text prompts use two templates: “A human is [action]” and “A robotic arm is [action]”. In the appendix’s six paired samples, the reported original scores are 30.6, 23.7, 31.2, 31.7, 27.5, 28.7, while the augmented scores are 32.6, 28.9, 32.3, 32.0, 28.7, 31.0. The paper interprets this as evidence that the augmented images are more semantically aligned with robot-centric descriptions (Li et al., 17 May 2025).
4. Pre-training regime and empirical performance
H2R is used for visual encoder pre-training rather than as a stand-alone control policy. The named pre-training frameworks are MAE and R3M, using a ViT-B/32 / ViT-Base style backbone depending on framework. Downstream policies include Behavior Cloning (BC) for Robomimic, Diffusion Policy (DP), and ACT (Li et al., 17 May 2025).
The principal empirical claim is that H2R consistently improves downstream success rates, with reported gains of 5.0%-10.2% in simulation and 6.7%-23.3% in real-world tasks across visual encoders and policy learning methods. The simulation benchmarks are Robomimic, RLBench, and PushT; the real-world system is a UR5 equipped with Robotiq gripper and Leaphand end-effectors, with an Intel RealSense L515 mounted on the side.
| Setting | Baseline | H2R |
|---|---|---|
| Robomimic, MAE average | 58 | 68.2 |
| Robomimic, R3M average | 55 | 61.3 |
| RLBench, MAE average | 3.3 | 13.3 |
| RLBench, R3M average | 10.0 | 15.0 |
| PushT, MAE | 59.2 | 64.5 |
| PushT, R3M | 15 | 22 |
On Robomimic, the MAE average increases from 58 to 68.2, with task-level changes such as MoveCan: 54 → 79.5, Square: 25.5 → 29.5, and Lift: 94.5 → 95.5. With R3M, the average rises from 55 to 61.3, including Square: 20.5 → 37.5. On RLBench, the MAE average moves from 3.3 to 13.3, and the R3M average from 10.0 to 15.0; listed MAE examples include Close Box: 0 → 5, Close Laptop Lid: 10 → 15, and Toilet Seat Down: 0 → 20. On PushT, MAE improves from 59.2 to 64.5, and R3M from 15 to 22 (Li et al., 17 May 2025).
The real-world UR5 results are similarly reported in paired before/after form. With SSv2 pre-training, DP on gripper tasks improves from 50 → 56.7 for MAE and 46.7 → 61.7 for R3M; on Leaphand tasks the gains are 40 → 60 for MAE and 21.7 → 43.3 for R3M. With ACT, gripper-task averages move from 26.7 → 35 for MAE and 28.3 → 40 for R3M, while Leaphand-task averages move from 33.3 → 46.7 for MAE and 15 → 38.3 for R3M. Cross-embodiment transfer from H2R-Franka-SSv2 to UR5 Leaphand remains beneficial but smaller, which the paper uses to argue that embodiment-matched augmentation is preferable though not strictly necessary (Li et al., 17 May 2025).
Ablations establish what the paper treats as the essential components of the aligner. H2R w/o Overlay, where the hand is removed but no robot is inserted, causes large degradation; examples listed are DP average: 60 → 30 and ACT average: 46.7 → 28.3. H2R w/o Retarget, where the robot is pasted without precise camera alignment, is also significantly worse, with examples DP average: 60 → 43.3 and ACT average: 46.7 → 30. Comparisons against CutMix1, CutMix2, and CutMix3 indicate that simple pasting can help on some tasks but is less consistent than pose- and viewpoint-matched compositing (Li et al., 17 May 2025).
5. Extension to generative human-to-robot grounding
A later development, H2R-Grounder, reformulates H2R alignment as paired-data-free human-to-robot video translation rather than static compositing (Ci et al., 10 Dec 2025). Its transferable representation is H2Rep, defined as inpainted background + overlaid 2D pose indicator. For robots, the system segments the arm with
projects the end-effector trajectory
inpaints the robot away,
and forms
The pose cue is rendered as a red dot for position and a blue arrow for orientation.
Training uses only unpaired robot videos, specifically DROID, described as about 76K third-person Franka videos. The generator is Wan 2.2 TI2V-5B, fine-tuned in an in-context learning style with a fixed text prompt, “A robotic arm is interacting with objects.” Both conditioning and target video are encoded with the same VAE:
with conditioning
The backbone is frozen and only LoRA adapters on the Q/K/V projections are trained under a flow-matching objective:
At test time, the person is segmented with
$^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$0
pose is estimated using ViT-Pose and HaMeR, and the hand pose surrogate uses the midpoint between index fingertip and thumb tip as position and thumb direction as orientation (Ci et al., 10 Dec 2025).
Evaluation is conducted on DexYCB and internet videos, with 22 participants in human studies and Gemini for VLM-based scoring. H2R-Grounder is reported as most preferred by humans across all named criteria: Motion consistency: 54.5%, Background consistency: 56.8%, Visual quality: 61.4%, and Physical plausibility: 63.6%. The corresponding VLM scores are 3.7, 4.9, 4.0, and 4.4. Ablations show that removing the pose indicator hurts motion control, removing LoRA degrades performance significantly, replacing the generator with VACE lowers SSIM and raises LPIPS, and scaling to a 14B backbone does not improve quality while hurting efficiency and sequence length (Ci et al., 10 Dec 2025). This suggests that later H2R work generalizes the aligner concept from geometric compositing to learned video generation while preserving the same core idea: a robot-compatible intermediate representation can mediate the embodiment gap.
6. Limitations, scope, and terminological ambiguity
The original H2R pipeline has explicit dependencies on the accuracy of HaMeR, SAM, and LaMa. It is reported to struggle in low-light conditions, partial occlusions, and poor hand visibility (Li et al., 17 May 2025). The implementation is limited to single-arm manipulation and does not yet cover dual-arm systems or humanoids. The paper also states that extension to navigation and mobile manipulation would be valuable. Reported real-world failure modes include grasp failure unnoticed by the policy, misaligned placement, poor contact localization, insufficient lifting trajectory, and perception noise.
H2R-Grounder has a related but distinct limitation profile. Its stated limitations are single-hand to single-arm only and Franka-only outputs, because training uses Franka robot videos; generalization to other robots would require additional fine-tuning or new LoRA adapters. The supplementary note cited in the data reports inference costs of about 13 seconds per frame and about 648 seconds for a 49-frame video on one H200 GPU (Ci et al., 10 Dec 2025).
A separate source of confusion is terminological. The string “H2R” is not unique to robotics. In arXiv literature it also appears in unrelated contexts, including H$^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$1 high-order harmonic generation, where the relevant topic is two-center interference in aligned H$^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$2 (Zwan et al., 2010), and in $^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$3 Higgs-sector physics, where alignment without decoupling is attributed to a non-abelian global $^{W}\mathbf{I}_H=\{^w\mathbf{i}_{H,x},^w\mathbf{i}_{H,y},^w\mathbf{i}_{H,z} \}, \qquad ^{W}\mathbf{I}_S=\{^w\mathbf{i}_{S,x},^w\mathbf{i}_{S,y},^w\mathbf{i}_{S,z} \}.$4 R-symmetry (Benakli et al., 2018). In robotics usage, however, H2R denotes human-to-robot alignment, either as a data augmentation technique for robot pre-training or, in later work, as a paired-data-free video translation paradigm (Li et al., 17 May 2025).
Taken together, these papers define H2R alignment as an attempt to preserve task semantics, scene context, and, increasingly, physical plausibility, while replacing the human embodiment with a robot-compatible one. The original H2R paper implements that goal through simulation and compositing; H2R-Grounder implements it through H2Rep-conditioned generative video modeling. A plausible implication is that the enduring contribution of the “H2R Aligner” idea lies less in any single pipeline than in the general principle that robot pre-training benefits when human video is first transformed into a representation whose visual statistics are closer to downstream robotic deployment.