EmbodiSwap: Synthetic Robot & Avatar Transfer

• EmbodiSwap is a dual-purpose method that transfers embodiment from in-the-wild human video to robot platforms and between avatars in mixed-reality setups.
• It employs a multi-stage synthetic overlay pipeline—from hand pose extraction to rendering in PyBullet—to produce paired training labels for zero-shot robot control.
• In mixed reality, the platform swaps avatar control streams to study self–other boundaries, revealing effects on joint action and body ownership.

EmbodiSwap is a method for producing photorealistic synthetic robot overlays over human video and using those overlays for zero-shot robot imitation learning; in related mixed-reality usage, the term also denotes a body-swapping platform for exchanging users’ avatar control streams in a shared augmented environment (Dessalene et al., 4 Oct 2025, He et al., 11 Sep 2025). Across these usages, the unifying concept is embodiment transfer: in the robotics setting, the transfer is from in-the-wild ego-centric human video to a target robot embodiment; in the mixed-reality setting, the transfer is between two human participants’ avatars. The term therefore spans both a data-generation pipeline for robot learning and an experimental apparatus for studying self–other boundaries under avatar exchange.

1. Embodiment transfer as a technical problem

In robot imitation learning, EmbodiSwap is introduced to bridge the embodiment gap between in-the-wild ego-centric human video and a target robot embodiment (Dessalene et al., 4 Oct 2025). The motivating premise is that egocentric human video of hand–object interactions is abundant, whereas robot demonstrations are expensive to collect and re-label for each new arm, gripper, or environment. EmbodiSwap addresses this by replacing the human hand with a photorealistic robot overlay and yielding paired “current frame → future end-effector pose” labels. Training a closed-loop policy on that synthetic data produces a zero-shot robot controller with no in-lab robot demos, no goal images, and no fine-tuning on real trajectories.

In mixed reality, the relevant problem is not robot embodiment but the manipulation of self-representation and co-representation during shared immersive interaction (He et al., 11 Sep 2025). The body-swapping platform exchanges the two users’ avatar control streams so that each participant inhabits the other’s avatar. The associated study used the Joint Simon Task to examine how body-swapping influences cognitive and perceptual boundaries between self and other.

A common misconception is that EmbodiSwap refers to a single platform with a single evaluation target. The current literature instead supports a narrower and more technical reading: one usage concerns synthetic robot video generation for imitation learning, while another concerns mixed-reality avatar exchange for joint-action experiments. The shared lexical root reflects a common emphasis on embodiment, not a single unified codebase or benchmark.

2. Synthetic robot overlay pipeline

The robotics EmbodiSwap pipeline proceeds frame by frame through a human egocentric clip and applies five stages: 3D Hand & Camera Pose Extraction, Body Segmentation & Inpainting, Robot Pose Re-targeting, Rendering & Blending, and Label Extraction (Dessalene et al., 4 Oct 2025). In the first stage, HaWoR is used to recover a MANO-parameterized 3D skeleton of the actor’s hand and the camera extrinsics, while UniDepthV2 estimates metric depth for each RGB frame. In the second stage, a low-res human silhouette from a trained segmenter is used to prompt SAM2 for a high-res binary body mask, and OmniEraser inpaints away the person and any hand-induced scene artifacts.

Robot pose re-targeting is defined directly from MANO joints $\{kp_i\}_{i=0}^{20}$. The paper computes a 6-DoF “gripper” pose as

$$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$

with $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$ or its negation for left hand. The homogeneous transform is then assembled as

$$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$

Rendering and blending are performed in PyBullet. The robot mesh is rendered at pose $T_g$ to produce an RGB–D robot image, and per-pixel composition compares the inpainted scene depth with the robot depth, selecting the nearer pixel so that the robot is occluded correctly by real objects. For supervision, each frame $t$ is assigned a look-ahead $\tau$, shorter for “open/close” and longer for “pour,” and the relative transform is computed as

$\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$

This $\Delta T_t$, comprising rotation and translation, is stored as the label for the synthetic frame.

The same pipeline is accompanied by temporal re-annotation using a Therblig-based ontology so that every snippet begins the instant a tool is grasped and ends once the postcondition is met. The dominant hand is also recorded. This structure is significant because it converts broad, weakly structured human video into action-aligned robot supervision with explicit future-pose targets.

3. Closed-loop policy learning and repurposed V-JEPA

After EmbodiSwap produces a dataset of $\{I_t^*, \Delta T_t\}$, a closed-loop vision–policy network is trained over the synthetic robot videos (Dessalene et al., 4 Oct 2025). At time $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$0, the policy takes as input the most recent rendered crop $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$1, an optional proprioceptive vector $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$2, and an optional action-location token $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$3, and predicts a one-step relative pose $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$4. That prediction is executed on the real robot, the next observation $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$5 is acquired, and the process repeats.

Supervision uses an $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$6 loss on translation and rotation: $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$7 Rotation is parameterized, for example, via a 6D continuous representation or quaternion log map.

The visual backbone is V-JEPA, repurposed from video understanding to imitation learning. V-JEPA has three modules: an X-encoder that masks part of the video and embeds visible tokens, a Predictor that takes visible embeddings and mask tokens and predicts missing features, and a Y-encoder that computes target features on the full video. The EmbodiSwap system freezes the X-encoder, typically a ViT-L or ViT-H pretrained for $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$8 human clips, and fine-tunes the Predictor together with a lightweight attention probe $$\mathbf{G}_c = \tfrac{1}{5}\sum_{i\in\{1,5,9,13,17\}} kp_i,\quad \mathbf{G}_z = (kp_5 - kp_0)\times (kp_{17}-kp_0),\quad \mathbf{G}_x = \bigl(\tfrac{1}{4}\sum_{i\in\{5,9,13,17\}}kp_i - kp_1\bigr),$$9. The probe consists of two cross-attention layers to fuse predicted features, proprioception $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$0, and location token $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$1, followed by two self-attention layers and a final MLP to output $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$2.

Three modifications are central. First, the input is often a single composited frame $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$3 plus positional mask tokens for future frames, rather than a full masked video. Second, proprioception and optional location tokens are appended. Third, the original MSE feature loss is replaced by the pose $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$4 loss. A plausible implication is that the system treats synthetic overlay generation and visual pretraining as complementary: EmbodiSwap resolves the embodiment mismatch in pixel space, while V-JEPA supplies a representation regime well matched to human-like motion statistics.

4. Dataset construction, training regimes, and quantitative results

The released synthetic corpus is generated over EPIC-Kitchens, HOI4D, and Ego4D, using five actions: Open, Close, Pour, Cut, and Place (Dessalene et al., 4 Oct 2025). EPIC-Kitchens contributes all five actions, HOI4D contributes Open, Close, and Place, and Ego4D contributes Pour and Cut. For each clip, annotators mark exact Therblig sub-action boundaries and the dominant hand, the five-stage pipeline is run on every frame, and image crops are stored at $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$5 with random cropping at scale $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$6–$\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$7 and horizontal flips for data augmentation. The release includes raw synthetic videos at approximately $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$8 clips, boundary annotations, and derived $\mathbf{G}_y = \mathbf{G}_z\times \mathbf{G}_x$9 pairs.

Two training regimes are reported. For vision-backbone ablations over 13 pretrained models, the entire backbone is frozen and only the two-layer attention probe is trained, with no proprioception or location token, for 40 epochs, batch $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$0, learning rate $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$1 with cosine decay to $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$2, weight decay $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$3, on a single RTX A5000 GPU. For the final real-world policy, only the X-encoder is frozen while the Predictor and probe are fine-tuned; proprioceptive joint angles are injected for open, close, pour, and cut, and omitted for place because that was empirically better.

The baselines are $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$4 (few-shot), fine-tuned on 30 in-lab robot demos per action with default $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$5 settings, horizon 3, and $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$6 steps, and $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$7 (ES), in which the $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$8 vision backbone is fine-tuned on EmbodiSwap data with zero real demos. In the vision-backbone comparison, V-JEPA (ViT-L) achieves the lowest average translation error, approximately $$T_g = [\,\hat{\mathbf{G}_x}\;\hat{\mathbf{G}_y}\;\hat{\mathbf{G}_z}\;\mathbf{G}_c\,].$$9, and rotation error, approximately $T_g$0 unitless, over the five actions, outperforming ResNet-50, MAE, DINOv2, R3M, Octo, RoboFlamingo, and $T_g$1. The paper notes that large-scale robot-only pretrained models perform worse on these synthetic egocentric clips, which it interprets as evidence for the value of predicting human-like hand trajectories.

In real-world zero-shot evaluation, $T_g$2 trained on 30 real demos achieves 15/85 overall success ($T_g$3), $T_g$4 (ES) achieves 24/85 ($T_g$5), and the V-JEPA + EmbodiSwap model achieves 70/85 ($T_g$6). The per-action breakdown is Open 19/20, Close 17/20, Pour 10/15, Place 10/15, and Cut 14/15. Failure modes include robot kinematic singularities, self-occlusions masking critical contact cues, and visually plausible but unsuccessful motions such as slightly missing the receptacle when pouring.

5. Mixed-reality EmbodiSwap platform and body-swap mechanics

A separate mixed-reality line of work provides a detailed implementation and evaluation of body-swapping relevant to an “EmbodiSwap” platform (He et al., 11 Sep 2025). The hardware stack comprises a Meta Quest Pro head-mounted display in color passthrough mode at 90 Hz and $T_g$7 px/eye, two Meta Quest controllers per user, and a Wi‑Fi 6 router for local networking with round-trip latency below $T_g$8. The software pipeline uses Unity 2022.3 LTS, Meta Mixed Reality Utility Kit for scene anchoring and passthrough integration, Meta Movement SDK for head and hand pose streaming at 60 Hz, Unity Netcode for GameObjects in host-client mode, an inverse kinematics solver to compute a full-body skeleton from head and controller poses, and ReadyPlayer.Me avatar import with automatic rig scaling.

Environment preparation includes scanning the physical environment and modeling key furniture as MRUK occluders, anchoring a virtual mirror of $T_g$9 at $t$0 in front of the user, and placing floor markers to standardize participant positions. Body-swap logic is defined per frame: UserA’s head and hand transforms are streamed to UserB’s avatar rig and vice versa. The avatar joint global transform is computed as

$t$1

On the swap command, implemented as touching index fingers, the two avatar control streams are exchanged in the networked pipeline.

The experimental protocol uses four within-subject blocks counterbalanced across participants: Standard Simon (alone, two-button), Go/No-Go (alone, single-button), Joint Simon Task (Self-Avatar), and Joint Simon Task (Swapped-Avatar). Each trial consists of a fixation cross for 500 ms, a target stimulus for 1000 ms, a response window of up to 1000 ms from stimulus onset using the grip button, and a blank inter-trial interval of 500 ms. Stimuli span geometric solids, colored spheres, digits, and arrows, with 100 trials per block. In the joint blocks, participants and experimenter stand on predefined markers, each responds only to one shape using the right grip button, and the swapped condition follows a 1.5-minute familiarization during which the swap occurs mid-phase.

6. Behavioral effects: closeness, ownership, and joint action

The mixed-reality study reports standard Simon and Joint Simon effects together with subjective embodiment and interpersonal closeness measures (He et al., 11 Sep 2025). In the single-participant tasks, Standard Simon yields a compatible mean RT of 1035.2 ms, an incompatible mean RT of 1087.9 ms, a difference of 52.7 ms, $t$2, $t$3, $t$4; Go/No-Go yields 865.4 ms versus 859.2 ms, a difference of $t$5 ms, $t$6, $t$7, $t$8. For the joint blocks, a linear mixed-effects model with fixed effect Condition and random intercept Participant estimates the Self-Avatar Joint Simon Effect at $t$9, $\tau$0, $\tau$1, $\tau$2, and the condition effect Swapped – Self at $\tau$3, $\tau$4, $\tau$5, $\tau$6.

Interpersonal closeness is measured using the 6-point Inclusion of Other in the Self scale. Mean IOS is 2.76 in the Self-Avatar condition and 3.34 in the Swapped condition, with mixed-model estimate $\tau$7, $\tau$8, $\tau$9, $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$0. Subjective embodiment measures indicate Body Ownership $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$1 with $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$2, Self-Location $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$3 with $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$4, Co-presence $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$5 with $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$6, Behavioral Interdependence $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$7 with $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$8, Agency $\Delta T_t = T_g(t+\tau)\,T_g(t)^{-1}.$9 with $\Delta T_t$0, and Attentional Allocation $\Delta T_t$1 with $\Delta T_t$2, all for Swapped minus Self.

The paper interprets the weakened Joint Simon Effect under swapping through a Referential Coding Account: spatial conflict arises when one distinguishes self-generated from other-generated actions, and swapping dissociates visual appearance from motor control, weakening body ownership so that the partner’s keypress ceases to function as an “external” spatial cue. The ownership–JSE analysis reports

$\Delta T_t$3

with an indirect effect of approximately $\Delta T_t$4 in mediation, $\Delta T_t$5. This suggests that increased closeness and decreased co-representational interference can co-occur under body-swap conditions rather than forming a single monotonic behavioral dimension.

7. Reproducibility, limitations, and interpretive boundaries

The robotics EmbodiSwap release includes code for generating the synthetic robot overlays from human videos and an arbitrary robot URDF, the synthesized robot dataset over EPIC-Kitchens, HOI4D, and Ego4D, and model checkpoints with inference code (Dessalene et al., 4 Oct 2025). The mixed-reality implementation specifies a concrete hardware and software stack, scene-occlusion setup, swap trigger, and counterbalanced protocol, which makes the body-swap paradigm directly reproducible at the level of system architecture and timing (He et al., 11 Sep 2025).

The main limitations are domain-specific. In the robot setting, reported failure modes are kinematic singularities, self-occlusions, and visually plausible but unsuccessful motions. In the mixed-reality setting, body ownership loss can undermine action coordination, trait variability matters, and network latency spikes above $\Delta T_t$6 can break the illusion. The MR study’s design implications therefore recommend short full-body swaps to boost perceived closeness before collaborative tasks, re-establishing original embodiment when millisecond-level coordination is critical, adaptive embodiment controls based on visuo-proprioceptive mismatch, and the use of mirrors and behavioral interdependence cues.

A broader interpretive boundary follows from the two literatures taken together. Embodiment manipulation does not uniformly improve all downstream metrics. In the robotics case, bridging the human–robot embodiment gap with photorealistic overlays and a V-JEPA backbone yields strong zero-shot manipulation performance. In the mixed-reality case, body-swapping increases closeness while weakening the Joint Simon Effect. A plausible implication is that EmbodiSwap-style systems should be evaluated against the specific function of embodiment transfer being optimized—policy generalization, subjective closeness, ownership, or joint coordination—rather than against an undifferentiated notion of “better embodiment.”