EMMA: Manipulation Media Adaptation

• EMMA is a framework that adapts manipulation across multiple media (vision, language, action) to enhance embodied task performance.
• It employs techniques like multimodal fusion, grounded chain-of-thought reasoning, and spatial look-ahead to translate sensory inputs into control actions.
• Empirical results demonstrate improved generalization, spatial reasoning, and cross-embodiment transfer across diverse robotic tasks.

Searching arXiv for the specified EMMA-related papers and recent context. arxiv_search(query="(Sun et al., 2024) OR (Dong et al., 26 Sep 2025) OR (Zhu et al., 4 Sep 2025) OR (Pantazopoulos et al., 2023) OR (Yang et al., 2023) OR (Wu et al., 8 Jun 2026) OR (Song et al., 12 May 2026) OR (Park et al., 29 Apr 2026)", max_results=10) Embodied Manipulation Media Adaptation (EMMA) denotes a family of embodied-AI formulations that adapt manipulation competence across representational media rather than treating perception, reasoning, and control as isolated modules. In the cited literature, this adaptation is instantiated across vision, language, action tokens, action-observation trajectories, text-world supervision, egocentric human demonstrations, generated robot videos, and cross-embodiment video renderings. The term is therefore not a single canonical model name; instead, it appears as an explicit framework in some papers and as a conceptually aligned design pattern in others. Emma-X formulates EMMA as adaptation across vision media, language media, and action media for robot manipulation (Sun et al., 2024). Other works use closely related EMMA formulations for generative visual transfer in real-world robot manipulation (Dong et al., 26 Sep 2025), egocentric human-to-robot mobile manipulation (Zhu et al., 4 Sep 2025), multitask action-as-text embodied task completion (Pantazopoulos et al., 2023), and cross-modality imitation from a parallel text world to a visual-world VLM agent (Yang et al., 2023).

1. Terminological scope and conceptual boundaries

Across the arXiv literature, the acronym “EMMA” is used for multiple systems with different expansions, while the underlying theme remains adaptation of embodied manipulation or task completion across media, modalities, or embodiments. This terminological plurality is itself part of the topic: some papers define EMMA directly as a manipulation framework, whereas others define a distinct acronym but are described in the provided material as implementing “EMMA-style” capabilities (Sun et al., 2024, Dong et al., 26 Sep 2025, Zhu et al., 4 Sep 2025, Pantazopoulos et al., 2023, Yang et al., 2023, Wu et al., 8 Jun 2026, Song et al., 12 May 2026, Park et al., 29 Apr 2026).

Paper	Expansion or EMMA role	Primary adaptation target
"Emma-X" (Sun et al., 2024)	Embodied Multimodal Action Model	Vision, language, and action media
"EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfer" (Dong et al., 26 Sep 2025)	Embodied Manipulation Media Adaptation	Generated visual domains for VLA post-training
"EMMA: Scaling Mobile Manipulation via Egocentric Human Data" (Zhu et al., 4 Sep 2025)	Egocentric Mobile MAnipulation	Human-to-robot cross-embodiment co-training
"Multitask Multimodal Prompted Training for Interactive Embodied Task Completion" (Pantazopoulos et al., 2023)	Embodied MultiModal Agent	Images, dialogue, trajectories, and actions as text
"Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld" (Yang et al., 2023)	Embodied Multi-Modal Agent	Text-world expertise distilled into visual-world control
"iMaC" (Wu et al., 8 Jun 2026)	EMMA operationalization	Image-as-action control for world models
"OmniHumanoid" (Song et al., 12 May 2026)	EMMA-style adaptation	Motion transfer across embodiments
"Alter-Art" (Park et al., 29 Apr 2026)	EMMA blueprint	Avatar-based artistic manipulation across media

A common misconception is to treat EMMA as only a visual data augmentation technique. The cited literature suggests a broader taxonomy. In Emma-X, “media adaptation” explicitly spans vision media, language media, and action media (Sun et al., 2024). In the generative-transfer EMMA, the emphasis is multi-view-consistent video generation and hard-sample-aware policy training (Dong et al., 26 Sep 2025). In egocentric mobile manipulation, the adaptation problem is cross-embodiment alignment between human egocentric demonstrations and robot actuation (Zhu et al., 4 Sep 2025). In action-as-text and text-world-distilled agents, the adaptation problem is between symbolic or linguistic action representations and embodied execution (Pantazopoulos et al., 2023, Yang et al., 2023).

This suggests that EMMA is best understood as a unifying research motif: embodied manipulation is improved when the supervisory or control signal is translated into a medium that preserves task-relevant spatial, temporal, or semantic structure.

2. Core technical principles of media adaptation

The most explicit manipulation-centered definition appears in Emma-X. There, EMMA adapts across three “media” of manipulation: vision media, consisting of RGB images and detected 2D gripper position; language media, consisting of natural-language task instruction and grounded chain-of-thought reasoning; and action media, consisting of continuous end-effector velocity control and gripper state, tokenized and generated as action tokens (Sun et al., 2024). Emma-X maps instructions and observations to embodiment-specific policies through multimodal fusion, grounded CoT, look-ahead spatial guidance, and autoregressive action generation over a 7D control vector encoding Cartesian velocities, orientation velocities, and gripper open/close (Sun et al., 2024).

A second principle is that adaptation should preserve geometric or kinematic validity when transferring across media. DreamTransfer, the generative engine in the 2025 EMMA framework, edits foreground, background, and lighting while preserving 3D structure and cross-view coherence through concatenated multi-view inputs and depth-conditioned ControlNet guidance (Dong et al., 26 Sep 2025). iMaC pushes this principle further by replacing low-dimensional action vectors with motion images and contact images, so that action content is expressed in the same medium as the world model’s observations and predictions (Wu et al., 8 Jun 2026). OmniHumanoid applies an analogous factorization to cross-embodiment video generation: transferable motion dynamics are separated from embodiment-specific appearance and morphology, and new embodiments are adapted via lightweight LoRA modules trained with unpaired videos (Song et al., 12 May 2026).

A third principle is that action representations can be unified with language-like sequence modeling. The multitask EMMA for interactive task completion treats action prediction as multimodal text generation and learns a “language of actions” that unifies navigation, manipulation, and dialog across modalities (Pantazopoulos et al., 2023). Its output grammar includes structured tokens such as <act>, <search>, <one match>, <multiple matches>, <no match>, <frame_token_i>, <visual_token_j>, and <stop> (Pantazopoulos et al., 2023). The parallel-TextWorld EMMA similarly represents high-level actions textually, but supervision comes from an LLM actor and critic operating in a text world aligned step-by-step with the visual environment (Yang et al., 2023).

A fourth principle is cross-embodiment adaptation. Egocentric Mobile MAnipulation co-trains human full-body motion data with static robot data, retargeting human navigation trajectories to differential-drive commands and aligning manipulation actions in an egocentric camera frame (Zhu et al., 4 Sep 2025). Alter-Art provides a teleoperation-centric instantiation: an artist inhabits a robot avatar through first-person stereo vision, haptic gloves, expressive face control, and variable stiffness actuation, with media-specific adaptations for dance, theater, and painting (Park et al., 29 Apr 2026).

3. Architectural families and representative formulations

Several architectural families recur in EMMA-related systems. Emma-X is fine-tuned from OpenVLA with a Prismatic-7B VLM backbone and uses a multimodal transformer over vision, language, and previously generated tokens. Its reasoning head generates CoT tokens, its spatial guidance head predicts the next-segment 2D gripper waypoint and a natural-language 3D movement plan, and its action head outputs tokenized 7D end-effector velocities plus gripper open/close (Sun et al., 2024). The associated policy is conditioned on auxiliary guidance,

$p_\theta(a_t \mid o_t, u, h_t),$

where $o_t$ is the image observation, $u$ is the instruction, and $h_t$ denotes grounded CoT and spatial look-ahead (Sun et al., 2024).

The multitask prompted EMMA of 2023 uses a BART-base style encoder-decoder with a 10K BPE vocabulary, a single-stream multimodal encoder, object-centric scene encoding, temporal embeddings, and sentinel tokens for frame and region references (Pantazopoulos et al., 2023). Its sequence model is defined as

$$p(a_{1:T} \mid X) = \prod_{t=1}^{T} p(a_t \mid a_{<t}, X),$$

with $X$ containing image frames, dialogue turns, and prior trajectory context (Pantazopoulos et al., 2023). The model unifies Contextual Routing, Action Execution, and Visual Grounding through prompts, and uses constrained decoding to enforce action grammar and affordance validity (Pantazopoulos et al., 2023).

The parallel-TextWorld EMMA uses a modular VLM consisting of a ViT-L vision encoder, a Q-Former with 32 query tokens, a linear projection into Vicuna-7B-v1.1 text embedding space, and a frozen language backbone that generates high-level textual actions (Yang et al., 2023). Its distinctive contribution is DAgger-DPO: on-policy trajectories from the visual student are translated into aligned text-world states, an LLM actor produces improved actions, an LLM critic produces retrospective feedback, and the student is updated with a DPO-style preference objective (Yang et al., 2023).

DreamTransfer-based EMMA uses a dual-branch diffusion Transformer with a geometry-guided ControlNet branch. Synchronized multi-view video frames and depth maps are concatenated width-wise and encoded by a video VAE, while a T5 encoder provides text conditioning for appearance edits (Dong et al., 26 Sep 2025). Generated videos are then paired with the original demonstration actions and used for VLA post-training, after which AdaMix reweights difficult samples (Dong et al., 26 Sep 2025).

iMaC uses a WAN2.2 image-to-video DiT with “addition-as-conditioning” in latent space. Motion images $C^m$ and contact-image streams $C^{s\rightarrow g}$ and $C^{r\rightarrow s}$ are VAE-encoded and added to future latent tokens, yielding

$$h_\tau = \left[ P_v(z^r); P_v(x_\tau) + P_m(E(C^m)) + P_{s\rightarrow g}(E(C^{s\rightarrow g})) + P_{r\rightarrow s}(E(C^{r\rightarrow s})) \right],$$

and trained with a flow-matching objective (Wu et al., 8 Jun 2026).

OmniHumanoid uses a DiT video backbone with branch-isolated attention. Tokens are split into text, conditioning, and denoising streams, and embodiment-specific LoRA is attached only to the denoising branch:

$o_t$0

with an asymmetric attention mask enforcing $o_t$1 and $o_t$2 (Song et al., 12 May 2026). This isolates transferable motion conditioning from embodiment-specific appearance modulation.

Alter-Art differs from the policy-learning papers because it is primarily an avatar-based teleoperation stack, but the control formulations listed in the provided material are canonical EMMA components: human-to-robot mapping $o_t$3, unicycle base dynamics, joint and Cartesian impedance control, admittance control for contact-rich manipulation, and passivity mechanisms such as passivity observers and scattering transformation for delay-robust teleoperation (Park et al., 29 Apr 2026).

4. Data regimes, supervision signals, and optimization

A defining characteristic of EMMA-related work is that adaptation is driven by structured supervisory signals rather than by raw task labels alone. Emma-X constructs a hierarchical embodiment dataset from BridgeData-v2 with approximately 60,000 trajectories, organized as tasks, subtasks, segments, and states, and augmented with Gemini-generated grounded CoT reasoning and spatial guidance (Sun et al., 2024). Segmentation is central: HDBSCAN clustering over a custom spatiotemporal/orientation distance is combined with gripper-state boundaries, which reduces hallucinations in grounding subtask reasoning generation and defines the next-segment waypoint targets (Sun et al., 2024). The distance is

$o_t$4

with a breakpoint when $o_t$5 (Sun et al., 2024).

Emma-X jointly optimizes actions, reasoning, and waypoint prediction:

$o_t$6

This formulation makes grounded CoT and spatial look-ahead explicit auxiliary guidance rather than post hoc explanations (Sun et al., 2024).

The prompted multitask EMMA uses a multitask objective over seven pretraining tasks and three embodied fine-tuning tasks:

$o_t$7

where pretraining includes masked language modeling, image-text matching, VQA, dense captioning, visual grounding, and relationship prediction, while embodied fine-tuning includes Contextual Routing, Action Execution, and Visual Grounding (Pantazopoulos et al., 2023). The data regime combines 10.2M image-text pairs with augmented trajectory data, including +180k training and +88k validation visual augmentations and +38k training and +15k validation CDF augmentations (Pantazopoulos et al., 2023).

The TextWorld-distilled EMMA begins from a behavior-cloning reference policy trained on 15,247 episodes totaling 178,585 image-text pairs, then performs DAgger-DPO over 12 trials with episode length 30 and per-trial 5 epochs (Yang et al., 2023). The optimization target is the DPO preference loss regularized by a reference policy:

$o_t$8

The positive action is the LLM-refined action $o_t$9; the negative action is the student action $u$0 (Yang et al., 2023).

DreamTransfer-based EMMA relies on multi-view RGB videos with aligned timestamps, estimated depth maps, and structured text prompts for foreground, background, and lighting (Dong et al., 26 Sep 2025). The generative module is optimized with the standard diffusion denoising objective, and generated clips are filtered by multi-view consistency, depth consistency, and text-video alignment before policy training (Dong et al., 26 Sep 2025). AdaMix then defines a sample score

$u$1

followed by

$u$2

so that harder samples receive higher batch weight (Dong et al., 26 Sep 2025).

Egocentric Mobile MAnipulation uses two complementary datasets: human data captured with Meta Project Aria smart glasses, including egocentric RGB and bimanual 6-DoF hand poses plus 6-DoF head pose, and robot data consisting of static manipulation teleoperation with egocentric RGB, wrist RGB, joint positions, and end-effector poses (Zhu et al., 4 Sep 2025). Human navigation is retargeted to differential-drive commands by constrained optimization over linear and angular velocities, while manipulation actions are transformed into the egocentric camera frame and normalized per source (Zhu et al., 4 Sep 2025). The policy is trained by supervised behavior cloning with Smooth-$u$3 and MSE losses, and an unsupervised phase-identification mechanism supplies binary navigation-versus-manipulation phase labels (Zhu et al., 4 Sep 2025).

iMaC’s data regime is video-centric rather than policy-centric. It uses paired multi-view RGB videos and action trajectories from eight long-horizon real-robot manipulation tasks, predicts RGB and depth jointly, and closes the loop by feeding generated RGB-D frames into the next chunk’s control construction (Wu et al., 8 Jun 2026). OmniHumanoid, by contrast, explicitly separates paired motion-aligned videos for shared motion transfer from unpaired videos for embodiment-specific adaptation (Song et al., 12 May 2026).

5. Empirical performance and application domains

The empirical literature covers fixed-base manipulation, dialog-guided household task completion, ALFWorld visual control, mobile manipulation, cross-embodiment video generation, image-conditioned world models, and artistic telepresence. The results are not directly commensurate because the tasks, action spaces, and metrics differ, but they collectively show that media adaptation can improve generalization, referential disambiguation, long-horizon reasoning, or embodiment transfer (Sun et al., 2024, Pantazopoulos et al., 2023, Yang et al., 2023, Dong et al., 26 Sep 2025, Zhu et al., 4 Sep 2025, Song et al., 12 May 2026, Wu et al., 8 Jun 2026, Park et al., 29 Apr 2026).

System	Benchmark or domain	Reported result
Emma-X (Sun et al., 2024)	12 real-world WidowX-250 tasks	h_Succ 71.66%, Succ 57.50%
Prompted EMMA (Pantazopoulos et al., 2023)	Alexa Arena DTC	MSR 36.81, NRA 8.69
TextWorld EMMA (Yang et al., 2023)	ALFWorld visual environment	average success 0.82
DreamTransfer EMMA (Dong et al., 26 Sep 2025)	zero-shot visual domains	over a 200% relative performance gain
DreamTransfer EMMA + AdaMix (Dong et al., 26 Sep 2025)	zero-shot visual domains	further improves by 13%
Egocentric EMMA (Zhu et al., 4 Sep 2025)	Handover Wine	82% vs 52% for Mobile ALOHA
OmniHumanoid (Song et al., 12 May 2026)	synthetic unseen embodiment	PSNR 25.47, SSIM 0.9039, MSE 0.0033
iMaC (Wu et al., 8 Jun 2026)	future-video prediction	FVD 489.51 ± 92.65

Emma-X reports especially strong gains in spatial reasoning tasks. On aggregate over 12 tasks, OpenVLA achieves h_Succ 45.41% and Succ 33.33%, ECoT achieves h_Succ 25.00% and Succ 11.67%, and Emma-X achieves h_Succ 71.66% and Succ 57.50% (Sun et al., 2024). Category-level average h_Succ is 71% for Spatial Relation, 87% for OOD Object, 83% for In-Domain, and 50% for OOD Instruction, each above OpenVLA and ECoT (Sun et al., 2024). Representative per-task success values include “Close microwave” at 100%, “Wipe the stove with towel” at 90%, and “Put the blue cube on the right plate” at 90% (Sun et al., 2024).

In Alexa Arena dialog-guided task completion, the unified prompted EMMA achieves a state-of-the-art Mission Success Rate of 36.81 and Number of Robot Actions of 8.69 (Pantazopoulos et al., 2023). Clarifications increase success by up to 3.55% absolute, and description clarifications give the largest object-localization boost at +5.14 for the unified model (Pantazopoulos et al., 2023). In ALFWorld’s visual environment, the TextWorld-distilled EMMA reports average success 0.82 on templated instructions and 0.68 on free-form human instructions; by comparison, MiniGPT-4, BLIP-2, LLaMA-Adapter, and InstructBLIP are substantially lower on the visual benchmark (Yang et al., 2023).

The 2025 DreamTransfer EMMA reports that DreamTransfer attains average pixel matching count 3270, compared with 2298 for RoboTransfer and 2097 for Cosmos-Transfer1; average Sq.Rel. 0.54, compared with 0.71 for Cosmos-Transfer1 and 1.30 for RoboTransfer; and average CLIPSim 24.68, compared with 24.61 and 23.94 (Dong et al., 26 Sep 2025). In downstream manipulation, generated-data training yields over a 200% relative performance gain in zero-shot visual domains compared to training on real data alone, and AdaMix adds approximately 13% improvement over fixed 50% mixing (Dong et al., 26 Sep 2025).

Egocentric Mobile MAnipulation reports 82% success on Handover Wine versus 52% for Mobile ALOHA when replacing one hour of mobile teleoperation with one hour of human egocentric mobile manipulation, while Table Service matches Mobile ALOHA and Grocery Shopping significantly outperforms it across subtasks with $u$4 over 50 trials (Zhu et al., 4 Sep 2025). With one hour of static robot manipulation data fixed, increasing human data from 15 to 60 minutes improves success from 0.36 to 0.82 (Zhu et al., 4 Sep 2025).

OmniHumanoid reports strong performance on unseen embodiment adaptation. On the synthetic held-out embodiment benchmark, it achieves PSNR 25.47, SSIM 0.9039, MSE 0.0033, Motion 9.06, Embodiment 8.43, BG 9.94, and Overall 7.92; on the real-world benchmark, it reports Motion 8.47, Embodiment 8.56, BG 9.95, and Overall 8.39 (Song et al., 12 May 2026). iMaC reports future-video prediction metrics of MSE 0.028 ± 0.010, FID 36.96 ± 9.16, PSNR 16.39 ± 1.41, SSIM 0.735 ± 0.037, and FVD 489.51 ± 92.65, outperforming Ctrl-World and ABot-PhysWorld on those metrics (Wu et al., 8 Jun 2026). Across six tasks, iMaC’s world-model scores correlate with real-world success from 0.833 to 0.956 on most tasks, with two lower-correlation cases at 0.678 and 0.428 attributed to missing-observation failure modes (Wu et al., 8 Jun 2026).

Alter-Art is evaluated qualitatively rather than through standardized robotic success metrics. It reports three qualitative case studies, eight paintings, and findings that artists rapidly developed a strong sense of agency and self-location within the robot body, while the robot’s physical constraints shaped creativity differently across dance, theater, and painting (Park et al., 29 Apr 2026).

6. Limitations, failure modes, and open research questions

The literature identifies several recurring failure modes. In Emma-X, latency increases because inference uses approximately 10× more tokens than OpenVLA, SAM-based gripper detection may degrade under occlusion or out-of-frame conditions, and misclassification in grounded CoT can cascade into waypoint and action errors (Sun et al., 2024). Ablations also show that removing grounded CoT, waypoint prediction, movement planning, or HDBSCAN-based segmentation causes large drops, especially in spatial relation tasks, which indicates strong dependence on auxiliary reasoning and segmentation quality (Sun et al., 2024).

In the prompted EMMA for interactive task completion, failures arise from incorrect Contextual Routing decisions, temporal binding errors in Action Execution, commonsense gaps, and persistent referential ambiguity in color-and-deliver missions, which remain at 0% success (Pantazopoulos et al., 2023). In the TextWorld-distilled EMMA, performance still depends on accurate visual-to-text mapping, sufficient LLM context for reflection memory, and the fidelity of the parallel worlds; severe occlusion, high clutter, or mismatched PDDL predicates can degrade preference quality (Yang et al., 2023).

Generative EMMA introduces different constraints. DreamTransfer relies on high-quality temporal depth consistency; poor depth estimates, heavy motion blur, or extreme occlusions can degrade geometric fidelity (Dong et al., 26 Sep 2025). Very high generated-data mixing ratios can slightly reduce success in fast precision tasks such as Throw Bottle (Dong et al., 26 Sep 2025). iMaC similarly depends on accurate depth and sufficient visual coverage; missing-observation failures occur when task-critical height relations or clearances are not visible in the available views (Wu et al., 8 Jun 2026). OmniHumanoid acknowledges quality gaps between its few-step streaming student and the full teacher, as well as challenges from large morphology disparities, severe self-occlusions, and complex contact-rich manipulation (Song et al., 12 May 2026).

Cross-embodiment systems add embodiment-specific failure modes. Egocentric Mobile MAnipulation can suffer from miscalibrated end-effector orientation, phase-prediction errors, and residual domain gaps between human and robot action distributions (Zhu et al., 4 Sep 2025). Alter-Art emphasizes low-latency immersive control but does not report numerical latency, stiffness, damping, payload, or workspace values; a plausible implication is that systematic quantitative evaluation of embodiment, teleoperation stability, and artistic task performance remains underdeveloped in this line of work (Park et al., 29 Apr 2026).

A broader misconception is that EMMA eliminates embodiment specificity. The cited works show the opposite. Emma-X still outputs embodiment-specific 7D velocities and depends on controller-frame mapping and calibration (Sun et al., 2024). Egocentric Mobile MAnipulation still requires explicit retargeting to differential-drive constraints and static robot data for embodiment-accurate arm control (Zhu et al., 4 Sep 2025). OmniHumanoid still requires embodiment-specific LoRA modules, and iMaC still requires robot-specific URDF/FK and gripper definitions (Song et al., 12 May 2026, Wu et al., 8 Jun 2026). What EMMA changes is not the existence of embodiment constraints, but the medium through which those constraints are learned, transferred, or rendered.

Taken together, the literature suggests three open directions. First, richer spatial hierarchies and longer-horizon guidance remain important, as already noted in Emma-X’s discussion of $u$5 waypoints (Sun et al., 2024). Second, physically grounded action media—whether motion/contact images, cross-view generated videos, or teleoperation haptics—remain promising because they preserve geometry that low-dimensional tokens often discard (Dong et al., 26 Sep 2025, Wu et al., 8 Jun 2026). Third, scalable embodiment transfer remains unresolved: current systems demonstrate progress through shared action languages, reflective distillation, unpaired video adaptation, egocentric co-training, or avatar telepresence, but none fully remove the need for calibration, auxiliary structure, or embodiment-specific adaptation (Pantazopoulos et al., 2023, Yang et al., 2023, Zhu et al., 4 Sep 2025, Song et al., 12 May 2026, Park et al., 29 Apr 2026).