Egocentric Mobile Manipulation (EMMA) Overview

• EMMA is an embodied robotics paradigm that couples first-person sensing with coordinated mobile and manipulation actions for task-conditioned execution.
• It leverages human demonstration data alongside static robot data to overcome teleoperation challenges and enable robust cross-embodiment transfer.
• EMMA systems integrate active perception, phase-conditioned control, and reactive whole-body coordination to achieve high success rates in tasks like Handover Wine and Laundry.

Searching arXiv for papers on egocentric mobile manipulation, EMMA, and closely related embodied whole-body mobile manipulation. Egocentric Mobile Manipulation (EMMA) denotes a class of embodied robotic systems in which a mobile manipulator performs task-conditioned interaction from its own first-person sensing perspective, typically coupling onboard visual observation, locomotion, whole-body reaching, and manipulation into a single execution loop. In the most literal recent usage, EMMA refers to “Egocentric Mobile MAnipulation,” a framework that trains mobile manipulation policies from egocentric human mobile-manipulation demonstrations co-trained with static robot manipulation data, thereby avoiding mobile robot teleoperation during training (Zhu et al., 4 Sep 2025). More broadly, the term is also useful for describing a technical regime that spans body-centric control, onboard perception, active vision, whole-body mobile reaching, and cross-embodiment transfer from egocentric human data to robots (Haviland et al., 2021, Wang et al., 2024, Xu et al., 3 Mar 2026). In that broader sense, EMMA sits at the intersection of visuomotor policy learning, reactive whole-body control, active perception, and mobile manipulation systems research.

1. Conceptual scope and defining properties

EMMA is defined by the coupling of three ingredients that are often separated in earlier mobile manipulation systems: first-person or robot-centric sensing, mobile whole-body action, and manipulation-oriented task execution. In the EMMA framework proper, the scalability bottleneck is identified as the expense of collecting mobile robot teleoperation data, and the proposed solution is to co-train egocentric human mobile-manipulation data with static robot manipulation data (Zhu et al., 4 Sep 2025). In adjacent systems work, the same regime appears under different names: reactive whole-body mobile manipulation that treats the base and arm as one kinematic structure (Haviland et al., 2021), embodied holistic control that coordinates base, arm, and wrist camera under the “Distant Mobility, Close Grasping” principle (Wang et al., 2024), and whole-body mobile manipulation learned from robot-free human demonstrations with explicit head-hand coordination (Xu et al., 3 Mar 2026).

A common misconception is that egocentric mobile manipulation is only a perception problem. The cited work suggests a broader interpretation. In EMMA, the egocentric camera stream is not merely an input modality; it anchors navigation retargeting, phase-conditioned action generation, and cross-embodiment alignment between human and robot observations (Zhu et al., 4 Sep 2025). In reactive whole-body control systems, the same body-centricity appears in a non-learning form: desired end-effector twists and pose errors are expressed in the robot base frame, and eye-in-hand perception updates goals online during execution (Haviland et al., 2021). This suggests that “egocentric” in EMMA is best understood as a systems-level property: sensing, action parameterization, and task progress are all organized around the robot’s embodied viewpoint.

Another misconception is that EMMA requires an end-to-end learned policy from raw video alone. The literature does not support such a narrow definition. Some highly relevant systems are optimization-based rather than learned, such as the holistic reactive controller in “A Holistic Approach to Reactive Mobile Manipulation” (Haviland et al., 2021) and EHC-MM (Wang et al., 2024). Others are hybrid modular systems using learned navigation and shape completion together with engineered grasp planning (Watkins, 2022, Watkins et al., 2021). EMMA therefore names a problem regime and an architectural tendency, not a single learning paradigm.

2. Embodiment, sensing, and representation

The embodied platforms associated with EMMA-like systems are diverse, but several recurring design choices appear. The EMMA framework uses a custom low-cost bimanual mobile manipulator with two 6-DoF ViperX 300 arms mounted on an AgileX TRACER differential-drive base, an Aria egocentric RGB sensor, and two Intel RealSense D405 wrist cameras (Zhu et al., 4 Sep 2025). HoMMI uses a holonomic mobile robot with torso, dual 7-DoF arms, a 2-DoF neck, wrist cameras, and head cameras, with demonstrations collected robot-free using three iPhones and ARKit collaboration (Xu et al., 3 Mar 2026). HoMeR operates on TidyBot++, a holonomic base with a 7-DoF Kinova arm, two base cameras, and a wrist-mounted fisheye camera (Sundaresan et al., 1 Jun 2025). Reactive control papers often use an end-effector-mounted RealSense D435, making the perception explicitly eye-in-hand (Haviland et al., 2021, Wang et al., 2024).

The representation of egocentric input varies with the system objective. EMMA proper uses egocentric RGB, wrist RGB, robot joint positions, and robot end-effector poses on the robot side, while human demonstrations provide egocentric RGB, head pose, and bimanual hand poses from Aria MPS (Zhu et al., 4 Sep 2025). HoMMI explicitly argues that raw egocentric RGB introduces a large human-to-robot embodiment gap and instead constructs an embodiment-agnostic visual representation by combining DINOv3 ViT features with a head pointmap, then masking embodiment-specific points after transforming the pointmap into gripper frames (Xu et al., 3 Mar 2026). MAPLE focuses on stationary dexterous manipulation rather than mobile embodiment, but its core representation-learning result is directly relevant: egocentric human video can be distilled into features predictive of future hand-object contact points and hand pose at contact, producing a manipulation-centric representation more directly useful than generic visual semantics (Gavryushin et al., 8 Apr 2025).

A useful contrast appears between systems that represent egocentric perception in image space and those that convert it into geometry. EMMA’s navigation retargeting begins from projected head trajectories in $SE(2)$, while manipulation signals are transformed into the observation-time egocentric camera frame and normalized separately for human and robot sources (Zhu et al., 4 Sep 2025). HoMMI transforms observations and actions into a left-gripper frame and predicts a 23-dimensional hand-eye action consisting of two 9D gripper poses, a 3D look-at point, and two gripper widths (Xu et al., 3 Mar 2026). EgoPush, though focused on non-prehensile rearrangement, makes the same broader point differently: in dynamic cluttered scenes, relative object-centric latents may be a better control currency than absolute world-state estimates (An et al., 20 Feb 2026). This suggests that EMMA systems frequently trade explicit global world models for body-centric or relation-centric representations aligned with onboard sensing.

3. Action spaces, control abstractions, and whole-body coordination

A central technical issue in EMMA is how to make a mobile manipulator’s high-dimensional body controllable from egocentric observations without forcing the policy to learn all low-level coordination. Several papers converge on an execution-layer abstraction in which the policy or supervisor specifies end-effector or hand-eye intent, while a whole-body controller realizes that intent with coordinated base-arm motion.

In the EMMA framework, mobile behavior is learned from human head trajectories projected onto the ground plane and retargeted to differential-drive commands by solving an optimization over velocities $\mathbf{z}=[(v_1,\omega_1),\ldots,(v_K,\omega_K)]$ subject to differential-drive dynamics and smoothness penalties (Zhu et al., 4 Sep 2025). Manipulation outputs are source-specific heads on a shared transformer trunk, and deployment is phase-conditioned by $\phi_t \in \{0,1\}$, indicating navigation or manipulation (Zhu et al., 4 Sep 2025). A decisive result is that removing phase identification causes complete task failure, with unintended base movement during grasping and unnecessary arm movement during navigation (Zhu et al., 4 Sep 2025). This establishes phase structure as an explicit control abstraction, not merely a training convenience.

Reactive whole-body control papers implement a different abstraction. In “A Holistic Approach to Reactive Mobile Manipulation,” the mobile base and arm are modeled as one kinematic structure, with whole-body differential kinematics

$$\dot{\mathbf{x}}_e = J_e(\mathbf{q}) \begin{pmatrix} \dot{\mathbf{q}}_b\ \dot{\mathbf{q}}_a \end{pmatrix},$$

and a QP that chooses base and arm velocities jointly while respecting limits, manipulability, and a slackened end-effector velocity constraint (Haviland et al., 2021). Desired end-effector motion is generated by base-frame position-based servoing,

$${}^{b}\nu_e^* = \beta \, \psi\!\left(({}^{b}T_e)^{-1} {}^{b}T_{e^*}\right),$$

so the entire control loop is body-centric (Haviland et al., 2021). EHC-MM extends this line by using a reachability-aware embodied weighting $sig(\omega)$ inside a whole-body QP and a Monitor-Position-Based Servoing scheme that interpolates between a monitoring pose and the grasp pose to keep a wrist-mounted camera pointed at the target during approach (Wang et al., 2024).

HoMeR and HoMMI push the same principle into imitation learning. HoMeR learns in a reduced end-effector action space and combines two modes: absolute end-effector keyposes for long-range movement and relative end-effector deltas for fine manipulation, while a kinematics-based whole-body controller solves for coordinated base and arm motion (Sundaresan et al., 1 Jun 2025). HoMMI predicts bimanual hand trajectories and a 3D look-at point rather than body joints, then uses a differential whole-body inverse-kinematics controller with posture, collision, and center-of-mass terms to realize those commands on a mobile robot (Xu et al., 3 Mar 2026). A plausible implication is that EMMA systems benefit from a layered action hierarchy: the learned component predicts task-space intent, while the robot-specific controller absorbs embodiment constraints.

4. Learning from egocentric human data and cross-embodiment transfer

The strongest recent claim associated with EMMA is that mobile manipulation policies can be learned without mobile robot teleoperation by co-training egocentric human mobile-manipulation data with static robot manipulation data (Zhu et al., 4 Sep 2025). Human demonstrations are collected with Meta Project Aria, producing egocentric RGB, 3D head pose, and bimanual hand poses, from which a projected base trajectory $h_{\text{base}}^t=(x^t,y^t,\theta^t)\in SE(2)$ and displacement-based waypoint history are derived (Zhu et al., 4 Sep 2025). Robot data, by contrast, consist only of static manipulation demonstrations with robot proprioception, wrist cameras, and egocentric RGB (Zhu et al., 4 Sep 2025). The shared architecture is a decoder-only transformer with modality-specific stems, a shared trunk, and source-specific action heads (Zhu et al., 4 Sep 2025).

The key embodiment-transfer step is navigation retargeting. Human head trajectories are converted into feasible differential-drive base commands by minimizing position, yaw, and smoothness errors subject to

$$x_{k+1}=x_k+v_k\cos(\theta_k)\Delta t,\quad y_{k+1}=y_k+v_k\sin(\theta_k)\Delta t,\quad \theta_{k+1}=\theta_k+\omega_k\Delta t,$$

with bounds on $v_k$ and $\omega_k$ (Zhu et al., 4 Sep 2025). The paper reports that removing this retargeting reduces Handover Wine success by 30%, and the reported failure mode is losing track of the human recipient because raw human trajectories are kinematically infeasible or poorly aligned for a differential-drive base (Zhu et al., 4 Sep 2025).

A separate research line, HoMMI, attacks the same transfer problem under stronger embodiment differences. It argues that simply adding an egocentric camera to a robot-free human demonstration interface enlarges both the visual and kinematic embodiment gaps, and it responds with a cross-embodiment hand-eye policy design: an embodiment-agnostic visual representation, a relaxed head action representation as a 3D look-at point, and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion (Xu et al., 3 Mar 2026). This is an especially important correction to a common misunderstanding: egocentric human data are not automatically transferable to robot policies. The literature consistently treats transfer as an explicit systems problem, requiring retargeting, action abstraction, frame alignment, or downstream robot-specific supervision (Zhu et al., 4 Sep 2025, Xu et al., 3 Mar 2026).

Adjacent work extends the role of egocentric human data beyond mobile policies. MAPLE shows that large-scale egocentric human video can improve dexterous robot manipulation by providing pseudo-labeled future contact points and hand poses at contact, which are then used as frozen visual features for downstream policies (Gavryushin et al., 8 Apr 2025). EgoMI shows that coordinated head and hand motion in egocentric human demonstrations is itself a major source of distribution shift, and that active head-motion modeling and keyframe memory improve transfer to a semi-humanoid robot (Yu et al., 31 Oct 2025). These results suggest that EMMA may scale along two complementary axes: more human mobile behavior for task structure and navigation, and richer egocentric human interaction priors for grasping, search, and active vision.

5. Active perception, scene grounding, and task specification

EMMA-like systems repeatedly treat perception not as a static upstream estimator but as an active component of manipulation. The EMMA framework includes an explicit phase predictor separating navigation from manipulation, and the authors argue that human mobile data improve state coverage and scene generalization, especially on tasks such as Handover Wine and Grocery Shopping (Zhu et al., 4 Sep 2025). HoMMI shows that disabling active neck control degrades performance from 90% to 75% on Laundry, from 85% to 55% on Delivery, and from 80% to 55% on Tablescape (Xu et al., 3 Mar 2026). EgoMI pushes the same point further: on Shelf Search, the active-head 29D policy achieves 35/40 success points, while the wrist-camera-only 20D policy scores 0/40; fixing the head despite providing the head-camera stream yields only 2/20 successful trials (Yu et al., 31 Oct 2025). These findings directly contradict any assumption that adding an egocentric camera alone is sufficient; viewpoint control itself is task-critical.

A related strand formalizes perception-conditioned trajectory prediction from egocentric streams. “Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation” introduces SP-VTP, where object and target are specified by first-frame spatial prompts and the model predicts future end-effector trajectories $\mathbf{z}=[(v_1,\omega_1),\ldots,(v_K,\omega_K)]$0 in the current end-effector frame (Li et al., 19 May 2026). Spatial prompts substantially outperform non-prompted baselines on cluttered scene splits; for example, overall FDE improves from 0.1739 with no prompt to 0.1147 with the combined BBox + visual prompt setting (Li et al., 19 May 2026). This suggests that in EMMA, manipulation intent may often be more effectively grounded by spatial prompting than by language alone in visually ambiguous scenes.

Scene grounding also appears in classical embodied pipelines. “Learning Mobile Manipulation” and “Mobile Manipulation Leveraging Multiple Views” use egocentric RGB-D, active viewpoint selection, and predicted image goals to support localization-free navigation and shape completion (Watkins, 2022, Watkins et al., 2021). In the latter, uncertain voxels near the occupancy decision boundary $\mathbf{z}=[(v_1,\omega_1),\ldots,(v_K,\omega_K)]$1 are used to compute a next-best-view, which then becomes a predicted panoramic navigation goal (Watkins et al., 2021). EgoPush generalizes this philosophy to long-horizon rearrangement: the privileged teacher is deliberately restricted to visually accessible cues, inducing active perception behaviors that a depth-conditioned student can imitate from egocentric observation (An et al., 20 Feb 2026). A plausible implication is that EMMA systems benefit when task-relevant information is structured so that perception-driven movement is necessary and observable, rather than hidden inside an omniscient planner or teacher.

6. Empirical performance, limits, and research directions

The current EMMA literature reports several concrete performance claims, but they should be interpreted in the context of differing tasks and embodiments. EMMA proper reports 82% success on Handover Wine versus 52% for a Mobile ALOHA-style baseline under the same 100 demonstrations of static robot data and one hour of mobile data, and reports that EMMA significantly outperforms the teleoperation baseline on Grocery Shopping while matching it on Table Service (Zhu et al., 4 Sep 2025). On Handover Wine, increasing human mobile-manipulation data from 15 minutes to 1 hour improves EMMA success from 0.36 to 0.82, compared with 0.26 to 0.52 for added mobile robot teleoperation data in the baseline (Zhu et al., 4 Sep 2025). In an unseen scene, EMMA reaches 54% success using only additional human demonstrations there, while the paper states that the teleoperation-trained baseline fails to complete the initial grasp (Zhu et al., 4 Sep 2025).

Reactive control systems report a different type of empirical advantage. The holistic controller in (Haviland et al., 2021) solves its QP in 4.8 ms on an Intel i5 NUC, about 200 Hz, and on real repetitive pick-and-place completes 100 objects with 113 grasp attempts, reporting that no failures were caused by motion control or actuation (Haviland et al., 2021). EHC-MM reports 95.6% real-world success in sequential multi-object grasping and a headline 52.8% increase in time efficiency, though the exact denominator for that percentage is not clearly specified in the text (Wang et al., 2024). HoMeR achieves 79.17% overall success across 3 simulated and 3 real household tasks using only 20 demonstrations per task, outperforming the next best baseline by 29.17 percentage points (Sundaresan et al., 1 Jun 2025). HoMMI reports 90% on Laundry, 85% on Delivery, and 80% on Tablescape from robot-free human demonstrations (Xu et al., 3 Mar 2026).

Despite this progress, the literature also converges on several limitations. EMMA assumes that tasks decompose into sparse navigation/manipulation phases and that human and robot egocentric viewpoints are aligned closely enough to share a visual trunk (Zhu et al., 4 Sep 2025). Holistic reactive controllers are local, purely reactive, and typically do not solve global obstacle-rich planning or richer semantic scene understanding (Haviland et al., 2021, Wang et al., 2024). HoMMI notes that its short observation history $\mathbf{z}=[(v_1,\omega_1),\ldots,(v_K,\omega_K)]$2 limits recovery and long-horizon memory, and that the system remains vision-only without tactile sensing (Xu et al., 3 Mar 2026). EgoPush explicitly lacks memory and object-state persistence beyond its object-centric latent (An et al., 20 Feb 2026). MAPLE and SPOT are powerful for manipulation priors and task grounding, but they do not include mobile base control (Gavryushin et al., 8 Apr 2025, Li et al., 19 May 2026).

These limitations suggest several research directions. One is tighter integration of active perception and whole-body control, combining hand-eye intent, phase structure, and observability-aware servoing in a single egocentric execution layer (Wang et al., 2024, Xu et al., 3 Mar 2026). Another is richer cross-embodiment transfer, where human egocentric data provide not just navigation trajectories but scene memory, grasp priors, and active search strategies (Zhu et al., 4 Sep 2025, Yu et al., 31 Oct 2025). A third is explicit risk sensitivity: recent work shows that risk-aware whole-body policies conditioned on egocentric depth can improve worst-case performance in unmapped environments while retaining reactive control (Groom et al., 4 Mar 2026). Taken together, the literature suggests that EMMA is evolving from a data-scaling idea into a broader systems paradigm: mobile manipulation grounded in the robot’s own viewpoint, with embodiment-aware representations, whole-body execution, and active control of what the robot sees as well as what it does.