EgoActor: Unified Egocentric Action Model

• EgoActor is a unified and scalable vision-language model that directly grounds high-level instructions into precise, spatially-aware egocentric actions for humanoid robots.
• It leverages a Qwen3-VL backbone with LoRA fine-tuning across all linear layers, using both Structured Language Actions and Natural Language Actions for diverse control commands.
• Extensive evaluations show that EgoActor outperforms navigation baselines in tasks like human-robot interaction, mobile manipulation, and traversability in both real and virtual settings.

to=arxiv_search.search 天天中彩票任选json {"query":"arXiv (Bai et al., 4 Feb 2026) EgoActor Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-LLMs", "max_results": 5} to=arxiv_search.search _日本毛片免费视频观看json {"query":"egocentric action representation actor observer first third person arXiv (Sigurdsson et al., 2018)", "max_results": 5} to=arxiv_search.search 彩神争霸大发json {"query":"EgoAgent joint predictive agent model egocentric worlds arXiv (Chen et al., 9 Feb 2025)", "max_results": 5} to=arxiv_search.search 大发娱乐json {"query":"EgoInteract synthetic egocentric videos generation interaction understanding anticipation arXiv (Leonardi et al., 18 May 2026)", "max_results": 5} to=arxiv_search.search 天天中彩票未json {"query":"EgoAvatar Egocentric View-Driven and Photorealistic Full-body Avatars arXiv (Chen et al., 2024)", "max_results": 5} EgoActor is a unified and scalable vision-LLM introduced together with the task of EgoActing, in which a humanoid robot must directly ground high-level instructions into various, precise, spatially aware actions from egocentric RGB-only observations. The model is designed to predict locomotion primitives, head movements, manipulation commands, and human-robot interactions within a single framework, thereby addressing the gap between abstract task planning and executable embodied behavior in partially observed, dynamically changing environments (Bai et al., 4 Feb 2026).

1. Task formulation and conceptual scope

EgoActor is defined through the task of EgoActing, which formalizes next-action prediction for humanoids as

$$a_t = \arg\max_{a \in \mathcal{A}} P(a \mid I,\; O_{1:t},\; a_{1:t-1},\; \Pi),$$

where $I$ is a natural-language instruction, $O_{1:t}$ is the history of egocentric observations, $a_{1:t-1}$ is the past action history, and $\Pi$ is the set of available low-level whole-body and manipulation policies. In this formulation, EgoActor does not terminate at subgoal generation or abstract skill selection; it predicts the next concrete humanoid action to execute under egocentric partial observability (Bai et al., 4 Feb 2026).

The task is motivated by several properties of humanoid deployment. Humanoids are described as unstable and timing-sensitive; real environments are only partially observed from onboard sensing; practical tasks require coordinated locomotion, active perception, manipulation, and social behavior; and existing systems often separate planning from execution too coarsely. EgoActing is therefore intended to capture a specifically humanoid control problem in which spatial alignment, active looking, posture adjustment, approach distance, and behavior switching all matter.

A defining feature of EgoActor is that it treats “grounding task planning into spatial-aware egocentric actions” as an intermediate layer between instruction following and low-level control. The relevant outputs are not merely “go to the table” or “interact with the person,” but actions such as turning by a specified angle, moving by a specified distance, looking up or down, or emitting a manipulation or interaction command at the correct moment. This suggests a representation of embodied control that is spatially parameterized yet expressed in a language-compatible form.

2. Model architecture and action language

EgoActor uses Qwen3-VL as its base transformer VLM and applies LoRA to finetune all linear layers. Training uses a learning rate of 3e-4 for one epoch on 16 A100 40GB GPUs. The model is evaluated in both 4B and 8B variants, with the paper reporting a trade-off between inference speed and performance across model sizes (Bai et al., 4 Feb 2026).

Temporal context is provided through prompt construction rather than through a separately specified recurrent memory module. Each prompt contains 10 sampled historical observations uniformly drawn from earlier images and 3 recent observation-action pairs as short-term anchors. The recent observations are processed at 480p, while historical observations are processed at 240p. The model is prompted as a system specialized in first-person images of embodied robots and is asked to predict the next action in a constrained output format.

EgoActor’s output space is hybrid. It combines Structured Language Actions (SLAs) for locomotion and active perception with Natural Language Actions (NLAs) for manipulation and human interaction. This gives the system a shared textual interface while preserving explicit spatial parameterization for movement.

Action family	Supported forms	Example
Structured Language Actions	move, turn, look, lateral move, rise up, lower down	Turn left 30.0 degrees
Natural Language Actions	manipulation and interaction commands	Pick up the water bottle
Terminal action	task completion / wait	Stop and no action

The supported skill inventory is broader than navigation alone. Movement skills include moving forward or backward, turning left or right, strafing via the prompt form Left/right sidewalk 0.40 meters, and changing body height through Rise up 0.12 meters or Lower down 0.08 meters. Active perception is represented through Look up/down 10.0 degrees. Human-interaction skills include Say hi, Speak, Ask, and confirm or denial gestures. Manipulation skills include Pick up, Pull, Place ... on ..., Open, Close, Wash, Pour from ... into ..., Turn on, Turn off, Point to, and Drop (Bai et al., 4 Feb 2026).

Execution is delegated to downstream subsystems. SLAs are parsed into velocity or angle commands; speech-related keywords such as Speak and Ask are routed to text-to-speech; predefined interaction keywords such as Say Hi are mapped to preset motions; and other NLAs are passed to pretrained manipulation VLA models. The manipulation subsystem is based on GROOT-N 1.5, which is fully fine-tuned for 40,000 steps with batch size 50 on an 80GB A800 GPU using about 700 samples of RGB-only tabletop manipulation data (Bai et al., 4 Feb 2026).

A common misconception is that EgoActor is itself a low-level whole-body controller. It is not. Its role is to predict a short, usable next action or action segment in language form and dispatch it to locomotion, manipulation, or interaction executors.

3. Supervision, data mixture, and annotation pipeline

EgoActor is trained with a broad multi-source supervision mixture rather than a single benchmark. The largest egocentric source is EgoTaskQA, from which the authors construct 160,000 EgoActing training samples, supplemented by 7,111 additional samples from 130 additional internet-collected egocentric videos. The preprocessing converts fine-grained temporal annotations and start/end frames of atomic actions into instruction-conditioned EgoActing samples with historical observations, recent observation-action pairs, and a target movement, manipulation, or terminal action (Bai et al., 4 Feb 2026).

The real-world EgoActing corpus further includes 398 egocentric videos recorded in local environments, yielding 150,214 EgoActing training samples. Virtual-environment supervision is drawn from two sources: approximately 3% of the VLN-CE Room-to-Room training set, producing 60,000 training samples, and a Habitat-Sim EgoActing dataset of 714 trajectories, split into 509 training trajectories and 205 validation trajectories from unseen environments, yielding 76,821 EgoActing samples.

Auxiliary supervision is deliberately heterogeneous. The model uses 44,160 spatial reasoning samples from 50% of the MindCube training set; 300,000 GQA samples plus 35,652 GPT-4o-annotated description samples from local environments for visual-language understanding; 241,603 data samples from RoboVQA, EgoPlan, and ALFRED for visual-language planning; 10,575 samples of unsupervised movement prediction between image pairs; and 3,629 EgoActing training samples collected from 70 successful traces of DAgger experience in real-world runs (Bai et al., 4 Feb 2026).

The annotation pipeline is correspondingly lightweight and scalable. For movement extraction, camera poses are estimated with MASt3R, and step-by-step movement actions are extracted at 1.5-second intervals. Human annotators write concise trajectory descriptions for real and virtual demonstrations, keeping route and target explicit, and a final NLA is appended to each trajectory. Because turning actions and natural language actions are underrepresented, they are oversampled during training.

This supervision design is central to EgoActor’s claim of scalability. Rather than relying on depth, maps, multiple cameras, or dense teleoperation, the method leverages broadly available egocentric RGB demonstrations, spatial reasoning question-answering, simulated trajectories, planning data, and limited on-policy correction.

4. Evaluation protocol and empirical performance

EgoActor is evaluated on a Unitree G1 humanoid equipped with Unitree Dex3-1 hands, a custom 2-DoF head, and a RealSense D455 camera. All experiments use 480p monocular RGB only. At inference time, EgoActor uses stochastic sampling with temperature 0.2, while the navigation baselines—NaVid-7B, Uni-NaVid-7B, and the VLM component of NaVILA-7B—use their original greedy settings (Bai et al., 4 Feb 2026).

The real-world human-robot interaction results are among the clearest demonstrations of the model’s scope. In single-person tasks, EgoActor-4B and EgoActor-8B both achieve 12/12 on Approach, Say hi, and Ask for location, while Request items reaches 11/12 for 4B and 12/12 for 8B. The strongest navigation baseline, by comparison, reaches 8/12 on Approach. In multi-person out-of-distribution disambiguation, the 8B model exceeds the 4B model across clothing, accessories, posture, direction, and gender cues, reaching 11/12 on clothing and 12/12 on direction (Bai et al., 4 Feb 2026).

On mobile manipulation, EgoActor is evaluated in approach-and-pick and approach-and-place settings for both seen and unseen objects. The 8B model is consistently stronger. For seen objects, it reaches 6/6 on bottle pick and 6/6 on apple and bottle place. For unseen objects, it reaches 6/6 on pink-cup pick and 5/6 on pink-cup place. The paper attributes a recurrent 4B failure mode to manipulation being triggered too early, when the robot is still too far from the target.

Traversability is tested in narrow door-like passages across 5 real-world rooms, including 3 seen and 2 unseen rooms, with entry and exit tasks from left and right starting positions. EgoActor markedly outperforms the navigation baselines overall. In unseen environments, for example, EgoActor-4B and EgoActor-8B each achieve 7/8 on entering from the left and 7/8 on entering from the right, whereas the baselines remain between 0/8 and 2/8 in those settings (Bai et al., 4 Feb 2026).

The virtual EgoActing benchmark uses 205 labeled EgoActing samples from unseen virtual environments. EgoActor-8B reaches 51.4% success within 0.5 m of the goal and 89.9% within 3.0 m; EgoActor-4B reaches 50.7% and 87.8% at the same thresholds. The navigation baselines are substantially lower under strict thresholds. EgoActor additionally reports Natural Language Action F1 of 0.60 for 4B and 0.62 for 8B, along with Final View Similarity of 0.41 for both models (Bai et al., 4 Feb 2026).

Qualitatively, the paper emphasizes several recurrent behaviors: looking down to verify obstacle positions, keeping gaze on the target near manipulation, moving backward and looking upward when only partial person views are available, and adapting action magnitude to scene geometry. These examples are used to argue that active perception is not auxiliary but part of the action policy itself.

5. Position within egocentric research

EgoActor belongs to a broader line of work that treats first-person perception as a privileged control or representation domain, but its immediate goal differs from earlier egocentric recognition and alignment systems. “Actor and Observer: Joint Modeling of First and Third-Person Videos” introduced Charades-Ego and ActorObserverNet to learn a shared representation between first-person and third-person videos for weakly supervised cross-view transfer, framing ego and exo video as coordinated views of the same behavior rather than isolated problems (Sigurdsson et al., 2018). EgoActor inherits the importance of egocentric grounding, but replaces cross-view metric learning with direct action prediction for humanoid execution.

It also differs from unified predictive egocentric world models such as EgoAgent, which jointly learns representation, future-state prediction, and next-body-motion prediction within a transformer. EgoAgent’s “act” component is 3D human skeleton forecasting conditioned on egocentric video and pose history, whereas EgoActor predicts executable language actions that route into locomotion, manipulation, and interaction controllers (Chen et al., 9 Feb 2025). A plausible implication is that EgoAgent and EgoActor occupy adjacent levels of the stack: one models embodied egocentric dynamics, the other operationalizes instruction-conditioned humanoid behavior.

Synthetic-data efforts also clarify EgoActor’s position. EgoInteract provides a Unity-based simulator for temporally coherent egocentric interaction episodes, with dense labels for temporal action segmentation, next-active object detection, anticipation, and hand-object interaction detection. Its central contribution is a controllable substrate for embodied first-person interaction modeling, especially around reach, grasp, hold, and release phases (Leonardi et al., 18 May 2026). EgoActor, by contrast, does not generate the underlying egocentric interaction world; it consumes egocentric observations and predicts the next robot action.

A different but related branch concerns avatar reconstruction and ego-driven animation. EgoAvatar combines person-specific full-body avatar modeling, monocular egocentric motion capture, ego-view-consistent geometry refinement, and free-view rendering from a head-mounted down-facing RGB camera (Chen et al., 2024). Whereas EgoAvatar centers on reconstructing and rendering the human carrier of the egocentric camera, EgoActor centers on instruction-conditioned robot behavior under first-person sensing.

Taken together, these comparisons place EgoActor at the intersection of egocentric action understanding, embodied VLM control, and humanoid task grounding. Its distinctive contribution is to make egocentric action prediction explicitly executable.

6. Limitations, misconceptions, and prospective development

The paper identifies several limitations that define the current scope of EgoActor. Most fundamentally, the system is not end-to-end. It depends on downstream locomotion policies, manipulation VLA models, speech routines, and, where relevant, high-level planners. EgoActor is therefore best understood as a bridging model between perception-and-instruction grounding and embodiment-specific executors, not as a complete autonomy stack (Bai et al., 4 Feb 2026).

Its temporal context mechanism is also limited. Because history is represented through sampled observations, recent observation-action pairs, and autoregressive text generation, the model can fall into locally optimal but incorrect decision patterns, especially on extended or multi-stage tasks. This limitation is consistent with the absence of a separately described long-horizon memory or persistent world model.

RGB-only sensing is simultaneously a practical strength and a structural constraint. The absence of depth, maps, and side-view coverage means that partial observability remains unresolved, and the paper notes that side collisions can occur when obstacles leave the field of view. The action interface is likewise interpretable but semi-discrete: movement is expressed through parameterized textual templates rather than through continuous end-to-end control.

Hardware-specific constraints also appear. Real-world stand-up and crouch-down actions were not supported because the Unitree locomotion policy lacked them, so body-height adjustment was demonstrated only in simulation. Additional failure cases include blurry or degraded virtual scenes, unfamiliar scene types such as churches or historical sites, and ambiguous instructions.

These constraints indicate that EgoActor is neither a pure planner nor a pure controller. Its significance lies in specifying a practical middle layer: an egocentric VLM that predicts short-horizon, spatially grounded actions quickly enough for iterative replanning, while spanning locomotion, active perception, manipulation, and human interaction in one textual action space. Future extensions suggested in the paper include tighter integration of planning and downstream skills, improved long-term context handling, broader natural-language action coverage, and further scaling of supervision (Bai et al., 4 Feb 2026).