OmniActor: Multimodal Agent Architectures

• OmniActor is a term describing two related constructs in multimodal agent design: one as an action-generation module and one as a full generalist agent.
• It leverages transformer-based diffusion methods and mixture-of-experts architecture to separate multimodal understanding from precise action execution.
• Empirical evaluations on embodied and GUI tasks demonstrate improved coordination and performance via dedicated module separation and shared perception layers.

Searching arXiv for the cited OmniActor papers to verify metadata and ensure up-to-date references. OmniActor denotes two distinct but conceptually adjacent constructs in recent multimodal agent research. In the iFLYTEK-Embodied-Omni line, “OmniActor” refers specifically to the Action-Generation Module (AGM), the action branch inside a unified vision–language–video–action architecture that plays the role of the “cerebellum” in a brain–cerebellum collaboration scheme (Zhang et al., 24 Jun 2026). In a separate line of work, “OmniActor” names a complete generalist agent for interleaved 2D GUI and 3D embodied interaction, built around a layer-heterogeneity Mixture-of-Experts design that shares shallow perception layers while separating deep action-oriented layers (Yang et al., 2 Sep 2025). The shared conceptual thread is a structural partition between multimodal understanding and action realization, but the two usages differ in granularity, architecture, and evaluation setting.

1. Terminological scope and naming

The term has an overloaded usage in the literature, and disambiguation is necessary for technical discussion.

Usage of “OmniActor”	Source	Meaning
OmniActor as AGM	"iFLYTEK-Embodied-Omni Technical Report" (Zhang et al., 24 Jun 2026)	The Action-Generation Module inside the unified “Omni” architecture
OmniActor as full agent	"OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds" (Yang et al., 2 Sep 2025)	A complete multimodal agent for GUI and embodied tasks

In the first sense, OmniActor is not a standalone agent but a module: the AGM is the action-generation component within iFLYTEK-Embodied-Omni, downstream of the Vision–LLM (VLM) and Video Generation Model (VGM). In the second sense, OmniActor is itself the full policy model $\pi$ that receives an image $I$, task instruction $T$, and optional history $H$, and outputs either GUI actions or embodied actions.

A plausible implication is that the name collision reflects convergent design intuitions rather than a common implementation lineage. Both systems explicitly invoke a cortex/cerebrum-versus-cerebellum analogy, but one uses it to organize submodules within a single embodied foundation model, whereas the other uses it to motivate partial parameter sharing across task families.

2. OmniActor as the action-generation module in iFLYTEK-Embodied-Omni

Within iFLYTEK-Embodied-Omni, the AGM is a Transformer-based diffusion model that inherits temporal-generation priors from the VGM, is initialized from the VGM’s weights, and plugs into the shared Omni Multi-Modal Self-Attention backbone so that its action tokens attend to the same unified context of vision, language, and video (Zhang et al., 24 Jun 2026).

Its inputs are organized as three token streams. The language instruction $\ell$ and current image observation $o_t$ are encoded by the VLM’s input encoder into text and image tokens. Optionally, a short history of predicted or real future-frame latents from the VGM forms a video-context stream. In parallel, a noisy action-latent chunk $z_a^{\sigma_a} \in \mathbb{R}^{n \times d}$ is produced by adding Gaussian noise at level $\sigma_a$ to the ground-truth action sequence in latent space.

The AGM then applies a three-part Transformer stack. First, modality-specific input blocks with $M$ layers process the language+image stream, the video stream, and the noisy action stream separately. Second, the three streams are concatenated and passed through $L$ layers of Omni Multi-Modal Self-Attention with a learned, modality-aware attention mask. At this stage, action tokens can attend to all language, image, and video tokens, while language and video tokens do not attend to future action tokens; language is causally masked and video is bidirectional only within its own stream. Third, the fused sequence is split back into modalities, and the action branch passes through $I$0 additional Transformer layers before a small MLP velocity head $I$1 predicts the action-velocity field $I$2.

The brain–cerebellum metaphor is operationalized through module roles. The VLM generates a sequence of subgoals in semantic space, such as “pick up bowl,” “move to stove,” and “turn knob.” The VGM translates each subgoal plus the current state into a predicted sequence of future visual latents. The AGM then conditions on those predicted visual latents, the original instruction $I$3, and the latest real image $I$4, and through iterative diffusion—specifically 30 denoising steps under V2A-style staged inference—refines the noisy action latents into a coherent control sequence for the planned subgoal. The report characterizes this as the cerebellar function of error correction, temporal smoothing, and fine-grain coordination over each action horizon.

The self-attention operator is written in the standard form

$I$5

$I$6

3. Curriculum, losses, and optimization for the AGM

The AGM is introduced through a four-stage curriculum designed to avoid conflicts among language, video, and action objectives (Zhang et al., 24 Jun 2026).

Stage I is VLM fine-tuning on image–text, spatial reasoning, embodied QA, and task planning, with autoregressive language modeling loss $I$7. Stage II is VGM training for world modeling on action-free human videos and robot videos without actions, using diffusion-based future video prediction loss $I$8. Stage III is AGM training: the VLM and VGM are frozen, the AGM is initialized from VGM weights, the data are action-annotated robot trajectories comprising 26.9% of the mixture, and the objective is diffusion-based action-velocity regression $I$9. Stage IV jointly fine-tunes all three branches and the Omni layers on all modalities, with each sample activating only the relevant objective terms.

The action-diffusion target is defined by sampling $T$0 and constructing

$T$1

The AGM predicts

$T$2

and minimizes

$T$3

where $T$4 masks padded or invalid action dimensions and $T$5 is a small constant.

The corresponding video-generation and language-modeling losses are

$T$6

and

$T$7

Joint fine-tuning uses

$T$8

The report states that regularization is implicit in dropout inside Transformer blocks and that no additional action-specific regularizer was used beyond the diffusion $T$9 weighting. It also emphasizes a division of labor across data sources: action-free human videos teach the VGM how the world evolves, action-labeled data teach the AGM how to turn those dynamics priors into motor commands, and spatial-reasoning and QA data ensure that the VLM can issue correct subgoals.

4. Empirical role of the AGM in embodied evaluation

Ablations isolate the AGM’s contribution on LIBERO-Plus under seven types of domain shift (Zhang et al., 24 Jun 2026). A two-branch MoT configuration in which video and action are combined achieves 85.9% average success, whereas a three-branch MoT with a separate AGM reaches 89.6%, a gain of 3.7 points. The stated interpretation is that dedicating capacity to action generation and interfacing it through Omni cross-attention improves performance.

On long-horizon benchmarks consisting of seven staged manipulation tasks on RoboTwin 2.0, the full Omni model with the cerebellar AGM achieves 88.3% on the Clean split and 89.0% on the Rand split, exceeding the nearest baseline by 1.3 points and 3.4 points respectively. End-to-end, the complete iFLYTEK-Embodied-Omni model, comprising VLM, VGM, and AGM, scores 89.6% zero-shot on LIBERO-Plus and 93.7% and 93.2% on the Clean and Rand splits of RoboTwin 2.0.

These measurements are presented as evidence that the AGM is the module that transforms high-level plans and predicted future states into temporally consistent low-level control. A plausible implication is that the architectural separation between semantic planning and action diffusion is beneficial specifically when long-horizon embodied behavior depends on both future-state anticipation and motor refinement.

5. OmniActor as a generalist GUI and embodied agent

In the second usage, OmniActor is a complete multimodal agent intended for tasks that require interleaved interaction with 2D virtual worlds and 3D real worlds (Yang et al., 2 Sep 2025). Its high-level goal is to build one multimodal agent $H$0 that, given an image $H$1, a task instruction $H$2, and optional history $H$3, can output either 2D GUI actions such as click, drag, and keystroke, or 3D embodied actions represented as 6-DoF end-effector motion plus gripper.

The architectural core is a Transformer-MoE backbone built from a pre-trained vision-language Transformer such as Qwen2-VL-7B. Layers $H$4 are fully shared across task families, while layers $H$5 are split into two experts, one for GUI and one for embodied interaction. A one-hot task indicator determines which expert path is used. The forward pass is described as follows: the image $H$6 is encoded by the frozen vision encoder of the base MLLM; the system prompt, task instruction, and history $H$7 are tokenized with the base text tokenizer; the joint embeddings pass through shared Transformer layers; and at layers $H$8 the computation forks into GUI or robot-specific blocks. The shared-layer update is written

$H$9

$\ell$0

The final head $\ell$1 or $\ell$2 decodes into discrete action tokens representing GUI operations or discretized 6-DoF values.

Although the paper describes the design as a Mixture-of-Experts, it notes that in practice there are only shared shallow layers and two deep experts. A generic MoE formulation is given by

$\ell$3

$\ell$4

$\ell$5

For OmniActor, $\ell$6 and the routing is a deterministic one-hot selection based on task type. The paper states that this strict separation prevents deep-layer interference while shared shallow layers preserve common perception pathways.

Action-space unification is central to the model. Every sample is cast into a quadruple consisting of system prompt, screenshot image, natural-language instruction, and action tokens. GUI actions are tokenized by the base text tokenizer. Embodied actions—$\ell$7, $\ell$8, and gripper—are each normalized to $\ell$9, uniformly discretized into $o_t$0 bins, and mapped to otherwise rarely used text-token IDs. The data pipeline includes GUI grounding pre-train of approximately 3.4 million steps on OS-Atlas, UGround, Aria-UI, and Aguvis, followed by trajectory fine-tune of approximately 4.1 million steps mixing Aguvis and LIBERO, with LIBERO resampled $o_t$1 to match GUI magnitude. The overall GUI-to-embodied sample ratio is approximately $o_t$2 in the trajectory stage.

Training uses supervised token prediction only:

$o_t$3

The paper explicitly states that no reinforcement objectives are used in the current release. The optimization schedule is AdamW with learning rate $o_t$4, cosine decay with warm-up equal to 3% of steps, batch size 128 for grounding and 64 for trajectory training, DeepSpeed ZeRO-3 parallelism, image size 448×448, LIBERO frames upsampled from $o_t$5, and network depth $o_t$6 with sharing threshold $o_t$7.

6. Benchmarks, ablations, limitations, and interpretation

The full-agent OmniActor is evaluated on GUI tasks—AndroidControl-Low, AndroidControl-High, and GUI-Odyssey—and on embodied tasks via LIBERO-90, defined as 90 unseen kitchen, living, and study demos with a maximum of 400 steps (Yang et al., 2 Sep 2025). GUI-Odyssey is described as cross-app navigation up to 800 steps.

Model	LIBERO-90	AC-Low	AC-High	Odyssey
OmniActor-GUI	–	89.4	73.8	63.0
OmniActor-EA	63.4	–	–	–
OmniActor	69.5	86.4	77.5	66.0

The comparison highlights given in the paper are that single-environment agents perform well in their own domain but cannot do both, that prior generalists such as Magma, GEA, and NaviMaster typically score below 50% on LIBERO or lag behind state-of-the-art GUI methods on AndroidControl and GUI-Odyssey, and that OmniActor matches or exceeds single-environment agents in both domains. Qualitative examples include a 15-step hotel-booking sequence on AndroidControl-High without intermediate errors and a pick-and-place task in LIBERO kitchen completed in fewer than 200 steps.

The ablation study focuses on how much to share across GUI and embodied data. A “Mixed-no-MoE” setting, corresponding to naïve joint training, degrades both GUI and robot accuracy by approximately 10–15%, which the paper attributes to deep-stage conflicts. A “Hard-MoE” setting with complete separation of all layers recovers performance but trails OmniActor by approximately 3% on average because it loses shared perception synergy. The layer-heterogeneity MoE, with shallow sharing and deep splitting, is presented as the configuration that best balances synergy and conflict.

The stated limitations are benchmark and representation specific. Embodied benchmark scale remains smaller than GUI, with a possible tendency to overfit to LIBERO-style scenes. Discretization of continuous 6-DoF actions may lose fine-grained control. GUI training on mobile screenshots may not generalize to desktop or web settings. Failure modes include clicking the wrong region in rare GUI layouts with unseen widgets and stalling around 300 steps for under-represented robot goals such as stacking in a study room.

Future directions listed in the paper include adding more embodied datasets such as CALVIN and RT-x, integrating on-policy RL or human-in-the-loop fine-tuning, extending the MoE to more specialized experts such as vision-only grounding and language-only planning, and replacing fixed task flags with fully learned gating networks. Taken together with the AGM usage of the term, this suggests that “OmniActor” has become associated with architectures that isolate action-specific computation while preserving shared multimodal understanding, but the exact implementation differs substantially between the embodied-module formulation and the full 2D/3D generalist agent.