OmniAction: Multimodal Action Intelligence

Updated 5 July 2026

OmniAction is a family of multimodal, action-centric frameworks spanning long-video understanding, human-computer interaction, mobile agents, and robotics.
It combines active perception, persistent textual memory, and unified token-level action spaces to convert multimodal inputs into executable actions.
Empirical results demonstrate increased efficiency and accuracy across diverse applications, from proactive robot manipulation to cross-modal digital follow-up actions.

Searching arXiv for papers using or defining “OmniAction” and closely related variants. OmniAction is a term used in recent arXiv literature to denote several related but non-identical constructs at the intersection of multimodal perception, reasoning, and action. In long-video understanding, it denotes comprehensive omni-modal action understanding over audiovisual streams through active perception and persistent textual memory (Xing et al., 17 Jun 2026). In human–computer interaction, it denotes prediction of digital follow-up actions from real-world multimodal sensory inputs (Li et al., 2024). In mobile agents, it denotes the execution backbone that grounds multimodal intent into Android operations (Ren et al., 7 May 2026). In generalist 2D–3D agents, it denotes a unified token-level action space spanning GUI and embodied control (Yang et al., 2 Sep 2025). In robotics, it also denotes a large-scale dataset for proactive manipulation from cross-modal contextual instructions (Wang et al., 27 Oct 2025). Taken together, these usages suggest that “OmniAction” functions less as a single canonical benchmark than as a family of formulations for action-centric omni-modal intelligence.

1. Terminological scope and major usages

The term appears in multiple technical contexts, each emphasizing a different stage of the perception–reasoning–action pipeline.

Work	Meaning or use of “OmniAction”	Domain
(Xing et al., 17 Jun 2026)	omni-modal action understanding objective supported by OmniAgent	long audiovisual streams
(Li et al., 2024)	prediction of digital follow-up actions	pervasive AR and multimodal HCI
(Ren et al., 7 May 2026)	execution backbone of X-OmniClaw	Android mobile agent
(Wang et al., 27 Oct 2025)	large-scale dataset for proactive manipulation	cross-modal robot intent recognition
(Yang et al., 2 Sep 2025)	unified token-level action space	GUI and embodied generalist agents

The long-video formulation is explicitly action-centric: it concerns “what action, when it starts/ends, who did it, causal relations” under joint audio-visual grounding (Xing et al., 17 Jun 2026). The HCI formulation instead centers on target information and executable digital follow-up actions such as “share with others,” “save for reference,” “search online,” “transcribe speech,” or “recognize a song” (Li et al., 2024). The mobile-agent and generalist-agent usages shift the emphasis from action understanding to action execution: one grounds structured intents into Android operations, and the other serializes GUI and robot actions into a single autoregressive vocabulary (Ren et al., 7 May 2026, Yang et al., 2 Sep 2025). The robotics-dataset usage frames OmniAction as supervision for proactive intention recognition from dialogue, environmental sound, and visual context rather than explicit commands (Wang et al., 27 Oct 2025).

In “Native Active Perception as Reasoning for Omni-Modal Understanding,” OmniAction is aligned with an objective of recognizing, localizing, and reasoning about actions and events in long audiovisual streams by actively deciding where and how to perceive, compressing transient high-dimensional media into a persistent textual memory, and using this memory to plan subsequent actions and produce final answers grounded in evidence (Xing et al., 17 Jun 2026). The corresponding model, OmniAgent, formulates video understanding as a POMDP-based iterative Observation–Thought–Action cycle. At each turn, it can issue one symbolic action among {get_frames, get_audio, get_clip, answer}; the environment executes the action, returns raw media, the agent distills it to text, and raw media are then purged from context.

The formalization is explicitly POMDP-based: $\mathcal{M} = (S, A, O, T, R, \Omega, Z).$ Here, latent video content is only partially observable through agent-initiated sensing. A persistent textual memory stores OTA tuples consisting of observation, thought, and action. The stated purpose of this design is strict information distillation and decoupling of reasoning complexity from raw video duration (Xing et al., 17 Jun 2026). In the paper’s complexity analysis, context growth depends on the number of turns $K$ , not video duration; watch-it-all baselines instead scale with retained visual or audio tokens.

This formulation is paired with two training procedures. Agentic Supervised Fine-Tuning uses best-of- $N$ trajectory synthesis with dual-stage quality control: outcome verification retains only correct trajectories under verifiable reward criteria, and a GPT-4o rationality audit filters “lucky guesses” using a 5-point coherence scale with a threshold of at least $3/5$ (Xing et al., 17 Jun 2026). Agentic Reinforcement Learning introduces TAURA, or Turn-aware Adaptive Uncertainty Rescaled Advantage, motivated by the claim that vanilla GRPO suffers from “Advantage Homogenization,” because one scalar advantage is broadcast to all turns.

Empirically, the paper reports positive test-time scaling: on VideoMME-Long, increasing the maximum turn limit $K$ from 6 to 52 improves accuracy monotonically from 53.4% to 59.6%, while average executed turns saturate at approximately 11.7 even when $K=52$ (Xing et al., 17 Jun 2026). On LVBench, OmniAgent-7B uses approximately 203 frames versus 768 for Qwen2.5-VL-72B, about 73% fewer frames, yet reports higher accuracy, 50.5% versus 47.3%. The benchmark summary includes 67.8% on VideoMME overall, 59.6% on VideoMME-Long, 48.4% on VSI-Bench AVG, 71.1% on MLVU M-AVG, 41.4% on Minerva AVG, 64.8% on DailyOmni, 47.2% on WorldSense, 37.1% on OmniVideoBench, 39.1 IoU on LongVALE, and 36.5 / 46.1 on VUE-TR Vision+Audio / Vision, described as state-of-the-art among open-source models (Xing et al., 17 Jun 2026).

A related but distinct encoder-centric line, OmniEncoder, addresses the fine-grained motion side of omni-modal action analysis by co-embedding vision and audio at a symmetrical 25 fps in a unified Transformer (Bai et al., 2 May 2026). Under the same decoder token budget as Qwen2.5-Omni-3B SFT, it reports 90.8 on Diving48, 68.7 on Something-Something v2, 97.8 on SLR500, 90.32 on NationalCSL6707, 82.6 on AVQA, 98.8 on Speaker Localization, and 82.31 on Speaker Identification. This suggests that encoder-level temporal fidelity and agentic active perception address complementary failure modes in omni-modal action understanding.

3. Digital follow-up actions in multimodal HCI

“OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs” defines a different but precise action-centric problem: reducing the friction users face when acting on multimodal information encountered in everyday life under “Pervasive Augmented Reality” (Li et al., 2024). The system distinguishes multimodal sensory inputs, target information, and digital follow-up actions. Target information can be one of five modalities—scene, object, text, speech, or sound—and follow-up actions are executable digital operations such as sharing, saving, searching, transcription, recognition, translation, and media manipulation.

The work is grounded in a five-day diary study with $N = 39$ participants recruited via dscout, yielding 382 total entries with approximately a 2:1 ratio of visual to audio. Entries included 254 visual submissions and 128 audio-related submissions. Many entries contained multiple actions: 183 had one action, 147 had two, 44 had three, and 8 had four (Li et al., 2024). The final design space consists of 7 general categories—Share, Save, Remind, Look up, Digital extract, Media manipulation, and Complex actions—and 17 specific categories. Reported general-action frequencies are Save 47.4%, Share 45.9%, Look up 32.1%, Remind 4.5%, Digital extract 12.3%, Media manipulate 2.8%, and Complex 2.1%.

The pipeline converts multimodal inputs to structured text through image captioning with InstructBLIP, object detection with Detectron2, OCR with Google Cloud Vision, sound classification with YAMNet, and speech-to-text with Google Cloud Speech in the prototype (Li et al., 2024). It then generates Chain-of-Thoughts from participants’ stated goals and reasons, and predicts both target information and follow-up actions constrained by the taxonomy. The multi-label action metric is defined as

$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \frac{C_i}{\min(G_i, P_i)},$

where $N$ is the number of test samples, $C_i$ the number of correct predictions, $K$ 0 the number of ground-truth labels, and $K$ 1 the number of predictions (Li et al., 2024).

Among the evaluated methods, GPT-4 with in-context CoT performs best. Reported top- $K$ 2 full-match accuracies for general actions are 60.3%, 69.9%, and 94.3% for top-1, top-2, and top-3; for specific actions they are 44.4%, 52.9%, and 67.1% (Li et al., 2024). The ablation on context shows that, for GPT-4 in-context learning on top-3 specific actions, accuracy rises from 52.5% without context to 67.1% with full context, with activity information described as especially important. The paper also reports a prototype Android app and an in-lab feedback session with $K$ 3, where ease of use, liking, and perceived potential/promise were rated at $K$ 4, $K$ 5, and $K$ 6, respectively, on 7-point Likert scales (Li et al., 2024).

4. OmniAction as an execution fabric and unified action space

In “X-OmniClaw Technical Report,” OmniAction is the execution backbone of a unified mobile agent. It closes the perception–memory–action loop by transforming scene-grounded intents into Android operations through hybrid UI grounding, reusable skills distilled from user behavior, and precise direct-access mechanisms (Ren et al., 7 May 2026). The agent loop is Observe → Ground → Plan → Execute. Grounding fuses XML metadata, screenshot understanding, OCR, and icon recognition; execution uses AccessibilityService-driven taps and swipes, text entry, system navigation, and direct-access through intents or deeplinks when available.

The report emphasizes hybrid grounding because Android exposes partially reliable structural signals and rich but noisy visual cues. XML contributes IDs, class names, content descriptions, and bounds; the visual channel contributes VLM-based screenshot understanding, OCR, icon detection, and layout priors (Ren et al., 7 May 2026). Behavior Cloning and Trajectory Replay then convert ordinary user navigation into reusable skills, while dumpsys activity introspection is used to capture action, data URI, and extras for direct-access replay. The paper reports demos across scenarios such as price-checking, one-tap video generation in CapCut, and replay to Meituan flash-sale pages, but does not present numeric evaluation tables.

OmniActor generalizes the execution-side meaning of OmniAction further by defining it as the unified, token-level action space that spans GUI and embodied tasks, allowing a single model to decode operations across 2D virtual and 3D physical environments (Yang et al., 2 Sep 2025). GUI actions such as click, tap, drag, scroll, type, keypress, and key combinations are serialized as text tokens using the LLM tokenizer. Embodied actions are 6-DoF end-effector displacements plus gripper state, normalized to $K$ 7, uniformly discretized into $K$ 8 bins, and mapped to rare token IDs from the same vocabulary.

This action-space unification is coupled with Layer-heterogeneity MoE. The first $K$ 9 transformer layers are shared between GUI and robot data, while deeper layers use modality-specific experts and separate heads (Yang et al., 2 Sep 2025). The stated motivation is empirical: gradients from GUI and embodied data align in shallow layers but diverge in deep layers. The model is trained on GUI grounding data from OS-Atlas, UGround, Aguvis, and Aria-UI, GUI trajectories from Aguvis, and embodied trajectories from LIBERO, totaling approximately 3.4M steps for grounding pretraining and approximately 4.1M for trajectory learning. Reported results are LIBERO 69.5, AndroidControl-Low 86.4, AndroidControl-High 77.5, and GUI Odyssey 66.0 for the full OmniActor system, compared with 63.4 for OmniActor-EA on LIBERO and 89.4 / 73.8 / 63.0 for OmniActor-GUI on the GUI tasks (Yang et al., 2 Sep 2025).

Taken together, these two works instantiate OmniAction at different levels of granularity. X-OmniClaw treats it as a mobile execution substrate centered on robust grounding and direct access. OmniActor treats it as a cross-domain action language enabling a single autoregressive policy to alternate between GUI and robot control.

5. Proactive robot manipulation and the OmniAction dataset

In “RoboOmni: Proactive Robot Manipulation in Omni-modal Context,” OmniAction is a dataset rather than a controller or execution layer (Wang et al., 27 Oct 2025). It is designed for scenarios where intent is not given as an explicit command but must be inferred from “cross-modal contextual instructions” arising from spoken dialogue, environmental sounds, and visual context. This setting is described as proactive intention recognition: inferring and verifying a user’s goal before acting.

The dataset scale is specified precisely. OmniAction contains 141,162 multimodal episodes expanded from 74,645 base trajectories filtered from Open-X Embodiment; it spans 112 manipulation skills and 748 manipulable objects (Wang et al., 27 Oct 2025). Audio diversity includes 5,096 distinct speaker timbres across six demographic categories, 2,482 non-verbal sound events, and 640 environmental backgrounds. Each sample has the triplet $N$ 0, where $N$ 1 is a multi-turn conversation with user turns as audio and assistant turns as text, $N$ 2 is a visual observation sequence, and $N$ 3 is a 7-DoF delta end-effector trajectory. The assistant’s final action onset is explicitly marked by [[ACT](https://www.emergentmind.com/topics/adversarial-camouflage-textures-act)].

The corpus covers six contextual instruction types: Sentiment Cues, Overlapping Voices, Non-Verbal Cues, Identity Cues, Dyadic Dialogue, and Triadic Dialogue (Wang et al., 27 Oct 2025). Its generation pipeline has three stages: textual scripting from Open-X Embodiment trajectories with GPT-4o-based rewriting into contextual dialogues; auditory realization with MOSS-TTSD, CosyVoice, and Google Gemini-TTS plus multi-speaker overlap and environmental sound insertion; and manual verification. The paper reports 98.7% agreement that task intent is recoverable from the multimodal context.

Evaluation is conducted through OmniAction-LIBERO. In OmniAction-LIBERO-TTS, defined as 40 LIBERO tasks multiplied by 6 contextual variants for 240 evaluation tasks, RoboOmni reports 85.6% average success (Wang et al., 27 Oct 2025). Suite-wise averages are 93.0% on Spatial, 85.8% on Goal, 84.0% on Object, and 79.5% on Long-Horizon. On OmniAction-LIBERO-Real, which uses 10 human volunteers recording spoken instructions in real environments, RoboOmni reports 76.6%, outperforming $N$ 4 at 73.8%, OpenVLA at 40.1%, OpenVLA-OFT at 6.5%, and NORA at 17.4%. Proactive intention recognition accuracy is reported at 88.9% for RoboOmni, compared with 50.0% for Qwen2.5-Omni-7B, 27.8% for Qwen2.5-Omni-3B, and 55.6% for ASR+GPT-4o (Wang et al., 27 Oct 2025). The paper also states that RoboOmni runs at 0.49× latency relative to an ASR+OpenVLA baseline normalized to 1.00×.

The learning objective is unified over text and action tokens: $N$ 5 This is central to the Perceiver–Thinker–Talker–Executor design, in which perception, interaction, and control are optimized as one autoregressive process (Wang et al., 27 Oct 2025).

A related but separately motivated embodiment framework is UniManip, which does not define OmniAction as a dataset or action space but presents a Bi-level Agentic Operational Graph that couples semantic reasoning with physical grounding for zero-shot manipulation (Liu et al., 13 Feb 2026). Its reported results—93.75% average overall success rate against 71.25% for the best VLA baseline, 82.5% average task success against 47.5% for ReKep, and 80.00% average success in long-horizon ablations with Failure Detection & Recovery—suggest a plausible control substrate for embodiment-centric versions of OmniAction (Liu et al., 13 Feb 2026).

6. Recurrent design patterns, evaluation regimes, and limitations

Across these lines of work, several recurrent technical motifs appear. One is selective perception. OmniAgent replaces watch-it-all processing with on-demand sensing through OTA turns (Xing et al., 17 Jun 2026). X-OmniClaw uses hybrid grounding instead of purely structural or purely visual UI understanding (Ren et al., 7 May 2026). cotomi Act, though not itself named OmniAction, contributes closely related techniques for long-horizon computer-use execution: adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and best-of- $N$ 6 action selection. On the 179-task WebArena human-evaluation subset, it reports 80.4% success versus a 78.2% human baseline; lazy observation reduces median task latency from 471 s to 184 s while maintaining comparable accuracy (Oyamada et al., 4 May 2026).

A second motif is structured memory. OmniAgent’s persistent textual memory records OTA tuples and is designed so that context growth depends on turn count rather than raw video length (Xing et al., 17 Jun 2026). X-OmniClaw explicitly separates working memory from long-term personal memory, enabling skill retrieval and personalization (Ren et al., 7 May 2026). cotomi Act externalizes organizational knowledge into task boards, wiki pages, and timelines, and reports that success increases with coverage of behavior-derived knowledge, with gains up to +10 percentage points over a roughly 51% zero-coverage baseline on WorkArena-L1 proxy evaluation (Oyamada et al., 4 May 2026).

A third motif is action abstraction. In OmniActions, actions are taxonomy-constrained digital operations (Li et al., 2024). In OmniActor, they are unified tokens over GUI and robot control (Yang et al., 2 Sep 2025). In RoboOmni, chat and action tokens share one autoregressive vocabulary (Wang et al., 27 Oct 2025). In UniManip, operations are formalized as transit and manipulation operators inside an Agentic Operational Graph, with verification and recovery operators closing the loop (Liu et al., 13 Feb 2026).

Limitations recur as well, but differ by domain. OmniAgent identifies latency from sequential OTA turns, reliance on decisive forensic cues, and the absence of external expert tools such as ASR, diarization, or tracking (Xing et al., 17 Jun 2026). OmniActions notes privacy constraints on audio collection, phone-centric action familiarity, label imbalance, and cognitive load when presenting many alternatives (Li et al., 2024). X-OmniClaw highlights dynamic UI changes, poorly labeled views, cross-app workflows, multilingual text, privacy constraints around memory, resource limits, and security guardrails for sensitive actions (Ren et al., 7 May 2026). RoboOmni notes synthetic speech, incomplete environmental coverage, and the lack of explicit dataset splits or public-availability details (Wang et al., 27 Oct 2025). OmniActor identifies limited robot-data diversity, reliance on known task type, and the simplicity of uniform action binning (Yang et al., 2 Sep 2025).

In aggregate, the literature positions OmniAction as a broad research agenda centered on omni-modal, action-oriented intelligence: perceiving selectively, compressing context into durable memory, inferring intent under partial observability, and executing grounded actions across video, mobile, GUI, and robotic environments. The exact instantiation varies, but the shared problem is stable: aligning multimodal evidence with verifiable action.