OmniPlay Benchmark for Omni-Modal Agents

• OmniPlay is an interactive benchmark for evaluating omni-modal agents that fuse text, image, audio, and video in dynamic, game-based environments.
• It features diverse game settings that create modality conflicts and test the agents' ability to reconcile contradictory sensory signals.
• Empirical results show that while models exhibit superhuman memory, they struggle with strategic planning and robust fusion under noisy conditions.

OmniPlay most directly denotes an interactive benchmark for evaluating omni-modal agents—models that can process text, image, audio, and video—inside dynamic game worlds rather than static one-shot QA settings. It was introduced to bridge an “evaluation chasm” between static multimodal benchmarks, which test perception and understanding but not agency, and interactive agent benchmarks, which usually remain largely vision-language only and therefore miss audio and richer temporal cues (Bie et al., 6 Aug 2025). In adjacent work, “OmniPlay” also appears as a broader ambition of omnimodal embodied play/operation and as a shorthand for omni-modal interaction in robotics; this suggests a wider research motif centered on multi-sensory agency, sequential decision-making, and action across interactive environments (Shi et al., 25 Jun 2026, Wang et al., 27 Oct 2025).

1. Formal definition and design philosophy

OmniPlay is built around what the paper calls modality interplay or modality interdependence. The central claim is that an agent should not merely receive multiple modalities at once; success should actually require the agent to combine them in a nontrivial way. This is formalized by requiring that a full omni-modal policy substantially outperform any policy that conditions on only one modality:

$$\max_{\pi} \sum_{\tau \in \mathcal{T}_G} P(\tau \mid \pi) \gg \max_m \left( \max_{\pi_m} \sum_{\tau \in \mathcal{T}_G} P(\tau \mid \pi_m) \right),$$

where $\pi_m$ is a policy that uses only a single modality $$m \in \{\mathcal{I}, \mathcal{V}, \mathcal{A}, \mathcal{T}\}$$, and $\mathcal{T}_G$ is the set of trajectories ending in a goal state (Bie et al., 6 Aug 2025).

The benchmark models each game as a generalized MDP with tuple $(S, A, T, G, \Omega, O)$, where $S$ is the state space, $A$ the action space, $T$ the transition function, $G$ the goal states, $\Omega$ the multimodal observation space, and $\pi_m$0 the observation function. At each step, the observation is

$\pi_m$1

and the model encodes each modality with separate encoders and fuses them:

$\pi_m$2

The appendix further specifies the trajectory probability under a policy $\pi_m$3 as

$\pi_m$4

with the optimal policy

$\pi_m$5

This formulation is diagnostic rather than merely competitive. OmniPlay deliberately injects modality conflict, where different channels recommend contradictory actions, in order to determine whether a model has a genuine fusion mechanism or only a brittle preference hierarchy. A plausible implication is that the benchmark is less concerned with aggregate multimodal accuracy than with the internal arbitration logic by which models reconcile agreement, ambiguity, and contradiction.

2. Game environments and the structure of cross-modal reasoning

OmniPlay comprises five game environments, each constructed to probe a distinct form of cross-modal reasoning. The environments create both synergy, where one modality alone is insufficient, and conflict, where one modality is intentionally misleading while others remain correct (Bie et al., 6 Aug 2025).

Environment	Modalities	Diagnostic role
Whispered Pathfinding	image, audio, text	3D maze-navigation with spoken guidance and textual state information
Myriad Echoes	video/audio/text; image/audio/text	sequence parsing and later reproduction by clicking icons
The Alchemist’s Melody	image, audio, text	hidden mapping between colors and musical notes
Phantom Soldiers in the Fog	video, audio, text	squad-based RTS command under fog-of-war
Blasting Showdown	audio and game-state cues	multi-agent arena combat scored by tournament metrics

In Whispered Pathfinding, the agent must reach a hidden target by combining visual observation with spoken guidance and textual state information. The environment is also the main site of conflict experiments, such as cases where a visual cue may say “turn right” while audio says “turn left,” or the text prompt provides contradictory orientation or target information. In Myriad Echoes, the agent first parses a stream of audio-visual cues and later reproduces the sequence by clicking icons, making deferred execution and multimodal memory central. In The Alchemist’s Melody, the task is abstract rule discovery: the agent must infer a hidden mapping between colors and musical notes and then reproduce a musical scale.

The remaining environments emphasize strategic planning and adversarial decision-making. Phantom Soldiers in the Fog is a squad-based RTS task under fog-of-war conditions in which the agent acts as commander using video, audio, and text to issue strategic commands under uncertainty. Blasting Showdown, inspired by Bomberman, evaluates survival and combat in a multi-agent arena where agents move, place bombs, and use audio cues; unlike the other tasks, it is assessed with AI-vs-AI tournament measures rather than normalized human comparison.

These environments operationalize the benchmark’s core thesis: omni-modality is not treated as the passive co-presence of channels, but as a condition in which the channels must be fused, weighted, or overruled under changing temporal and strategic demands.

3. Evaluation protocol, baselines, and scoring

The benchmark evaluates six leading omni-modal models: Gemini 2.5 Pro, Gemini 2.5 Flash, Qwen-2.5-Omni (7B), MiniCPM-o-2.6 (8B), Baichuan-Omni-1.5 (7B), and VITA-1.5 (7B). It also uses two baselines: a Random agent, which uniformly samples an action from the available action set, and a Human expert, based on five experienced human players aged 22–28, each with over 500 hours of gaming experience. Each human completed 10 warm-up episodes per game before recorded evaluation, and the final human baseline is the mean over all collected episodes. Evaluation uses fixed seeds, so every agent sees the exact same sequence of scenarios; the paper reports 50 seeds for Whispered Pathfinding and Myriad Echoes, 30 for Phantom Soldiers, and 50 AI-vs-AI games for Blasting Showdown (Bie et al., 6 Aug 2025).

The main cross-task metric is Normalized Performance Score (NPS), which scales a model’s raw task score between random and human baselines:

$\pi_m$6

Under this convention, $\pi_m$7 denotes random performance, $\pi_m$8 human-level performance, and $\pi_m$9 superhuman performance. Blasting Showdown is excluded from NPS because it is competitive AI-vs-AI rather than a human-vs-model benchmark.

Task-specific metrics are tailored to each environment. Whispered Pathfinding uses Mean Steps, Min/Max Steps, Invalid Actions, and a trimmed mean, with NPS based on the inverse of trimmed mean steps. Myriad Echoes uses Success Rate, Mean Score, Coordinate Accuracy, Icon Accuracy, and Parsing Failure Rate, with NPS defined as a weighted sum of 50% Mean Score from execution, 25% Coordinate Accuracy, and 25% Icon Accuracy. The Alchemist’s Melody uses a composite score consisting of Hit Rate, Step Efficiency, Correct Streak, Error Penalty, Color Error Penalty, and Exploration. Phantom Soldiers in the Fog uses Success Rate and Normalized Score in a 50/50 weighted sum. Blasting Showdown uses Win Rate, Total Kills, Total Deaths, and K/D ratio.

A common misunderstanding would be to treat OmniPlay as a single aggregate leaderboard. Its scoring design instead partitions performance into memory fidelity, execution accuracy, efficiency, and strategic competence, thereby exposing which sub-capacities of omni-modal agency remain weak even when overall multimodal responsiveness appears strong.

4. Empirical results: memory strength, reasoning weakness

The central empirical result is a sharp split between superhuman memory and weak reasoning or planning. In Myriad Echoes, especially on harder settings, models such as Gemini 2.5 Pro exhibit extremely strong performance; the paper highlights an NPS of 399.2 for Gemini 2.5 Pro on Myriad Echoes (Hard), which it interprets as evidence of high-fidelity memory or transcription strength. By contrast, in tasks requiring deeper reasoning, strategic planning, or robust decision-making under uncertainty, all six models fall short of the human baseline, and some even score below random in the normalized scale. In Blasting Showdown, the best model reaches only a 36.1% win rate, which the authors treat as evidence of weak adaptive multi-agent strategy (Bie et al., 6 Aug 2025).

The same contrast appears in planning-heavy and conflict-heavy environments. The paper specifically notes that Gemini 2.5 Pro performed poorly in Phantom Soldiers in the Fog on harder settings. The broader interpretation is that omni-modal models can be excellent transcribers and short-term memory machines without thereby exhibiting robust strategic intelligence. This distinction is not incidental; it is the benchmark’s principal diagnostic finding.

The results also support the benchmark’s emphasis on fusion robustness rather than modality count alone. In hard settings of Whispered Pathfinding under audio conflict, Gemini 2.5 Pro’s trimmed mean steps increase from 36.2 to 133.7, indicating a major collapse in navigation efficiency under contradictory guidance. The paper interprets such failures as evidence that current models often arbitrate among modalities through fragile implicit hierarchies rather than stable cross-modal reasoning.

5. Brittle fusion, modality conflict, and the “less is more” paradox

OmniPlay’s most distinctive contribution is its diagnosis of brittle fusion mechanisms. Under modality conflict, performance degrades sharply across models, and the degradation is asymmetric: some models are relatively resilient to audio conflict but more vulnerable to textual conflict, while others display the inverse pattern. This exposes hidden modality preference hierarchies rather than genuinely synergistic fusion. The benchmark therefore treats conflict not as a corner case but as a probe of how a model actually combines evidence (Bie et al., 6 Aug 2025).

A second major result is the counter-intuitive “less is more” paradox, where removing sensory information can improve performance. In Whispered Pathfinding, MiniCPM-o-2.6 on Hard improves when vision is removed, with mean steps dropping from 110.8 to 55.0. The interpretation offered in the paper is that extra modalities are not automatically beneficial: when fusion is weak, an added modality can function as distraction, noise, or a source of misleading salience.

Noise robustness experiments reinforce the same diagnosis. In Phantom Soldiers, visual noise causes catastrophic degradation for all models; Gemini 2.5 Pro’s score drops from 78.81 to 14.2 under image noise, and Qwen-2.5-Omni collapses to zero. Audio noise is also damaging, though less severely. Modality substitution experiments add a complementary result: when audio guidance is replaced by its textual equivalent in Phantom Soldiers, performance often improves, especially for proprietary models; Gemini 2.5 Flash jumps from 31.2 to 70.2. The paper argues that the semantic content itself is not the main problem; the failure lies in how the models process and fuse the modality.

These findings directly challenge the assumption that scaling model size or adding more input channels necessarily yields more robust interactive intelligence. The benchmark instead argues for explicit research emphasis on synergistic fusion, conflict resolution, robust reasoning under noise, adaptive modality weighting, and long-horizon planning in multi-sensory environments. A plausible implication is that future progress in omni-modal agents may depend as much on architectural and training choices for arbitration as on scale alone.

6. OmniPlay in the broader omni-modal and embodied-agent landscape

The benchmark sits within a broader cluster of work on omni-modal interaction, embodied autonomy, and persistent agents. In “Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy,” OmniPlay is described as the broader ambition of omnimodal embodied play/operation, while OmniAct is presented as the concrete framework for turning that ambition into a practical system for persistent autonomy in real-world environments. OmniAct is designed to handle speech, vision, language, web/API tools, IoT devices, robot manipulation, and mobile navigation in a single long-horizon loop, using a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification (Shi et al., 25 Jun 2026).

In robotic manipulation, RoboOmni is framed as an answer to what an “OmniPlay”-style query would demand: not just task execution from an explicit command, but proactive inference of human intent from spoken dialogue, environmental sounds, and visual cues, followed by confirmation and action. Its Perceiver–Thinker–Talker–Executor architecture unifies intention recognition, interaction confirmation, and action execution, and its OmniAction dataset contains 141,162 multimodal episodes, 5,096 distinct speaker timbres, 2,482 non-verbal sound events, 640 environmental backgrounds, 112 skills, and 748 objects (Wang et al., 27 Oct 2025).

At the model level, M2-omni is directly relevant as a unified multimodal sequence modeling framework that accepts arbitrary combinations of text, image, video, and audio and generates multimodal sequences interleaving with audio, image, or text outputs. Its significance for OmniPlay-like systems lies in providing a streaming-capable, interleaved multimodal interaction paradigm rather than a text-only control interface (Guo et al., 26 Feb 2025).

A separate line of work on embedded visualization is conceptually adjacent rather than agentic. Omnioculars overlays context-specific analytics directly onto a basketball game view and has been described as a strong related system and likely foundational precursor if OmniPlay is interpreted as a target concept for augmented viewing. This usage extends the thematic range of “omni” interaction beyond game-playing agents into interactive media augmentation (Lin et al., 2022).

Taken together, these related systems indicate that OmniPlay names both a specific benchmark and, in surrounding literature, a broader orientation toward multi-sensory, interactive, and often embodied intelligence. The benchmark’s distinct contribution is to make the fusion problem measurable: not merely whether a model can see, hear, and read, but whether it can combine those channels into coherent sequential decisions when they agree, disagree, or mislead.