MMAO-Bench: MultiModal All in One Benchmark

• The paper introduces MMAO-Bench as a comprehensive benchmark assessing both uni-modal and omni-modal reasoning across vision, audio, and language.
• It employs 1,880 human-curated samples spanning 44 tasks with rigorous quality control to ensure genuine cross-modal dependency.
• MMAO-Bench reveals a compositional law where omni-modal performance scales as the product of uni-modal strengths.

Searching arXiv for MMAO-Bench and closely related multimodal benchmark papers to ground the article with current references. MMAO-Bench, short for MultiModal All in One Benchmark, is a benchmark for evaluating omni models: systems that jointly process and reason over vision, audio, and language. It is designed to assess both uni-modal and omni-modal understanding capabilities within a single framework, with an emphasis on whether true cross-modal integration is required rather than merely available. The benchmark contains 1,880 human-curated samples across 44 task types, supports I+V+A omni-modal evaluation and I/V/A uni-modal evaluation, uses MC/MO question formats, and covers both English and Chinese. Its stated motivation is that existing multimodal evaluation is fragmented and that prior omni-modal benchmarks often either overstate cross-modal reasoning or have quality problems; the paper specifically reports that 77% of questions in WorldSense are solvable without video or audio, and that 25% of questions in OmniBench contain erroneous answers (Chen et al., 21 Oct 2025).

1. Definition, scope, and scientific motivation

MMAO-Bench is framed around a specific scientific question: how uni-modal competence relates to omni-modal competence. In the benchmark’s terminology, uni-modal capabilities are modality-specific skills such as image understanding, video understanding, or audio understanding assessed in isolation; cross-modal capabilities involve integrating information across modalities, especially alignment and recognition; and omni-modal capability is the broader target of preserving strong uni-modal performance while also solving tasks that require integrated reasoning over image, video, audio, and text (Chen et al., 21 Oct 2025).

The benchmark is therefore not only a leaderboard instrument but also a probe of capability composition. Its main claim is that fragmented evaluation obscures the relationship between modality-specific strengths and fully integrated multimodal reasoning. MMAO-Bench addresses this by combining uni-modal subsets with genuinely omni-modal tasks under one ability framework, and by enforcing stronger modality dependence through construction and ablation. The paper reports 100% question-answer accuracy after quality control and 98% solvability by true cross-modality, meaning that nearly all omni-modal questions require multimodal integration rather than being answerable from a single modality (Chen et al., 21 Oct 2025).

This design situates MMAO-Bench within the broader transition from traditional multimodal systems toward omni models. A plausible implication is that the benchmark is as much about diagnosing modality bottlenecks and alignment failures as about measuring raw task accuracy.

2. Capability framework and benchmark composition

MMAO-Bench organizes model ability into two major dimensions, perception and reasoning. The perception layer comprises Object Perception, Attribute Perception, Scenario Perception, Spatial Perception, Cross-Modal Conversion, Semantic Understanding, and Cross-Modal Alignment. The reasoning layer comprises General Reasoning, STEM Reasoning, Code Reasoning, Spatial Reasoning, Temporal/Sequential Reasoning, and Complex Reasoning. The paper also notes, in its future-work discussion, that harder STEM and code reasoning remain under-covered in the current release (Chen et al., 21 Oct 2025).

The benchmark contains 1,250 human-crafted omni-modal questions and 630 enhanced uni-modal samples, for a total of 1,880. In the comparison table, MMAO-Bench is characterized as supporting I+V+A omni-modal evaluation, I/V/A uni-modal evaluation, 44 tasks, 1,880 QA pairs, MC/MO question formats, and EN/ZH language coverage. The bilingual scope is part of its differentiation from earlier omni-modal datasets, which the paper describes as English-centric (Chen et al., 21 Oct 2025).

A distinctive feature is the Multi-Step Open-Ended Questions (MO) format. Rather than using only independent, single-step multiple-choice items, MMAO-Bench includes a new format in which a complex problem is decomposed into multiple dependent subquestions that must be answered sequentially in open text. Each MO item is worth 10 points total, and the reference answer allocates scores to subquestions according to their difficulty and importance. This format is intended to reveal performance decay with increasing reasoning depth and to be more discriminative among advanced models (Chen et al., 21 Oct 2025).

3. Data construction, curation, and quality control

The benchmark is built from real-world photos and videos collected primarily through crowdsourcing, supplemented with copyright-free websites and a small amount of public data from MMVU, LongVideoBench, and VideoVista. The topics span society, culture, art, life, literature, and science. Audio realism is treated as a first-class design concern: aside from background sounds and music, all dialogue is manually recorded by human speakers, with more than 20 participants contributing so that vocal characteristics better reflect real-world diversity (Chen et al., 21 Oct 2025).

Annotation combines human experts and high-quality crowd workers. Experts contribute task-specific professionalism and capability decomposition, while crowd workers—described as mostly college students with multimodal interaction experience and varied backgrounds—contribute realism and diversity. Annotators first select suitable materials from a tagged library, then write prompts and answers, and finally record dialogue audio manually where needed so that each QA instance is genuinely multimodal (Chen et al., 21 Oct 2025).

Quality control is unusually central to the benchmark’s claims. Each item undergoes at least three rounds of independent inspection. A preliminary model-based check filters ambiguous questions, non-unique answers, or task-type violations. The benchmark then performs modality ablation experiments, removing one modality at a time to test whether the question remains answerable from the remainder. Finally, human reviewers manually inspect and revise the data. This process underwrites the reported 100% quality-check pass rate and 98% true cross-modal solvability (Chen et al., 21 Oct 2025).

For the uni-modal component, the paper argues that public uni-modal benchmarks suffer from data leakage. It therefore combines self-constructed private data with a limited amount of carefully screened public data, with public supplements accounting for 11% of the total and coming mainly from AV-Odyssey and WorldSense. Selection criteria include comprehensiveness, diversity, quality, and discriminability, and many WorldSense items that are solvable from a single modality are deliberately excluded (Chen et al., 21 Oct 2025).

4. Evaluation protocol and methodological design

MMAO-Bench’s scoring is reported primarily as accuracy-like percentages for MC items and aggregated point-normalized scores for MO items and ability groups. The paper does not provide a formal global benchmark equation. Instead, it reports task-wise and ability-group scores and decomposes results into perception versus reasoning, and within perception into cross-modal alignment versus cross-modal recognition (Chen et al., 21 Oct 2025).

To keep uni-modal evaluation practical, the paper introduces a dataset-compression procedure. It starts from a baseline dataset of roughly 8,000 samples drawn from 15 open-source benchmarks with evaluation results from 12 models. Models are split 8/4 into training and test sets for the compression study. Each question is represented as an $x$-dimensional vector of model scores, then K-Means with $K=48$ clusters questions into groups of similar model-performance profile, followed by hierarchical sampling proportional to cluster size. The compressed subset is evaluated using Margin of Error (MoE) and Root Mean Square Error (RMSE), and ranking preservation is checked with Spearman rank correlation coefficient (SRCC) and Pearson linear correlation coefficient (PLCC). The procedure is repeated with 5 random splits and 10-fold cross-validation. At a 10% compression rate, the paper reports nearly 100% CI coverage, PLCC > 0.98, SRCC > 0.98, and RMSE = 0.06, while reducing evaluation cost by over 90% (Chen et al., 21 Oct 2025).

The tested models are current omni-capable systems that accept text, visual, and audio input simultaneously. The open-source or open-weight family includes MiniCPM-o-2.6, Qwen-2.5-omni-3B, Qwen-2.5-omni-7B, Baichuan-omni-1.5, and Ming-lite-omni-1.5. The closed-source family includes Gemini-2.0-Flash, Gemini-2.5-Flash, and Gemini-2.5-Pro. The text also mentions Qwen-3-omni, although it does not appear in the reported result tables. All integrations use official implementations, and for video each model receives raw video and handles frame sampling with its own native strategy (Chen et al., 21 Oct 2025).

5. Empirical findings and the proposed compositional law

The benchmark’s headline results are strongly tiered. On the summary dimensions labeled Visual, Audio, Omni-MC, and Omni-MO, Gemini-2.5-Pro scores 80.10, 78.70, 76.61, and 68.84 respectively, while Gemini-2.5-Flash scores 76.64, 75.20, 68.23, and 62.00. Among open-source systems, Qwen-2.5-omni-7B reaches 52.81, 68.11, 33.23, and 29.12, and MiniCPM-o-2.6 reaches 41.85, 63.10, 26.77, and 21.92. Two empirical patterns are explicit: open-source models trail Gemini models substantially on true omni-modal tasks, and Omni-MO is harder than Omni-MC for every model (Chen et al., 21 Oct 2025).

Ability-level analysis shows that cross-modal alignment is harder than cross-modal recognition, and that spatial reasoning is the hardest reasoning category. The paper reports Gemini-2.5-Pro at 81.02 on alignment and 83.33 on recognition, while even this model reaches only 55.05 on spatial reasoning. The best open-source model on spatial reasoning is Baichuan-omni-1.5 at 35.78 (Chen et al., 21 Oct 2025).

The paper further states that Gemini-2.5-Pro is only 8.3% behind humans overall. It also reports an asymmetry: humans score better on reasoning than perception (81.3% > 74.3%), whereas the model’s relative strength is the reverse. The authors interpret this as human-comparable perception but not yet human-level reasoning (Chen et al., 21 Oct 2025).

The benchmark’s most novel scientific claim is a compositional law between uni-modal and omni-modal performance. In the paper’s wording, “the performance of omni-modal ability is power-law related to the multiply of uni-modals' performances.” The main text does not provide an explicit fitted equation with coefficients. The qualitative picture is a two-regime structure: weaker models exhibit a bottleneck or short-board effect, in which weakness in any required modality suppresses omni-modal capability, while stronger models exhibit synergistic promotion, in which modalities reinforce one another and omni-modal performance rises above the simple baseline suggested by uni-modal composition (Chen et al., 21 Oct 2025).

The ablation studies support that interpretation. In visual ablations, weak models often benefit more from replacing raw visual input with captions than from using raw visual information directly, suggesting that extraction and integration of visual evidence remain bottlenecks. In audio ablations, some weak models improve when raw audio is replaced by ASR or sound descriptions, again indicating a bottleneck in native audio understanding. The MO analysis shows degradation as question depth increases for every model, but less sharply for stronger systems; this is presented as evidence that current systems acquire perception before reasoning, and that multi-step multimodal reasoning remains the frontier (Chen et al., 21 Oct 2025).

6. Position within the multimodal benchmark landscape and stated limitations

MMAO-Bench belongs to a growing family of comprehensive multimodal benchmarks, but its specific contribution is the joint evaluation of uni-modal and omni-modal capability within one bilingual, quality-controlled framework. In nearby benchmark space, MMOU targets long and complex real-world videos with 15,000 questions over 9,038 videos and 13 skill categories (Goel et al., 14 Mar 2026); MEGA-Bench scales multimodal evaluation to 505 tasks and 8,186 samples with heterogeneous outputs and 45 metrics (Chen et al., 2024); UniM introduces 31,026 interleaved any-to-any instances across 7 modalities and 30 domains (Li et al., 5 Mar 2026); Uni-MMMU focuses on the bidirectional coupling of understanding and generation across 885 instances in 8 domains (Zou et al., 15 Oct 2025); and MMBench-Live addresses continuous benchmark evolution with 5.9K newly generated evaluation instances (Liu et al., 2 Jul 2026). This suggests that MMAO-Bench occupies a distinctive niche: it is narrower than the largest all-task suites, but more directly targeted at diagnosing how omni capability composes from uni-modal ability.

The paper also states several limitations. Despite 44 task types, ability coverage is still incomplete, especially for harder STEM and code reasoning. The benchmark does not yet include the private test set that the authors identify as future work for avoiding benchmark hacking. Although the benchmark emphasizes reduced contamination and strong data quality, public-source supplementation remains. Finally, while the proposed compositional law is a central scientific claim, it is not formalized in the main text with a precise fitted function (Chen et al., 21 Oct 2025).

Taken together, these features make MMAO-Bench a benchmark for studying multimodal capability composition rather than merely multimodal task breadth. Its central contribution is not only that it evaluates image, video, audio, and language together, but that it does so in a way intended to separate uni-modal strength, cross-modal dependence, alignment difficulty, and multi-step omni-modal reasoning failure modes.