---
title: 'MMAO-Bench: MultiModal All in One Benchmark'
url: https://www.emergentmind.com/topics/multimodal-all-in-one-benchmark-mmao-bench
type: topic
---

# MMAO-Bench: MultiModal All in One Benchmark

Searching arXiv for MMAO-Bench and closely related multimodal benchmark papers to ground the article with current references.
MMAO-Bench, short for **MultiModal All in One Benchmark**, is a benchmark for evaluating **omni models**: systems that jointly process and reason over **vision, audio, and language**. It is designed to assess both **uni-modal** and **omni-modal** understanding capabilities within a single framework, with an emphasis on whether true cross-modal integration is required rather than merely available. The benchmark contains **1,880 human-curated samples** across **44 task types**, supports **I+V+A** omni-modal evaluation and **I/V/A** uni-modal evaluation, uses **MC/MO** question formats, and covers both **English and Chinese**. Its stated motivation is that existing multimodal evaluation is fragmented and that prior omni-modal benchmarks often either overstate cross-modal reasoning or have quality problems; the paper specifically reports that **77%** of questions in WorldSense are solvable without video or audio, and that **25%** of questions in OmniBench contain erroneous answers [2510.18915].

## 1. Definition, scope, and scientific motivation

MMAO-Bench is framed around a specific scientific question: how **uni-modal competence** relates to **omni-modal competence**. In the benchmark’s terminology, **uni-modal** capabilities are modality-specific skills such as image understanding, video understanding, or audio understanding assessed in isolation; **cross-modal** capabilities involve integrating information across modalities, especially alignment and recognition; and **omni-modal** capability is the broader target of preserving strong uni-modal performance while also solving tasks that require integrated reasoning over image, video, audio, and text [2510.18915].

The benchmark is therefore not only a leaderboard instrument but also a probe of capability composition. Its main claim is that fragmented evaluation obscures the relationship between modality-specific strengths and fully integrated multimodal reasoning. MMAO-Bench addresses this by combining uni-modal subsets with genuinely omni-modal tasks under one ability framework, and by enforcing stronger modality dependence through construction and ablation. The paper reports **100% question-answer accuracy** after quality control and **98% solvability by true cross-modality**, meaning that nearly all omni-modal questions require multimodal integration rather than being answerable from a single modality [2510.18915].

This design situates MMAO-Bench within the broader transition from traditional multimodal systems toward **omni models**. A plausible implication is that the benchmark is as much about diagnosing modality bottlenecks and alignment failures as about measuring raw task accuracy.

## 2. Capability framework and benchmark composition

MMAO-Bench organizes model ability into two major dimensions, **perception** and **reasoning**. The perception layer comprises **Object Perception**, **Attribute Perception**, **Scenario Perception**, **Spatial Perception**, **Cross-Modal Conversion**, **Semantic Understanding**, and **Cross-Modal Alignment**. The reasoning layer comprises **General Reasoning**, **STEM Reasoning**, **Code Reasoning**, **Spatial Reasoning**, **Temporal/Sequential Reasoning**, and **Complex Reasoning**. The paper also notes, in its future-work discussion, that harder STEM and code reasoning remain under-covered in the current release [2510.18915].

The benchmark contains **1,250 human-crafted omni-modal questions** and **630 enhanced uni-modal samples**, for a total of **1,880**. In the comparison table, MMAO-Bench is characterized as supporting **I+V+A** omni-modal evaluation, **I/V/A** uni-modal evaluation, **44 tasks**, **1,880 QA pairs**, **MC/MO** question formats, and **EN/ZH** language coverage. The bilingual scope is part of its differentiation from earlier omni-modal datasets, which the paper describes as English-centric [2510.18915].

A distinctive feature is the **Multi-Step Open-Ended Questions (MO)** format. Rather than using only independent, single-step multiple-choice items, MMAO-Bench includes a new format in which a complex problem is decomposed into multiple dependent subquestions that must be answered sequentially in open text. Each MO item is worth **10 points** total, and the reference answer allocates scores to subquestions according to their difficulty and importance. This format is intended to reveal performance decay with increasing reasoning depth and to be more discriminative among advanced models [2510.18915].

## 3. Data construction, curation, and quality control

The benchmark is built from real-world photos and videos collected primarily through **crowdsourcing**, supplemented with copyright-free websites and a small amount of public data from **MMVU**, **LongVideoBench**, and **VideoVista**. The topics span **society, culture, art, life, literature, and science**. Audio realism is treated as a first-class design concern: aside from background sounds and music, all dialogue is manually recorded by human speakers, with **more than 20 participants** contributing so that vocal characteristics better reflect real-world diversity [2510.18915].

Annotation combines **human experts** and **high-quality crowd workers**. Experts contribute task-specific professionalism and capability decomposition, while crowd workers—described as mostly college students with multimodal interaction experience and varied backgrounds—contribute realism and diversity. Annotators first select suitable materials from a tagged library, then write prompts and answers, and finally record dialogue audio manually where needed so that each QA instance is genuinely multimodal [2510.18915].

Quality control is unusually central to the benchmark’s claims. Each item undergoes at least **three rounds of independent inspection**. A preliminary model-based check filters ambiguous questions, non-unique answers, or task-type violations. The benchmark then performs **modality ablation experiments**, removing one modality at a time to test whether the question remains answerable from the remainder. Finally, human reviewers manually inspect and revise the data. This process underwrites the reported **100%** quality-check pass rate and **98%** true cross-modal solvability [2510.18915].

For the uni-modal component, the paper argues that public uni-modal benchmarks suffer from data leakage. It therefore combines self-constructed private data with a limited amount of carefully screened public data, with public supplements accounting for **11% of the total** and coming mainly from **AV-Odyssey** and **WorldSense**. Selection criteria include comprehensiveness, diversity, quality, and discriminability, and many WorldSense items that are solvable from a single modality are deliberately excluded [2510.18915].

## 4. Evaluation protocol and methodological design

MMAO-Bench’s scoring is reported primarily as accuracy-like percentages for **MC** items and aggregated point-normalized scores for **MO** items and ability groups. The paper does not provide a formal global benchmark equation. Instead, it reports task-wise and ability-group scores and decomposes results into perception versus reasoning, and within perception into **cross-modal alignment** versus **cross-modal recognition** [2510.18915].

To keep uni-modal evaluation practical, the paper introduces a dataset-compression procedure. It starts from a baseline dataset of roughly **8,000 samples** drawn from **15 open-source benchmarks** with evaluation results from **12 models**. Models are split **8/4** into training and test sets for the compression study. Each question is represented as an \(x\)-dimensional vector of model scores, then **K-Means with \(K=48\)** clusters questions into groups of similar model-performance profile, followed by hierarchical sampling proportional to cluster size. The compressed subset is evaluated using **Margin of Error (MoE)** and **Root Mean Square Error (RMSE)**, and ranking preservation is checked with **Spearman rank correlation coefficient (SRCC)** and **Pearson linear correlation coefficient (PLCC)**. The procedure is repeated with **5 random splits** and **10-fold cross-validation**. At a **10% compression rate**, the paper reports nearly **100% CI coverage**, **PLCC > 0.98**, **SRCC > 0.98**, and **RMSE = 0.06**, while reducing evaluation cost by over **90%** [2510.18915].

The tested models are current omni-capable systems that accept text, visual, and audio input simultaneously. The open-source or open-weight family includes **MiniCPM-o-2.6**, **Qwen-2.5-omni-3B**, **Qwen-2.5-omni-7B**, **Baichuan-omni-1.5**, and **Ming-lite-omni-1.5**. The closed-source family includes **Gemini-2.0-Flash**, **Gemini-2.5-Flash**, and **Gemini-2.5-Pro**. The text also mentions **Qwen-3-omni**, although it does not appear in the reported result tables. All integrations use official implementations, and for video each model receives raw video and handles frame sampling with its own native strategy [2510.18915].

## 5. Empirical findings and the proposed compositional law

The benchmark’s headline results are strongly tiered. On the summary dimensions labeled **Visual**, **Audio**, **Omni-MC**, and **Omni-MO**, **Gemini-2.5-Pro** scores **80.10**, **78.70**, **76.61**, and **68.84** respectively, while **Gemini-2.5-Flash** scores **76.64**, **75.20**, **68.23**, and **62.00**. Among open-source systems, **Qwen-2.5-omni-7B** reaches **52.81**, **68.11**, **33.23**, and **29.12**, and **MiniCPM-o-2.6** reaches **41.85**, **63.10**, **26.77**, and **21.92**. Two empirical patterns are explicit: open-source models trail Gemini models substantially on true omni-modal tasks, and **Omni-MO** is harder than **Omni-MC** for every model [2510.18915].

Ability-level analysis shows that **cross-modal alignment** is harder than **cross-modal recognition**, and that **spatial reasoning** is the hardest reasoning category. The paper reports **Gemini-2.5-Pro** at **81.02** on alignment and **83.33** on recognition, while even this model reaches only **55.05** on spatial reasoning. The best open-source model on spatial reasoning is **Baichuan-omni-1.5** at **35.78** [2510.18915].

The paper further states that **Gemini-2.5-Pro** is only **8.3%** behind humans overall. It also reports an asymmetry: humans score better on reasoning than perception (**81.3% > 74.3%**), whereas the model’s relative strength is the reverse. The authors interpret this as human-comparable perception but not yet human-level reasoning [2510.18915].

The benchmark’s most novel scientific claim is a **compositional law** between uni-modal and omni-modal performance. In the paper’s wording, “the performance of omni-modal ability is power-law related to the multiply of uni-modals' performances.” The main text does not provide an explicit fitted equation with coefficients. The qualitative picture is a two-regime structure: weaker models exhibit a **bottleneck** or **short-board effect**, in which weakness in any required modality suppresses omni-modal capability, while stronger models exhibit **synergistic promotion**, in which modalities reinforce one another and omni-modal performance rises above the simple baseline suggested by uni-modal composition [2510.18915].

The ablation studies support that interpretation. In visual ablations, weak models often benefit more from replacing raw visual input with captions than from using raw visual information directly, suggesting that extraction and integration of visual evidence remain bottlenecks. In audio ablations, some weak models improve when raw audio is replaced by ASR or sound descriptions, again indicating a bottleneck in native audio understanding. The **MO** analysis shows degradation as question depth increases for every model, but less sharply for stronger systems; this is presented as evidence that current systems acquire perception before reasoning, and that multi-step multimodal reasoning remains the frontier [2510.18915].

## 6. Position within the multimodal benchmark landscape and stated limitations

MMAO-Bench belongs to a growing family of comprehensive multimodal benchmarks, but its specific contribution is the joint evaluation of **uni-modal** and **omni-modal** capability within one bilingual, quality-controlled framework. In nearby benchmark space, **MMOU** targets long and complex real-world videos with **15,000** questions over **9,038** videos and **13** skill categories [2603.14145]; **MEGA-Bench** scales multimodal evaluation to **505** tasks and **8,186** samples with heterogeneous outputs and **45** metrics [2410.10563]; **UniM** introduces **31,026** interleaved any-to-any instances across **7** modalities and **30** domains [2603.05075]; **Uni-MMMU** focuses on the bidirectional coupling of understanding and generation across **885** instances in **8** domains [2510.13759]; and **MMBench-Live** addresses continuous benchmark evolution with **5.9K** newly generated evaluation instances [2607.01813]. This suggests that MMAO-Bench occupies a distinctive niche: it is narrower than the largest all-task suites, but more directly targeted at diagnosing how omni capability composes from uni-modal ability.

The paper also states several limitations. Despite **44 task types**, ability coverage is still incomplete, especially for harder **STEM** and **code** reasoning. The benchmark does not yet include the **private test set** that the authors identify as future work for avoiding benchmark hacking. Although the benchmark emphasizes reduced contamination and strong data quality, public-source supplementation remains. Finally, while the proposed **compositional law** is a central scientific claim, it is not formalized in the main text with a precise fitted function [2510.18915].

Taken together, these features make MMAO-Bench a benchmark for studying multimodal capability composition rather than merely multimodal task breadth. Its central contribution is not only that it evaluates image, video, audio, and language together, but that it does so in a way intended to separate uni-modal strength, cross-modal dependence, alignment difficulty, and multi-step omni-modal reasoning failure modes.

Source: https://www.emergentmind.com/topics/multimodal-all-in-one-benchmark-mmao-bench