MMAE: A Massive Multitask Audio Editing Benchmark

Abstract: We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.

• The paper introduces MMAE, the first taxonomy-driven benchmark for scalable, instruction-based audio editing across multiple modalities and complexities.
• It presents a rubric-based evaluation protocol with over 17,700 atomic rubrics across 2,000 curated samples to test instruction following and context preservation.
• Empirical results reveal that state-of-the-art models struggle with complex, mixed-modality tasks, often achieving near-zero exact match rates and significant performance gaps.

MMAE: A Comprehensive Benchmark for General-Purpose Instruction-Based Audio Editing

Motivation and Context

Recent progress in audio editing has paralleled breakthroughs in vision, shifting from atomic event manipulations to interactive, instruction-based workflows. However, evaluation protocols for audio systems lag: existing benchmarks remain narrow and fragmented by subdomain, typically only covering speech or basic sound event edits. The need for complex, open-ended, and instruction-centric benchmarks—analogous to recent multimodal and visual editing testbeds—remains unmet. The MMAE benchmark is introduced to fill this critical evaluation gap, offering the first scalable, taxonomy-driven, and rubric-based benchmark spanning the full multitask, multimodality regime in audio editing (2606.07229).

Benchmark Taxonomy and Task Coverage

MMAE’s design follows a parallel taxonomy along three orthogonal axes: modality, complexity, and operation, enabling compositional coverage of realistic, open-world audio editing use cases. The modality axis comprises seven distinct audio categories: sound, music, speech, and all their 2-way and 3-way mixtures. This fully accounts for the diverse and often mixed-modality nature of real-world audio editing tasks.

Figure 1: Distribution of audio modalities covered in the MMAE benchmark, illustrating comprehensive coverage of isolated and mixed audio domains.

Task complexity is stratified into six levels, from basic single operations and multi-part edits to multi-instruction, multi-audio, multi-round, and multi-hop scenarios. This stratification enables dynamic stress-testing of model reasoning, memory, and compositional capabilities, well beyond simplistic transformations. The operation axis spans both local and global transformations, with eight action types including addition, removal, replacement, extraction, alteration, background/foreground change, and global shifts.

Figure 2: Representative samples spanning MMAE’s taxonomy and the rubric-based evaluation protocol. Tasks range from simple single-modality edits to complex multi-round or compositional scenarios.

Rubric-Based Evaluation Paradigm

MMAE establishes a rubric-based evaluation protocol, extending recent advances in reward modeling outside verifiable domains. Each benchmark sample is annotated with a tailored set of atomic, independent rubrics—over 17,700 in total—enabling precise discrimination of model capabilities across instruction following and context consistency. All rubrics adhere to principles of completeness, atomicity, orthogonality, and objectivity, minimizing ambiguity and maximizing interpretability. A state-of-the-art multimodal LLM serves as an external judge, performing triple-check voting and randomized query shuffling to mitigate bias.

The evaluation measures:

• Instruction Following Rate (IFR): assessing faithfulness to user instructions,
• Consistency Rate (CR): strict preservation of unrelated content,
• Exact Match Rate (EMR): global perfection, i.e., all rubrics satisfied for a sample.

This protocol explicitly partitions execution fidelity and preservation, avoiding the conflation typical of prior coarse metrics like signal-level measures or mean opinion scores.

Data Curation Methodology

MMAE's data pipeline combines expert-driven brainstorming, systematic taxonomy formulation, and dynamic balancing, followed by human-agent collaborative annotation using agent-powered semantic caption extraction (e.g., Omni-Detective) and LLM-based rubric draft generation with iterative human refinement.

Figure 3: The data curation pipeline integrates scenario elicitation, targeted taxonomy balancing, human-agent rubric drafting, and strict multi-stage quality inspections.

A multi-layer manual inspection protocol guarantees fidelity at every stage. The final release comprises 2,000 unique high-fidelity samples, each sampled from diverse real-world scenarios, annotated for both instructions and multidimensional task metadata.

Figure 4: An example of the professional data annotation and review platform employed, supporting scalable structured curation with version control.

Empirical Evaluation and Analysis

Evaluation of five state-of-the-art instruction-based audio editing models—Step-Audio-EditX, Ming-UniAudio, MMEdit, Audio-Omni, and SmartDJ—exposes clear and systemic bottlenecks:

• Exact Match Rates remain persistently below 5% and drop to absolute 0% for complex mixed-modality tasks, demonstrating that current methods systematically fail to achieve the holistic precision required for real-world editing demands.
• Average IFR and CR are modest at best (e.g., Step-Audio-EditX: 44.86%/58.88%; Audio-Omni: 50.73%/56.93% on ≤10 s subset), but these mask a wide dispersion and do not imply reliable success at the sample level.
• A fundamental trade-off between IFR and CR emerges. Baselines like the Identity model “win” on CR yet fail instruction following, confirming that current architectures cannot simultaneously guarantee both precision and context preservation.
• Higher task complexity and increased modality compositionality consistently degrade performance across all models. Segregated analysis reveals that cross-modality, multi-step, and multi-audio operations are severe failure points.
• The correlation between average metric improvement and holistic success is weak (i.e., higher IFR/CR does not directly translate to higher EMR), signifying a divergence akin to mean-seeking vs. mode-seeking phenomena in generative modeling.
• Introduction of agentic planners (as with SmartDJ) does not yield consistent improvements and may even reduce context fidelity, indicating fundamental robustness limitations in current model architectures for compositional or sequential edits.

Implications and Future Directions

The MMAE benchmark exposes clear practical and theoretical limitations of current multitask audio editing architectures. The empirical results indicate an urgent need for:

• Architectures with robust support for multimodal and compositional task understanding,
• Models that can maintain strict context preservation while executing complex, open-ended instructions,
• Training protocols that directly optimize for exact match rates, not just mean per-rubric performance,
• Advances in agentic or planner-augmented editing that consider error propagation and cascading failure in iterative editing scenarios.

By fixing a rigorous, scalable evaluation protocol, MMAE facilitates fine-grained diagnosis and progress tracking in audio editing research. It anticipates future developments such as unified, modality-agnostic editing agents, robust atomic edit models, and compositional generalization frameworks analogous to those found in advanced visual and multimodal editing.

Conclusion

MMAE stands as the first large-scale, systematic benchmark enabling rigorous evaluation of instruction-based, general-purpose audio editing across multimodal, compositional, and realistic use cases. The rubric-based protocol provides objective, local, and global criteria for assessing editing performance. Empirical evidence from evaluating leading models reveals low reliability, with exact match rates near zero for challenging tasks, highlighting profound capability gaps. MMAE establishes clear targets and diagnostic tools, laying the foundation for the next generation of universal, robust, multimodal audio editing systems.