OmniBench: Multimodal AI Benchmark Suite

• OmniBench is a comprehensive suite featuring distinct benchmarks, datasets, and protocols for testing multimodal reasoning, virtual agent planning, RAG systems, and video editing.
• It employs rigorous metrics such as accuracy, coverage rate, and logical consistency to offer standardized, reproducible evaluations across various AI domains.
• The suite drives practical insights into tri-modal fusion challenges, retrieval alignment, and compositional task structures, highlighting both achievements and current limitations.

OmniBench is a suite of distinct benchmarks, datasets, and evaluation platforms for assessing advanced capabilities of artificial intelligence systems across multiple domains and modalities. The term includes several published works, each contributing to a different aspect of universal benchmarking for multimodal models, virtual agents, retrieval-augmented generation, code reasoning, and biological algorithms. Each OmniBench instance establishes standardized, reproducible protocols to measure reasoning, instruction following, and practical effectiveness in challenging, real-world scenarios.

1. Taxonomy of OmniBench Works and Benchmarking Scope

There are multiple research artifacts titled "OmniBench," each independently addressing a specific benchmarking challenge:

• OmniBench (OLMs, tri-modal reasoning): Designed to assess simultaneously integrated reasoning over visual, acoustic, and textual inputs for Omni-LLMs (OLMs) (Li et al., 23 Sep 2024).
• OmniBench (Virtual Agent Capabilities): A scalable, graph-based multidimensional benchmark for evaluating essential capabilities of virtual agents, including complex planning and multi-app decision-making (Bu et al., 10 Jun 2025).
• OmniBench-RAG: Provides a multi-domain evaluation platform for Retrieval-Augmented Generation (RAG) systems, quantifying accuracy and efficiency trade-offs over nine knowledge domains (Liang et al., 26 Jul 2025).
• OmniGenBench: Evaluates omnipotent multimodal generation across 57 sub-tasks via rigorous dual-mode protocols (Wang et al., 24 May 2025).
• Omnibenchmark: Continuous benchmarking infrastructure for bioinformatics, focused on reproducibility via modular workflows and FAIR data handling (Mallona et al., 25 Sep 2024).
• OmniBench-99 (Video Editing): Assesses text-guided video editing across multiple editing types and scenarios (Chen et al., 3 Dec 2024).

Each work establishes unique benchmark tasks, metrics, and evaluation datasets tailored to the modality and agent type under consideration.

2. Benchmark Construction Principles and Methodological Features

Tri-Modal OLM Benchmark (Li et al., 23 Sep 2024)

• Requires recognition, interpretation, and reasoning over image, audio, and text inputs.
• QA samples demand integrated use of all modalities: removing any modality reduces accuracy; models with substituted textual alternatives (image captions or audio transcripts) perform substantially better, indicating weak cross-modal fusion.
• Tasks span spatial, causal, and abstract reasoning categories.
• High-quality human annotations verified through multi-stage quality control and supported by ablation using vision-LLMs (e.g., LLaVA-1.6-34B).

Virtual Agent Benchmark (Bu et al., 10 Jun 2025)

• Uses directed acyclic graphs (DAGs) to represent tasks composed of multiple subtasks with explicit dependencies.
• Five dimensions of complexity (dependency, instruction, knowledge, hierarchy, branch) categorically control scenario difficulty.
• Tasks synthesized via bottom-up automated pipelines leveraging advanced MLLMs for subtask discovery and validation.
• OmniEval, the associated evaluation framework, provides subtask-level, graph-centric metrics: coverage rate (CR) and logical consistency (LC), with strong correlation to human expert judgment (Pearson r ≈ 0.95 for CR, 0.93 for LC).

Retrieval-Augmented Generation (Liang et al., 26 Jul 2025)

• Benchmarks RAG models on nine domains (culture, geography, history, health, mathematics, nature, people, society, technology).
• Employs standardized metrics:
  • Improvements quantifies absolute accuracy gains: $$\text{Improvements} = S_{\text{RAG}} - S_{\text{base}}$$.
  • Transformation aggregates normalized efficiency ratios: $$\text{Transformation} = w_\text{time} r_\text{time} + w_\text{gpu} r_\text{gpu} + w_\text{mem} r_\text{mem}$$; overheads and gains are tracked component-wise.
• Dynamic test case generation via logic programming enables robust and reproducible task diversity.

Video Editing Benchmark (Chen et al., 3 Dec 2024)

• OmniBench-99 covers 99 diverse videos, systematically annotated for four edit types and eight real-world editing scenarios.
• Evaluation includes both automatic metrics (CLIP frame consistency, PickScore) and rigorous human scoring (alignment, temporal consistency, structural coherence, overall quality).
• Separates performance by editing type versus scenario, revealing differential capacity for fine-grained edits not captured by prior benchmarks.

3. Core Evaluation Metrics and Statistical Protocols

All OmniBench variants employ carefully constructed metrics to ensure statistical validity and enable fine-grained model assessment:

• Accuracy (tri-modal QA): $$\text{Accuracy} = \frac{\# \text{Correct Answers}}{\# \text{Total Questions}}$$
• Coverage Rate (virtual agent graph): $w(s_i) = \frac{d(s_i)}{\sum_j d(s_j)}$; $$\text{CR} = \frac{\sum_i w(s_i) I(s_i)}{\sum_i w(s_i)}$$
• Logical Consistency (agent planning): $\text{LC} = \frac{CS_\text{agent}}{CS_\text{max}}$
• Improvements/Transformation (RAG): Direct comparison of pre/post RAG metrics across accuracy and resource dimensions.

Benchmarks typically report not just point estimates but full distributions, outlier analysis, and ablation under various modality/dataset constraints.

4. Representative Results and Discoveries

Tri-Modal Reasoning (OLMs, (Li et al., 23 Sep 2024))

• Open-source OLMs achieve 18%–38% accuracy; best proprietary model reaches 42.9%.
• Models with substituted textual alternatives (caption or transcript) achieve higher accuracy (up to 60%)—this suggests genuine cross-modal fusion is limited; textual modes are still dominant.

Virtual Agent Capabilities (Bu et al., 10 Jun 2025)

• Even large, closed-source models (GPT-4o) are at 20.5% CR on graph tasks versus 80.1% for humans.
• Subtask identification and long instruction following are limiting factors; model performance drops by ~6 points for harder scenarios.
• Agents fine-tuned on compositional, graph-structured data outperform those trained on manually-annotated single-trajectory datasets.

Retrieval-Augmented Generation (Liang et al., 26 Jul 2025)

• Improvements in accuracy range from +17% (culture) to −25% (math); Transformation shows RAG introduces efficiency overhead for most domains except math (where misalignment reduces reasoning burden at the cost of accuracy).
• The effectiveness of RAG is tightly bound to domain materials: A plausible implication is that resource curation and retrieval structuring are critical for reliable gains.

5. Technical and Practical Implications for Model Training and Design

The benchmarks collectively expose foundational challenges in next-generation multimodal modeling:

• Tri-modal fusion remains unsolved; models trained on paired image-text or audio-text data generalize poorly in integrated settings.
• Compositional task graphs dramatically improve agent planning and robustness, but require automated, multi-level evaluation frameworks.
• Retrieval alignment is essential; generic, chunk-based approaches are insufficient for domains requiring symbolic or rule-based reasoning.
• Universal video editing models must account not only for style-type edits but also diverse real-world scenarios; benchmarks reveal capacity gaps in current methods.
• A plausible implication is that architecture, training data, and benchmarking co-evolve: advances in evaluation infrastructures such as OmniBench drive both upstream model improvements and the development of more complex, realistic test suites.

6. Limitations, Open Problems, and Future Directions

Reported limitations include:

• Bias toward text modalities given training resource scarcity in audio/image/video curation.
• Scaling issues in computation for automated test generation, multi-domain profiling, and full-system evaluation.
• Insufficient granularity in resource metrics (per-step energy, latency); future benchmarks will require deeper instrumentation.
• For RAG, health and math domains demonstrate substantial negative gains, indicating a need for hybrid or domain-specific retrieval frameworks.

Future research, as suggested by the authors, should prioritize:

• Large-scale, balanced, and diverse tri-modal datasets.
• Advancements in cross-modal encoder-decoder architectures for robust, generalized fusion.
• Plug-and-play integration of agent-specific intent, compositional reasoning, and scenario-aware pipelines.
• Community-contributed test data and open-source tools to broaden both benchmarking and model development.

7. Summary Table: OmniBench Instances

Variant	Purpose/Domain	Principal Metric(s)
OmniBench (OLM)	Tri-modal reasoning (image/audio/text)	Multimodal QA accuracy
OmniBench (Agent)	Virtual agent capabilities (DAGs)	CR, LC across subtasks
OmniBench-RAG	RAG effectiveness (9 domains)	Improvements, Transformation
OmniGenBench	Multimodal generation (57 tasks)	OmniScore (consistency, realism, aesthetics)
Omnibenchmark (Bio)	Bioinformatics benchmarking	F1, ARI, modular DAG-routed metrics
OmniBench-99	Video editing (types + scenarios)	CLIP/frame, PickScore, human MOS

Each instance of OmniBench advances the rigor and breadth of AI benchmarking, driving deeper understanding and more robust model comparison in increasingly multimodal, scenario-rich, and domain-diverse environments.