UmniBench: Unified Multimodal Evaluation

• UmniBench is an omni-dimensional benchmark that evaluates unified multimodal models by integrating generation, editing, and understanding tasks into a closed-loop, self-evaluative pipeline.
• It systematically covers 13 domains and over 200 concepts, yielding fine-grained, modality-agnostic performance metrics without needing external human evaluation.
• Benchmark results highlight key challenges in counterfactual reasoning and multi-step inference, underscoring the need for stronger bidirectional model coupling.

UmniBench is an omni-dimensional benchmark specifically devised for the evaluation of unified multimodal models (UMMs), which are architectures that jointly tackle multimodal understanding, generation, and editing tasks. Addressing the inadequacy of traditional, decoupled assessments that treat understanding, generation, and editing as isolated competencies, UmniBench formulates an integrated, self-evaluative paradigm where UMMs generate content and then immediately answer structured queries about their outputs. Through a systematic coverage of 13 domains and over 200 concepts, UmniBench delivers both unified and fine-grained, modality-agnostic evaluation, and has been used to benchmark 24 large-scale models spanning both unified and single-ability paradigms (Liu et al., 19 Dec 2025).

1. Rationale and Objectives

UmniBench was motivated by the emergence of UMMs whose capabilities interleave visual/language understanding, conditional generation, and multimodal editing. Earlier benchmarks (e.g., VQAv2 for visual understanding, COCO Captioning for image generation) are inherently siloed, and do not quantify the mutual dependencies or error propagation between comprehension, synthesis, and manipulation. UmniBench was thus conceived to:

• Evaluate understanding, generation, and editing within a single, closed-loop test pipeline that mirrors typical UMM usage (e.g., generate an image, edit per instruction, answer visual questions).
• Eliminate need for external scoring modules or costly, low-throughput human evaluation by leveraging the model’s own understanding capability to self-score each stage.
• Probe both model generalization and domain specialization by broad conceptual coverage across 195+ concepts.
• Enable decoupled evaluation for fine-grained ability profiling, but with a main focus on end-to-end performance.

This unified evaluation paradigm allows comprehensive inspection of model weaknesses, especially in scenarios where generation and understanding must work in concert.

2. Benchmark Architecture and Evaluation Process

The UmniBench evaluation pipeline organizes tasks as sequential triads: Generation, Interaction (editing), and Counterfactual. Each stage is built as follows:

• Model receives a prompt (p₁ for generation), synthesizes an image, and then must answer three auto-generated QA pairs regarding the output. This process is repeated twice: once after an editing interaction (p₂) and once more following a counterfactual edit (p₃).
• All QA pairs are constructed to probe (1) entity recognition, (2) attribute attribution, and (3) inference about interactions or changes—per stage, yielding nine QAs per case.

The prompts and QA pairs are generated via a pipeline coupling LLM proposal, domain-expert filtering (to ensure visual groundability and answerability), followed by automated rule-checking that enforces visual grounding and removes scenarios solvable by commonsense alone. Manual review is then performed to assure rigor.

This self-generate/self-evaluate structure ensures test set immunity to training data leakage, as all images paired to questions are synthesized ab initio by the model under evaluation.

3. Task Coverage and Scenario Taxonomy

UmniBench spans 13 major domains, each anchored by approximately 15 expert-vetted concepts. These domains encompass:

1. Spatial (e.g., “above/below,” “left/right”)
2. Plant (e.g., “leaf color change,” “fruit growth”)
3. Fluid (e.g., “liquid mixing,” “pouring”)
4. Physical (e.g., “collision,” “balance”)
5. Cooking, Arts & Crafts, Weather/Environment, Animal, Office, Personal Care, Playground, Gardening, Household

Each domain scenario is further diversified by constructing original and counterfactual entity pairs, with stages that require attribute modifications and causal edits (e.g., “change the pouring liquid’s color”). This compositional approach yields a test suite of breadth and fine-grained control, and allows domain-specific as well as holistic benchmarking of UMM abilities (Liu et al., 19 Dec 2025).

4. Evaluation Metrics

UmniBench applies a suite of complimentary metrics across modalities and tasks:

• Understanding (QA) Metrics: Accuracy, Precision, Recall, and F1, all computed directly from model answers to structured QAs (e.g., $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat y_i=y_i]$$).
• Generation Metrics: BLEU-n, ROUGE-L, and CLIPScore; the latter measures cosine similarity between text (prompt or answer) and generated image embeddings.
• Editing Metrics: Levenshtein edit distance ($\mathrm{ED}(s, t)$) between original and edited content.
• Unified Overall Score: Mean accuracy over all QA pairs across all three stages, summarized as $$\text{Score}_\text{overall} = \frac{1}{9}\sum_{j=1}^9 \mathbf{1}[\hat a_j = a_j] = \tfrac13(\mathrm{GenAve} + \mathrm{IntAve} + \mathrm{CntAve})$$.

The benchmark supports decoupled evaluation by substituting fixed state-of-the-art (SOTA) generation, editing, or understanding modules to isolate and profile each specific ability.

5. Results and Comparative Model Performance

UmniBench has been used to evaluate 24 popular models, both unified (comprising all three abilities) and single-ability. For eight leading UMMs, stage-wise accuracy is summarized in the following table:

Model	Generation	Interaction	Counterfactual	Overall
Bagel-Think	87.39	80.03	66.58	77.85
Ovis-U1	91.01	77.32	64.69	77.45
Bagel	86.54	76.47	62.95	75.14
UniPic2	85.34	79.37	60.32	74.86
GPT-4o	83.29	80.62	65.83	76.49
OmniGen2	84.84	59.66	47.34	63.56
OneCAT	83.29	54.71	45.53	60.76
Lumina-DiMOO	76.20	42.45	38.47	51.90

Performance consistently declines across stages from generation to counterfactual, with a sharp accuracy drop (ca. 20–25 points) in the counterfactual stage, highlighting the challenge of conditional reasoning and complex editing.

Domain-level analysis reveals that abstract domains such as “Spatial” and “Plant” are the most difficult (∼57–60%), with common, visually grounded activities (e.g., “Household,” 80%) significantly easier.

Single-ability models (e.g., Kimi-VL-A3B for understanding at 75.14%, Nano-Banana for editing at 73.68%) often outperform unified models in their specialized task, but underperform on integrated, multi-stage evaluation.

6. Analysis: Strengths, Weaknesses, and Recommendations

UMMs exhibit robust initial generation capabilities (often >80%) and can maintain stable editing in simple interaction tasks (e.g., GPT-4o >80% accuracy on editing QAs). However, significant deficits remain:

• Counterfactual reasoning and editing yield a pronounced accuracy decline, with most models averaging ∼60% and some models dipping below 50%.
• Abstract reasoning (Spatial, Plant domains) and multi-step inference are identified failure points.
• Specialist models can surpass generalist architectures in targeted domains, contraindicating the presence of a universally dominant UMM.

Recommended directions for improvement include stronger bidirectional coupling between understanding and generation modules, explicit augmentation with synthetic counterfactuals and multi-turn dialogues for training, and integration of geometric/symbolic reasoning (e.g., 3D spatial transformers) for domains showing persistent gaps.

7. Significance and Future Directions

UmniBench establishes a rigorous, unified standard for the holistic assessment of UMMs, aligning evaluation procedures with practical, real-world deployment scenarios where comprehension, synthesis, and editing are inseparable. Its design—grounded in self-evaluation, broad conceptual coverage, and customizable task decoupling—facilitates objective benchmarking and guides research toward performance gains in the most challenging aspects of model generalization.

Potential extensions include:

• Synthesis of richer counterfactual and multi-turn editing scenarios.
• Integration of behavioral/statistical human-likeness metrics.
• Modular ensemble pipelines combining best-in-class specialist modules within an adaptive, UMM-style framework.

Continuous benchmarking with UmniBench is expected to play a central role in closing current ability gaps and informing architectural innovation for the next generation of UMMs (Liu et al., 19 Dec 2025).