UEval: A Benchmark for Unified Multimodal Generation

Abstract: We introduce UEval, a benchmark to evaluate unified models, i.e., models capable of generating both images and text. UEval comprises 1,000 expert-curated questions that require both images and text in the model output, sourced from 8 real-world tasks. Our curated questions cover a wide range of reasoning types, from step-by-step guides to textbook explanations. Evaluating open-ended multimodal generation is non-trivial, as simple LLM-as-a-judge methods can miss the subtleties. Different from previous works that rely on multimodal LLMs (MLLMs) to rate image quality or text accuracy, we design a rubric-based scoring system in UEval. For each question, reference images and text answers are provided to a MLLM to generate an initial rubric, consisting of multiple evaluation criteria, and human experts then refine and validate these rubrics. In total, UEval contains 10,417 validated rubric criteria, enabling scalable and fine-grained automatic scoring. UEval is challenging for current unified models: GPT-5-Thinking scores only 66.4 out of 100, while the best open-source model reaches merely 49.1. We observe that reasoning models often outperform non-reasoning ones, and transferring reasoning traces from a reasoning model to a non-reasoning model significantly narrows the gap. This suggests that reasoning may be important for tasks requiring complex multimodal understanding and generation.

• The paper introduces UEval as a benchmark addressing unified multimodal generation by integrating image and text synthesis evaluation in real-world scenarios.
• It employs 1,000 expert-curated tasks with a detailed, rubric-based protocol that separates image quality and textual accuracy for fine-grained assessment.
• Empirical results show that reasoning models significantly outperform non-reasoning and open-source models, underscoring the value of chain-of-thought prompting.

UEval: A Benchmark for Unified Multimodal Generation

Motivation and Limitations of Existing Approaches

Unified multimodal models are intended to seamlessly generate both images and text within a single system. However, the field's evaluation paradigms have stagnated around two core tasks: visual question answering (VQA), which produces text from images, and text-to-image (T2I) generation, which produces images from text. These paradigms are fundamentally insufficient for capturing unified multimodal reasoning, where a real-world scenario frequently demands both visual synthesis and textual explanation in an intertwined, context-dependent manner.

Conventional evaluation protocols, including LLM-as-a-judge metrics, either rate image quality or textual correctness in isolation and do not provide reliable, fine-grained assessments for complex, open-ended multimodal generation. Furthermore, benchmarks that target unified generation remain scarce and lack standardized, scalable evaluation protocols that reflect the richness and difficulty of real-world multimodal reasoning.

Benchmark Design and Construction

UEval addresses these deficiencies with a comprehensive benchmark that targets unified multimodal generation. The benchmark comprises 1,000 expert-curated questions across 8 heterogeneous, real-world tasks, distinctively requiring outputs that combine both images and text. The task suite is split into closed-ended (space, textbook, diagram, paper) and open-ended (art, life, tech, exercise) categories, covering scenarios from architectural explanations to stepwise procedural guides and creative artwork generation.

Each sample consists of a query and corresponding reference answers—both text and images—and is associated with a customized, fine-grained grading rubric. Rubric criteria are initially generated by a frontier multimodal LLM (Gemini-2.5-Pro), then revised and validated through human annotation. This process yields 10,417 unique criteria, ensuring precise, sample-specific evaluation and circumventing the limitations of generic, static metrics.

Figure 1: UEval requires models to reason across modalities and generate both images and text, in contrast to previous benchmarks focusing on a single modality per task.

Evaluation Protocol: Rubric-Based, Automated, and Human-Validated

The evaluation framework hinges on robust, data-dependent rubrics. For every sample, the rubric explicitly splits between criteria evaluating image quality (e.g., relevance, accuracy, labeling, temporal consistency for multi-step tasks) and textual quality (e.g., completeness, technical accuracy, coherence, and alignment). The rubric generation pipeline involves a primary LLM-based draft, iterative consolidation by human annotators, and final quality control by independent reviewers to ensure all criteria are unambiguous, comprehensive, and faithful to the reference answers.

An MLLM, Gemini-2.5-Pro by default, automatically judges model outputs against the rubric. This process is reproducible, scalable, and strongly correlated with human evaluation (Pearson $r=0.88$), as verified across a sample of tasks:

Figure 2: Rubric drafting and response evaluation pipeline for UEval, encompassing both model-assisted rubric proposal and comprehensive human validation.

Figure 3: Examples of rubric items for closed-ended and open-ended UEval tasks, highlighting the shift from purely factual assessment to higher-order alignment and process consistency.

Empirical Study: Model Comparison and Diagnostic Insights

UEval benchmarks 9 leading unified models—open-source (Janus-Pro, Show-o2, MMaDA, BAGEL, Emu3.5) and proprietary frontier (Gemini-2.0-Flash, Gemini-2.5-Flash, GPT-5-Instant, GPT-5-Thinking)—across all 8 tasks.

Key Results

• Frontier models substantially outperform open-source models. The highest score, achieved by GPT-5-Thinking, reaches 66.4/100 while the top open-source model (Emu3.5) attains only 49.1/100.
• Model performance varies distinctly between tasks. Knowledge-intensive domains (textbook, diagram) are less challenging compared to procedural and creative tasks (art, life), with the latter exhibiting large failure modes such as label inconsistencies or temporal disjunctions in visual guides.

Figure 4: UEval scores across tasks, with consistent superiority of GPT-5-Thinking and other proprietary models over the open-source cohort.

• Reasoning models have a marked advantage. Models that natively generate reasoning traces (e.g., GPT-5-Thinking) consistently surpass non-reasoning variants (e.g., GPT-5-Instant), especially in complex, open-ended or stepwise scenarios.

Detailed Analysis

• Rubric-based scores expose modality-specific bottlenecks: Even the strongest unified models lag significantly in image generation versus text generation (e.g., GPT-5-Thinking image score: 49.1, text score: 83.8), indicating room for architectural or training improvements in visual synthesis pipelines.
• Judge reliability is crucial: Judge models such as Qwen3-VL-235B-Thinking and Gemini-2.5-Pro deliver mutually consistent scores with GPT-5-Thinking, whereas weaker or differently aligned models yield divergent and less reliable outcomes.
• Human-LMM agreement: Fine-grained criterion-wise evaluation demonstrates ~90% rubric agreement between automated and human judges across sampled outputs.

Reasoning Transfer and Chain-of-Thought Effects

A salient experimental result involves reasoning trace transfer: by extracting the reasoning steps generated by a high-performing reasoning model (GPT-5-Thinking) and appending them to the prompt of a non-reasoning model, visual generation quality improves considerably for strong models like GPT-5-Instant and Gemini-2.5-Flash. However, weaker models such as BAGEL are unable to leverage these cues, suggesting that sufficient generative capacity is a precondition for exploiting externally supplied reasoning signals. This finding empirically demonstrates that Chain-of-Thought (CoT) prompting is beneficial not only for text but also for image-centric tasks in unified systems.

Figure 5: Transferred reasoning traces from GPT-5-Thinking significantly enhance the multimodal outputs of non-reasoning models, but only for models with strong generation capacity.

Benchmark Implications, Theoretical and Practical

UEval sets a new standard for multimodal evaluation by (1) requiring unified, contextually appropriate image-text generation; (2) scaling fine-grained, data-dependent rubric construction to 1,000 real-world samples; and (3) enabling reproducible, automated grading with frontier LLMs while maintaining high alignment with expert human judgment. The benchmark directly exposes the limitations of modern unified generation models, notably their struggles with visual consistency, multi-step planning, and high-level image-text alignment, particularly in open-ended tasks.

Practically, UEval provides a rigorous testbed for rapid benchmarking of novel unified models, and its rubric paradigm is readily extensible to new tasks and modalities. The experiments further validate the utility of advanced LLMs as standardized evaluators for complex, multimodal content—a vital step for large-scale, realistic model assessment.

Theoretically, the results reinforce a central hypothesis: explicit reasoning, through CoT or similar mechanisms, is integral to unified multimodal generation, notably in scenarios demanding causal, compositional, or stepwise visual synthesis.

Conclusion

UEval introduces a scalable, rigorously human-validated benchmark for unified multimodal generation. The results demonstrate the current robustness gap between proprietary and open-source models and highlight reasoning as a critical component for effective multimodal synthesis. As the field progresses toward unified generative paradigm in AI, UEval establishes a solid foundation for both method development and evaluation, motivating future directions including improved visual reasoning, consistency-aware generation architectures, richer benchmarking for other modalities, and ethical, robust judge modeling.