- The paper introduces IMUG-Bench, a 3,113-sample benchmark spanning 12,034 turns, 19 domains, and 97 tasks to test interleaved multimodal understanding and image generation.
- The evaluation finds that closed-source models lead open-source systems, while dynamic questions expose major weaknesses; GPT-5 Image Mini drops from 83.5 to 50.6 between static and dynamic understanding.
- The paper shows that image-generation scores decline sharply across turns because of exposure bias, while Chain-of-Thought planning improves BAGEL’s average image score from 54.1 to 65.9.
IMUG-Bench is a benchmark designed to evaluate unified multimodal models (UMMs) on multi-turn, interleaved image–text dialogues that jointly require understanding and generation. The paper addresses a concrete gap: existing unified benchmarks are largely single-turn or static, and none explicitly measures exposure bias—the degradation that occurs when models condition on their own imperfect outputs across turns. The benchmark comprises 3,113 samples and 12,034 interaction turns spanning 19 domains and 97 tasks, organized into three classes (Static Spatial, Temporal Causal, Hybrid), and introduces dynamic understanding questions whose ground-truth answers depend on images the model itself generated earlier.
Benchmark design and construction
Each task instance is a multi-turn interaction of 2 to 6 turns (average 3.87), where every turn is one of three question types. Static-MCQ evaluates understanding under stable multimodal context with fixed answers; Dynamic-MCQ evaluates the model's ability to understand its own previously generated images, with ground truth determined by a judge VLM only after the model has answered; Image-Generation evaluates instruction following, visual consistency, and multi-turn editing. Construction follows a pipeline of manual template design, LLM/VLM-based filling (using DeepSeek-V3.2 for text-only questions and Qwen3-VL-235B-A22B-Thinking for image-input questions), and two-person human verification at every stage. Images come from public platforms (Pexels, Pixabay, Openverse, Flickr, Unsplash, Wikimedia, CloudAtlas), targeted generation with image models, and real-world capture or professional content creation, all manually reviewed before inclusion.
Scoring differs by output modality. For MCQs, a regex-based rule assigns a formatting weight wfmt​∈{0,0.5,0.75,1} multiplied by the count of correct minus incorrect selections, normalized per correct option. For image outputs, the authors adopt a VLM-as-a-Judge protocol with dynamically filled evaluation-points covering instruction fulfillment and cross-turn consistency, each scored 0–5 by Qwen3-VL-235B-A22B-Thinking. Human validation against two-person ratings yields Pearson correlations above 0.72 for all model-level and class-level averages, with an overall mean of 0.804, supporting the reliability of automated scoring—though the validation covers only five representative models sampled one response per task, so coverage of the full score distribution is limited.
Main evaluation results
The authors evaluate eight UMMs: six open-source (BAGEL 15B, BLIP3-o 4B/8B, OmniGen2 7B, Ovis-U1 3B, UNIWORLD-V1 20B) and two closed-source (GPT-5 Image Mini, Nano Banana). GPT-5 Image Mini leads decisively, achieving the best understanding and generation scores in all three classes with an average of 82.3, followed by Nano Banana at 79.5. Among open-source models, BLIP3-o 8B performs best at 65.0—a gap of over 17 points to the leader—and ranks first among open-source models in generation.
Three findings stand out. First, closed-source models are not only higher-scoring but more stable across classes, whereas open-source models fluctuate substantially; UNIWORLD-V1, for example, scores 46.6 overall despite a 20B parameter count, with text-modality scores as low as 21.9 on turn 1. Second, many open-source models show pronounced understanding–generation imbalance (e.g., OmniGen2 scores 70.0 on Temporal Causal understanding but only 43.4 on generation), which propagates across turns because generation depends on prior constraint alignment. Third, all open-source models fail on four shared domains—Stack Simulation, Geometric Coloring, Mirror Reasoning, and Paper Crafting—indicating a common weakness in fine-grained spatial and geometric reasoning under interleaving. Notably, even closed-source models degrade noticeably on Paper Crafting and Mirror Reasoning relative to their other tasks.
A supplementary comparison between Static-MCQ and Dynamic-MCQ shows that all models score substantially lower on the dynamic variant—for instance, GPT-5 Image Mini drops from 83.5 to 50.6, Nano Banana from 80.2 to 43.4, and OmniGen2 from 72.3 to 30.0. This indicates that accumulated errors in dialogue history impair understanding of self-generated content, an ability no existing benchmark had isolated before.
Exposure bias
The central empirical finding is a modality-asymmetric turn-wise trend: text-question scores show no consistent monotonic decline with turn number, while image-question scores fall sharply for nearly all models. GPT-5 Image Mini's image scores drop from 94.3 at turn 1 to 54.8 at turn 6; BAGEL falls from 68.5 to 24.7; Ovis-U1 from 61.8 to 27.3. The authors attribute this to exposure bias specific to generation: early-turn errors are written into later context, forcing subsequent turns to condition on an imperfect history, producing error accumulation and distribution drift. Because multi-turn generation must satisfy both the current instruction and cross-turn consistency constraints, this failure mode is invisible in single-turn evaluations. The finding implies that leaderboard results from static benchmarks systematically overstate UMM competence in realistic long-horizon interaction.
A case study illustrates three recurring failure modes: weak instruction following and format violations in open-source models, failure to preserve prior edits across turns even in closed-source models, and hallucination in final Dynamic-MCQ turns where models report attributes (e.g., shoe colors) absent from their own generated images.
Test-time scaling experiments
Using BAGEL as a representative open-source model, the authors evaluate three inference-time strategies against the baseline image-score average of 54.1:
| Method |
Static Spatial |
Temporal Causal |
Hybrid |
Average |
| Baseline |
57.9 |
49.3 |
52.8 |
54.1 |
| Chain-of-Thought |
72.0 |
60.1 |
60.4 |
65.9 |
| Self-Verification (max retries 1) |
59.0 |
50.7 |
53.3 |
55.2 |
| Best-of-N (N=5) |
62.0 |
52.7 |
60.1 |
58.6 |
Chain-of-Thought—which requires the model to summarize the request and state an explicit generation plan before generating, without appending reasoning to history—yields the largest gain (+11.8 average) and most clearly flattens the turn-wise decay curve. Self-Verification uses the model's stronger understanding capability to check instruction following and cross-turn consistency, then redraws based on diagnostic feedback; Best-of-N generates N candidates and lets the model select the best. Both reduce error accumulation but depend on extra inference cost.
The ablations reveal non-monotonic behavior worth noting. Self-Verification peaks at 1–2 retries (55.2 and 54.6 average) and degrades below useful levels at 3 retries (53.7), occasionally underperforming the baseline—overly frequent retries can be counterproductive, plausibly because repeated conditioning on failed images injects additional noise. Best-of-N improves monotonically up to N=5 but with sharply diminishing returns beyond N=3 (57.4 → 58.2 → 58.6), indicating a performance boundary set by the base model's candidate quality and selection accuracy. All strategies mitigate but do not eliminate exposure bias; the authors concede that further methodological work is needed to fully overcome decay in multi-turn generation.
Limitations and open questions
Several limitations bear on interpretation. The test-time scaling analysis is conducted on a single model (BAGEL) with a fixed configuration (one retry, N=3 in the main comparison), so the generality of the observed gains across architectures and scales remains unverified. Scoring relies on a single judge VLM (Qwen3-VL-235B-A22B-Thinking), introducing potential self-family bias even though human correlation is strong; the human validation sample is also modest. History construction differs across models—some support true interleaved input while others require serialized image lists or horizontal concatenation (Ovis-U1)—so part of the open-source/closed-source gap may reflect interface constraints rather than intrinsic capability. Finally, the benchmark caps interactions at six turns; whether the observed decay continues linearly, accelerates, or saturates over longer horizons is left open, as is whether training-time interventions (e.g., self-correction data or rollout-based fine-tuning) can address exposure bias more fundamentally than inference-time patching.
Conclusion
IMUG-Bench provides a systematic evaluation framework for UMMs in multi-turn interleaved settings, combining dynamic ground-truth determination, validated VLM-based scoring, and broad domain coverage. Its principal empirical contributions are the quantification of generation-side exposure bias—as distinct from stable understanding performance—and evidence that lightweight test-time scaling, particularly Chain-of-Thought planning, meaningfully mitigates it. The persistent large gap between closed-source and open-source models, the shared spatial/geometric weaknesses of open systems, and the sharp drop on Dynamic-MCQ together define a clear agenda for improving the robustness of future UMMs in long-horizon multimodal interaction.