KVBench: Knowledge-Intensive Image Synthesis Benchmark

• KVBench is defined as a curriculum-grounded benchmark evaluating text-to-image generation with strict adherence to factual, structural, and symbolic correctness.
• It features 1,800 expert-curated prompts from over 30 textbooks across Biology, Chemistry, Geography, History, Mathematics, and Physics, evaluated with 5,158 checklist items.
• KVBench introduces KE-Check, a two-stage refinement process using structured prompt elaboration and checklist-guided auditing to mitigate logical, symbolic, and coverage errors.

KVBench is a curriculum-grounded benchmark for evaluating knowledge-intensive text-to-image generation. It was introduced to study settings in which image synthesis must satisfy not only semantic alignment but also domain knowledge, structural constraints, and symbolic conventions, with an explicit focus on high-school educational content across Biology, Chemistry, Geography, History, Mathematics, and Physics (Zhao et al., 24 Apr 2026). In this formulation, the central problem is not merely whether an image appears plausible, but whether it is scientifically correct by textbook standards, including the correctness of labels, component relations, diagram topology, and symbolic notation.

1. Conceptual scope and problem setting

KVBench is defined around a distinction between natural image generation and knowledge visualization. Conventional text-to-image systems are typically optimized for perceptual realism and coarse semantic alignment, whereas knowledge visualization requires strict adherence to factual content, structural organization, and domain-specific graphical conventions. The benchmark therefore treats scientific correctness, structural constraints, and symbolic conventions as first-class evaluation targets rather than secondary qualitative properties (Zhao et al., 24 Apr 2026).

This framing is motivated by recurring failure modes in current text-to-image models. The benchmark identifies logical reasoning deficits, symbolic precision issues, multilingual robustness gaps, and entity coverage failures. These include mis-ordered steps in process diagrams, violations of physical laws in physics setups, garbled or unreadable text, missing labels, omitted entities, spurious entities, and incorrect structural relationships. A common misconception is that visually plausible images are sufficient for educational or scientific use; KVBench is explicitly designed to show that plausibility and textbook-grade correctness can diverge substantially (Zhao et al., 24 Apr 2026).

2. Corpus construction and benchmark composition

KVBench is built from senior high-school curricula and consists of 1,800 expert-curated prompts derived from over 30 authoritative textbooks. It covers 6 subjects, 150 concepts per subject, and 2 languages, English and Chinese. The benchmark also includes one textbook reference image per prompt and fine-grained checklists with 4–6 atomic constraints per sample, totaling 5,158 checklist items (Zhao et al., 24 Apr 2026).

Aspect	Value	Function
Subjects	6	Biology, Chemistry, Geography, History, Mathematics, Physics
Concepts per subject	150	Curriculum grounding
Languages	2	English and Chinese
Total prompts	1,800	Evaluation set size
Checklist items	5,158	Fine-grained constraint evaluation

The data collection pipeline begins with source selection from authoritative textbooks and automatic pre-filtering using Qwen2.5-VL-32B to identify schematic diagrams and extract original captions. Candidate image-caption pairs are then reviewed by PhD-level domain experts for scientific accuracy, visual clarity, and conceptual richness. This procedure is intended to avoid noisy web-scale artifacts and to anchor the benchmark in textbook-standard visualizations rather than generic internet imagery (Zhao et al., 24 Apr 2026).

The benchmark content spans multiple diagrammatic regimes. Biology includes structural and conceptual diagrams such as cells, organs, biological systems, and cycles. Chemistry includes molecular structures, reaction schemes, and apparatus setups. Geography includes physical geography diagrams, maps, and spatial relationship schematics. History includes symbolic and culturally grounded depictions as well as timelines or causal diagrams. Mathematics covers graphs, coordinate plots, geometric constructions, and symbolic expressions. Physics includes experimental setups, vector diagrams, and process illustrations. This distribution suggests a benchmark that mixes object presence, spatial reasoning, symbolic rendering, and domain-specific visual syntax (Zhao et al., 24 Apr 2026).

3. Prompt tracks and evaluation methodology

Each benchmark sample is associated with two prompt tracks. The first is a Brief Caption taken directly from the textbook caption. It is intentionally abstract and often omits visual details, thereby testing internal knowledge and reasoning. The second is a Detailed Caption generated by a re-captioning process in which Qwen2.5-VL-32B receives the reference image and brief caption, after which the output is manually checked and refined by five PhD-level experts. The detailed version contains fine-grained visual details such as components, spatial relations, and labels (Zhao et al., 24 Apr 2026).

KVBench evaluates generated images through a checklist-based QA protocol rather than relying on metrics such as FID or CLIPScore. For a sample with constraint set

$\mathcal{C}(x) = \{c_1, c_2, \dots, c_N\},$

the benchmark constructs 4–6 binary, atomic checklist items using Qwen2.5-VL-32B conditioned on the reference image, brief caption, and detailed caption. These items cover key object presence, visual attributes, spatial relations, and reasoning-derived outcomes. For each generated image and checklist item $c_i$, an MLLM evaluator, specifically Qwen2.5-VL-32B-Instruct, assigns

$S_i \in \{0,1\}.$

The sample-level alignment score is then

$S = \frac{1}{N}\sum_{i=1}^{N} S_i.$

Aggregate scores for a model, subject, language, or prompt type are obtained by averaging $S$ over the relevant subset (Zhao et al., 24 Apr 2026).

The benchmark further validates automatic judging against human annotations. Three MLLM evaluators—GPT-4o, Gemini-2.0, and Qwen2.5-VL—are compared with expert judgments. Qwen2.5-VL achieves an overall accuracy of 80.26% against humans and Cohen’s kappa $\kappa = 0.7447$ with $p < 0.0001$, which the benchmark treats as sufficient evidence for automatic large-scale evaluation (Zhao et al., 24 Apr 2026).

4. Benchmarking results and diagnostic findings

KVBench evaluates 14 state-of-the-art open- and closed-source models. In the Brief Caption setting, reported overall scores for closed-source systems include FLUX.2-max at 21.97 (zh) and 34.98 (en), GPT-Image at 40.90 (zh) and 41.60 (en), Seedream-4.0 at 44.74 (zh) and 39.61 (en), and Gemini-3-Pro-Image at 51.36 (zh) and 34.40 (en). Open-source or open-weight systems include Janus-Pro at 8.81 (zh) and 21.61 (en), SD3.5-Large at 12.18 (zh) and 31.07 (en), FLUX.1-dev at 21.47 (zh) and 33.93 (en), FLUX.2-dev at 40.51 (zh) and 36.88 (en), and Qwen-Image at 20.09 (zh) and 25.74 (en) (Zhao et al., 24 Apr 2026).

Across most settings, closed-source models outperform open-source models, often by substantial margins. The benchmark also reports systematic language effects: English generally outperforms Chinese, which is associated with weaker multilingual robustness and difficulty in accurate Chinese text rendering. Subject-wise, Biology and Chemistry are relatively higher-scoring across models, while History and Geography are more difficult because they require cultural and contextual knowledge. Mathematics and Physics are particularly challenging because of symbolic precision and multi-step reasoning requirements (Zhao et al., 24 Apr 2026).

The benchmark’s error analysis distinguishes three principal failure modes. Visual readability failure concerns garbled or unreadable text in diagrams and is especially damaging in mathematics, physics, and labeled biological figures. Entity coverage failure concerns missing key components or adding irrelevant ones and is described as the most prevalent failure type even among stronger closed-source systems. Semantic logic failure concerns incorrect structural design, impossible configurations, and violations of basic principles, such as invalid bond structures or miswired circuits. These categories make the benchmark diagnostically interpretable: it does not only rank models, but localizes recurrent forms of scientific hallucination (Zhao et al., 24 Apr 2026).

A further result concerns prompt specificity. Detailed captions usually improve performance by supplying more explicit constraints. However, for some weaker open-source models, including Janus-Pro and OmniGen2, more detailed prompts reduce performance, which the benchmark attributes to weak long-instruction following: extra detail can act as noise rather than guidance (Zhao et al., 24 Apr 2026).

5. KE-Check and benchmark-guided refinement

KVBench was introduced together with KE-Check, a two-stage framework intended to improve scientific fidelity. The problem is formalized by defining a concise prompt $x$, a T2I generator $G$, and generated image

$I = G(x).$

Knowledge visualization requires satisfying constraint set

$c_i$0

and violations are represented as

$c_i$1

KE-Check reduces this violation set through Knowledge Elaboration and Checklist-Guided Refinement (Zhao et al., 24 Apr 2026).

In the first stage, Knowledge Elaboration transforms the concise prompt into a structured, knowledge-rich prompt

$c_i$2

using an MLLM-driven procedure with structured outputs such as <analysis>, <checklist>, and <final_caption>. The elaborated prompt explicitly encodes entities, symbols, labels, layout, and diagram style. The initial image is then generated as

$c_i$3

In the second stage, a structured checklist is used to audit constraint satisfaction. For each item, a violation indicator

$c_i$4

defines the current violation set

$c_i$5

These violations are aggregated into refinement instructions for editing or regeneration (Zhao et al., 24 Apr 2026).

Empirically, KE-Check substantially improves Qwen-Image on the Brief Caption track. Baseline Qwen-Image scores are 20.09% (zh) and 25.74% (en), while Qwen-Image + KE-Check reaches 47.67% (zh) and 30.24% (en). Subject-level gains in Chinese are particularly large: Biology rises from 14.76 to 54.73, Chemistry from 15.17 to 55.47, Geography from 29.00 to 55.87, History from 21.97 to 38.75, Mathematics from 19.16 to 34.89, and Physics from 20.53 to 46.33 (Zhao et al., 24 Apr 2026).

Ablation results attribute most of the improvement to structured prompt enrichment, with further gains from refinement. Baseline Qwen-Image at 20.09 (zh) and 25.74 (en) improves to 36.17 and 28.12 with Knowledge Elaboration, then to 46.50 and 30.69 with refinement without checklist, and finally to 47.67 and 30.24 with full KE-Check (Zhao et al., 24 Apr 2026).

6. Significance, limitations, and research implications

KVBench’s main contribution is methodological. It shifts evaluation from generic visual plausibility to explicit scientific fidelity, and it does so with curriculum grounding, textbook-derived prompts, bilingual evaluation, and checklist-based scoring. This makes it particularly relevant for educational diagrams, scientific communication, and any text-to-image deployment in which incorrect imagery can mislead rather than merely underperform (Zhao et al., 24 Apr 2026).

The benchmark also clarifies what current systems do not yet solve. Open-weight models lag behind proprietary systems, multilingual robustness remains limited, and symbolic precision remains a major obstacle. A plausible implication is that knowledge-intensive generation requires explicit intermediate structure—constraints, checklists, or domain-grounded elaboration—rather than raw end-to-end prompt following alone. KE-Check operationalizes this implication by turning the benchmark’s own evaluative logic into a refinement pipeline (Zhao et al., 24 Apr 2026).

KVBench has stated limitations. Its scope is restricted to high-school subjects rather than college-level or professional domains. It evaluates only static 2D images, not video or interactive visualizations. Language coverage is limited to English and Chinese. The corpus is drawn from selected authoritative textbooks and therefore reflects specific curricula and, especially for History, particular cultural perspectives. Finally, the benchmark relies on MLLM judges; despite strong agreement with human evaluation, this still leaves residual judgment error (Zhao et al., 24 Apr 2026).

Within those limits, KVBench establishes a concrete benchmark for knowledge-intensive text-to-image generation and a corresponding diagnostic vocabulary for scientific hallucination, structural error, and symbolic failure. For research, it provides a standardized target for improving knowledge-grounded visual synthesis. For deployment, it indicates that current models should not be trusted in scientifically sensitive settings without explicit constraint modeling, auditing, and refinement (Zhao et al., 24 Apr 2026).