ART Benchmark: Multimodal Art Evaluation

• ART Benchmark is a suite of evaluation datasets and protocols designed to assess MLLMs’ ability to interpret art through multimodal inputs and creative comment generation.
• It employs detailed annotation methods and diverse metrics, including human judgment, BLEU scores, and WEO, to measure both generative and discriminative performance.
• The benchmark highlights key challenges in creative, culturally-aware reasoning and precise scene grounding, driving improvements in multimodal AI model development.

Art benchmarks in the context of multimodal LLMs (MLLMs) are evaluation datasets and protocols specifically designed to probe the nuanced understanding and generation capabilities of AI models in the art domain. These benchmarks challenge models to interpret, reason about, or creatively engage with multimodal inputs—such as images, videos, and accompanying textual or cultural information—in ways that reflect the expertise of practitioners and the complexity of visual arts and creative commentary.

1. Motivation and Definition

Art benchmarks arise from the recognition that traditional evaluation datasets for MLLMs do not adequately capture the domain-specific complexities inherent to art. The “comment art” framework, as defined in GODBench, refers to user-generated comments under videos that evoke humor, satire, emotional resonance, or deep cultural insight. Producing or evaluating such content requires advanced multimodal reasoning, encompassing visual, textual, and contextual cultural understanding, as well as divergent and imaginative thinking (Lei et al., 16 May 2025).

Similarly, VQArt-Bench targets the visual question answering (VQA) task in art and cultural heritage, where models must interpret symbolic meaning, narrative structure, and iconological context beyond surface-level recognition (Alfarano et al., 14 Oct 2025). Prior datasets suffer from limited semantic diversity, over-reliance on template-based questions, and a focus on generic imagery, all of which incentivize shortcut learning rather than true comprehension or creativity.

2. Benchmark Construction and Annotation

Art benchmarks employ unique data construction methodologies tailored to their respective domains and tasks.

GODBench consists of 67,073 Kuaishou platform videos (average 55.5 s, 31 categories, 100+ subcategories) and 1,577,201 comments, categorized as "GOD-level" (highly upvoted, creativity-verified), high-quality, and ordinary. Annotation is undertaken by 31 professional annotators with multi-round expert review, leveraging detailed labeling guidelines. GOD-level comments are exhaustively labeled for all creativity subcategories.

VQArt-Bench is generated via a four-stage multi-agent pipeline, beginning with text cleaning of curator notes or Wikipedia entries to exclude off-topic content, followed by topic selection across seven visual reasoning dimensions, open-ended question generation, conversion to multiple-choice with visually plausible distractors, and a final LLM-based and human expert validation. The final dataset comprises 14,463 validated questions over 2,000–3,000 unique artworks, with stringent manual review to ensure factual grounding and minimize hallucination. Sample rates indicate >98% annotation correctness (Alfarano et al., 14 Oct 2025).

3. Evaluation Dimensions and Formal Criteria

Art benchmarks define evaluation dimensions that align with higher-order cognitive and artistic competencies.

GODBench: Five Creativity Dimensions

1. Rhetorical Techniques: Humor, puns, metaphor, satire.
2. Divergent Associations: Imaginary completion, role immersion, surrealism.
3. Clever Writing: Poetic forms, structural innovation, rhythm, elision.
4. Interactive Virality: Memes, catchphrases, cultural references, intertextuality.
5. Emotional Resonance: Authenticity, emotional impact, dark humor.

Tasks span selection, ranking, and classification (discriminative), as well as explanation and creation (generative).

VQArt-Bench: Seven Reasoning Dimensions

Let $I$ be the image, $O$ object instances with class labels $c$, attributes $A$, and locations $L$:

1. Instance Identity: $f_{\mathrm{ID}}(I, o) = c$
2. Instance Attribute: $f_{\mathrm{Attr}}(I, o, p) = a$, for property $p$
3. Instance Location: $f_{\mathrm{Loc}}(I, o) = (x, y)$ or relative terms (left/right, above/below)
4. Instance Counting: $$f_{\mathrm{Count}}(I, c^*) = |\{o \in O: c(o) = c^*\}|$$
5. Spatial Relation: $O$0, for relation $O$1
6. Instance Interaction: $O$2, for action $O$3
7. Visual-Inspired Reasoning: $O$4, for causal or contextual inferences

This explicit taxonomy underpins both the annotation protocols and model evaluation (Alfarano et al., 14 Oct 2025).

4. Evaluation Metrics

Metrics capture both discrimination and generative performance, extending beyond traditional classifiers.

GODBench:

• Accuracy (selection/classification), normalized discounted cumulative gain (NDCG) for ranking
• BLEU-1/2, DIST-1, ROUGE-L, BERT-F1 for generative similarity
• LLM-as-Judge (GPT-4o, scoring 0–5 on Creativity, Quality, Style, Impact, with weighted sum)
• Human preference voting in head-to-head comparisons
• Weighted Entity Overlap (WEO) for entity innovation, calculated as:

$O$5

where $O$6 scores entity frequency.

VQArt-Bench employs simple multiple-choice accuracy, both globally:

$O$7

and per-dimension. No consensus or weighting is applied. All answers are required to be unambiguously grounded in the visual content (Alfarano et al., 14 Oct 2025).

5. Reasoning Frameworks: Ripple of Thought

To address the limitations of standard reasoning paradigms, GODBench introduces the Ripple of Thought (RoT) framework, inspired by the physics of wave propagation and interference (Lei et al., 16 May 2025):

1. Ripple Initiation: Multilayered video analysis (OCR, scene parsing, emotional/cultural cues)
2. Ripple Focalization: Extraction of entity tuples (type, identity, attributes), storylines (actions, sequences), and environments (location, time, context)
3. Ripple Diffusion: Generation of candidate creative “ripples” via
   • Sequential, jumping (k-step), branching, and embedded association pathways (the latter incorporates trending memes/cultural hooks)
   • Produces multiple comment drafts
4. Wave Interference: Selection of highest-quality output via scoring over relevance, creativity, resonance
5. Luminous Imprint: Postprocessing for conciseness and safety

RoT enables more divergent, multi-hop associative reasoning, in contrast to linear chain-of-thought or compositional variants.

6. Experimental Findings and Model Benchmarking

Empirical results on both GODBench and VQArt-Bench reveal the current limitations and relative strengths across state-of-the-art MLLMs.

GODBench:

• Discriminative tasks: Human selection accuracy 84.21%; best open-source model ~46.43%; GPT-4o 54.19%; fine-tuned +RoT models reach ~72.72%.
• Generative tasks: RoT brings improvements (BLEU-1: 9.51 vs GPT-4o 6.36 and Qwen2-VL(7B) 5-shot 8.73), outperforms baselines in WEO entity innovation and LLM judge scores.
• Human preference: RoT outputs are preferred 35–40% of the time, rivaling verified human GOD-level comments in multiple dimensions (Lei et al., 16 May 2025).

VQArt-Bench (model accuracies excerpted):

Model	Attribute	Counting	Reasoning	Overall
Gemini 2.5	0.73	0.66	0.80	0.71
GPT-4o	0.66	0.59	0.72	0.64
Kimi-VL	0.69	0.64	0.83	0.67
LLaVA	0.44	0.26	0.50	0.39

Counting remains the weakest dimension; even high-accuracy models rarely exceed 0.70 on counting items. Visual-inspired reasoning generally yields higher scores, suggesting that broad commonsense knowledge is present, but detailed scene grounding (especially for enumeration and action recognition) remains limited (Alfarano et al., 14 Oct 2025).

7. Ongoing Challenges and Future Research Directions

Current benchmarks highlight systematic limitations in MLLM capabilities for art-related tasks:

• Existing models—even large proprietary ones—struggle with creative, culturally attuned comment generation (GODBench) and deep semantic reasoning in art (VQArt-Bench).
• Divergent association and entity innovation are particularly challenging; metrics such as WEO reveal where RoT-style multi-path reasoning holds advantage.
• The unique compositional, iconographic, and narrative structure of art is underrepresented in typical pretraining corpora.

Recommended directions include integration of domain-specific ontologies (e.g., Panofskian iconography), hybrid structured reasoning (e.g., symbolic scene graphs combined with LLM text generation), curriculum fine-tuning from low- to high-level perception, incorporating more modalities (audio, live interactions), and introducing open-ended engagement or human-in-the-loop critique—all motivated by the benchmarks’ findings (Lei et al., 16 May 2025, Alfarano et al., 14 Oct 2025).

Art benchmarks such as GODBench and VQArt-Bench set a new, more demanding standard for MLLMs, providing rigorous, multidimensional challenges that probe the boundary between statistical pattern recognition and genuine creative or expert-level visual-semantic understanding.