VQArt-Bench: Art and Cultural Heritage VQA

• VQArt-Bench is a large-scale, semantically structured Visual Question Answering benchmark for art and cultural heritage that rigorously tests multimodal models.
• It employs a multi-agent pipeline to generate diverse and verified questions across seven semantic dimensions including identity, attribute, and visual-inspired reasoning.
• Empirical evaluations show that while models perform well on reasoning tasks, they struggle with counting and other precise visual metrics, highlighting key improvement areas.

VQArt-Bench is a large-scale, semantically structured Visual Question Answering (VQA) benchmark designed for art and cultural heritage analysis. It addresses inadequacies in prior art-VQA datasets—such as limited semantic depth and over-reliance on superficial attribute recognition—by presenting multimodal models with nuanced, non-trivial, and linguistically varied questions that probe both surface and deep levels of visual and semantic understanding. VQArt-Bench leverages a novel, multi-agent pipeline to ensure questions are verifiable, grounded in image evidence, and span a broad taxonomy of visual interpretive skills. It serves as a rigorous testbed for the evaluation of state-of-the-art Multimodal LLMs (MLLMs) on visual reasoning, symbolic understanding, and complex narrative comprehension within real artworks (Alfarano et al., 14 Oct 2025).

1. Benchmark Construction Pipeline

The foundation of VQArt-Bench is an automated, agentic pipeline with explicit controls to ensure the semantic legitimacy and linguistic diversity of questions:

• Data Collection and Cleaning: Artworks are sourced from approximately 30,000 Wikipedia pages, filtered to include only entries with sufficiently descriptive text (≥400 words). A LLM preprocessor extracts visual-element-describing sentences, discarding artist biographies and extraneous context.
• Multi-Agent Question Generation:

1. Topic Selector: Receives a snippet pertaining to one artwork and outputs candidate topics (objects, actions, spatial relations, symbolic elements), each with grounded textual evidence.
2. Question Generator: For each topic, produces a linguistically varied open-ended question answerable solely from the image and its validated snippet.
3. Question Refiner: Converts open-ended questions into 4-way multiple-choice format by generating three plausible, contextually grounded distractors to minimize shortcut exploitation.
4. Judge: Evaluates each candidate Q&A + image pair for unambiguity, non-triviality, well-formedness, and strict conformity to a defined semantic dimension.

• Human Validation: A 25% stratified sample of accepted Q&A pairs undergoes expert review for grounding and correctness, achieving >98% hallucination-free status.

This multi-phase design targets the production of questions that genuinely require both visual and semantic reasoning and avoids the annotation shortcuts endemic to prior datasets (Alfarano et al., 14 Oct 2025).

2. Semantic Taxonomy and Question Dimensions

Every question in VQArt-Bench is annotated along exactly one of seven dimensions, inspired by cognitive-scientific hierarchies and designed to span the core axes of visual-semantic understanding:

1. Instance Identity: Object or figure classification queries (e.g. “Which animal stands at the woman’s feet?”).
2. Instance Attribute: Queries probing properties or qualities of an entity (e.g. “What is the color of the drapery?”).
3. Instance Location: Spatial position queries (e.g. “Where is the dove positioned relative to the halo?”).
4. Instance Counting: Numerical queries on visually distinct objects (e.g. “How many cherubs are visible?”).
5. Spatial Relation: Questions on the arrangement or relationship of objects (“What is the relationship between the tree and the throne?”).
6. Instance Interaction: Actions or functional relations between entities (e.g. “What gesture does the saint make toward the child?”).
7. Visual-Inspired Reasoning: Causal, inferential, or intent-based reasoning (e.g. “Why is the figure’s hand raised?”).

This structure moves well beyond prior art-VQA benchmarks with binary (visual/knowledge) splits and provides direct control over the distribution and type of visual interpretive challenges presented (Alfarano et al., 14 Oct 2025).

3. Dataset Composition and Statistics

After agentic filtering and validation, VQArt-Bench comprises 14,463 multiple-choice questions distributed across ≈30,000 artworks, with a non-uniform question-per-artwork mapping. The precise breakdown by semantic dimension is shown below:

Dimension	Question Count
Instance Identity	2,031
Instance Attribute	2,598
Instance Location	2,100
Instance Counting	1,710
Spatial Relation	2,067
Instance Interaction	1,794
Visual-Inspired Reasoning	2,163
Total	14,463

Formally, the total number is

$N_{\rm total} = \sum_{d=1}^{7} N_d = 14,463,$

where $N_d$ denotes the cardinality of questions in dimension $d$. The dataset design enforces approximately even distribution across dimensions, explicitly mitigating shortcut learning and template bias (Alfarano et al., 14 Oct 2025).

4. Evaluation Protocol and Performance Metrics

VQArt-Bench employs a strictly multiple-choice (4-way) format with no provision for generative, free-text answers. The evaluation metric is categorical accuracy—i.e., the fraction of correctly selected answers across the entire dataset or by dimension. The scoring functions are: $$\mathrm{Acc}_{\rm overall} = \frac{1}{N_{\rm total}} \sum_{i=1}^{N_{\rm total}} \mathbb{1}(\hat y_i = y_i)$$

$$\mathrm{Acc}_{d} = \frac{1}{N_d} \sum_{i \in \mathcal{Q}_d} \mathbb{1}(\hat y_i = y_i)$$

where $\hat y_i$ is the predicted answer, $y_i$ is the ground truth, and $\mathcal{Q}_d$ is the set of question indices for dimension $d$.

No engineered loss functions or training objectives are utilized: all models are evaluated zero-shot with the explicit prompt “Question + 4 choices → select A/B/C/D.” This setup emphasizes a model’s out-of-the-box visual reasoning and semantic understanding (Alfarano et al., 14 Oct 2025).

5. Empirical Benchmarking of Multimodal Models

VQArt-Bench presents a rigorous evaluation of 14 contemporary MLLMs, including both closed-source and open-source architectures. Each model’s performance is reported per semantic dimension and overall. Key accuracy results include:

Model	Attribute	Location	Counting	Sp. Rel.	Interact	Identity	Reasoning	Overall
Gemini 2.5	0.73	0.73	0.66	0.72	0.75	0.74	0.80	0.71
GPT-4o	0.66	0.66	0.59	0.65	0.68	0.67	0.72	0.64
Kimi-VL	0.69	0.70	0.64	0.66	0.67	0.72	0.83	0.67
Gemma 3 27B	0.41	0.39	0.39	0.41	0.47	0.54	0.54	0.42

Major empirical findings:

• Counting tasks are systematically more difficult for all models (accuracy range 0.26–0.66)—notably lower than for other semantic dimensions—despite being trivial for human annotators.
• Visual-Inspired Reasoning scores are highest (up to 0.83), suggesting that large models rely heavily on world knowledge and priors for inference-type questions, potentially at the expense of precise visual enumeration.
• Closed-source models (e.g., Gemini 2.5) consistently outperform open-source models by 5–10 percentage points overall.
• Scale effects are pronounced among open-source models: Gemma 3 27B exceeds smaller variants by +9 points overall.

This stratification highlights which dimensions are not yet robustly modeled by state-of-the-art MLLMs and points to persistent limitations even in large, proprietary architectures (Alfarano et al., 14 Oct 2025).

6. Advances Over Prior Art and Limitations

VQArt-Bench introduces multiple advances relative to earlier art-VQA datasets such as AQUA (Garcia et al., 2020) and VISCOUNTH, which primarily employed rigid, template-generated questions with limited semantic and linguistic diversity. In contrast, VQArt-Bench:

• Provides a multi-dimensional taxonomy of question types, including fine-grained categories targeting visual abstraction, relational reasoning, and iconography.
• Employs an LLM-driven, multi-agent pipeline to minimize annotation shortcuts and hallucinations, as validated by extensive expert review.
• Explicitly mitigates bias toward surface attribute recognition and statistical artifacts by distributing questions across semantic categories with controlled distractor construction.

Limitations of the current benchmark include:

1. Restriction to 4-way multiple-choice questions; no evaluation of models’ free-form generative capabilities.
2. Reliance on Wikipedia-sourced artworks, potentially biasing the dataset toward Western or well-documented objects.
3. Absence of ontological grounding (e.g., no formal linkage to ICONCLASS or IICONGRAPH), which might be necessary for advanced multi-hop reasoning.
4. Residual reliance on LLMs in the generative pipeline, which can overlook rare or subtle iconographic features.

Ongoing and proposed extensions encompass free-text evaluation, augmentation with non-Western art collections, integration with formal art-historical ontologies, and adaptation to multilingual, cross-cultural scenarios (Alfarano et al., 14 Oct 2025).

7. Significance and Future Directions

VQArt-Bench establishes a demanding evaluation regime for multimodal models, shifting the field from surface-level attribute recognition to deeper forms of artistic understanding, such as symbolic meaning extraction and narrative interpretation. It exposes systematic weaknesses—particularly in enumeration and precise instance localization—across models of diverse architectures and training regimens. A plausible implication is that future model improvements will require both abstract visual representation learning and tighter integration with canonical art-history ontologies for robust multi-hop reasoning.

Extension avenues include:

• Adaptation for free-form (generative) VQA,
• Expansion to underrepresented art forms and traditions,
• Ontological enrichment for semantic grounding,
• Systematic dynamic dialogue for stepwise reasoning evaluation.

VQArt-Bench is positioned as a milestone for research aiming to bridge human-level visual-literary competence and automated multimodal interpretation in complex art domains (Alfarano et al., 14 Oct 2025).