CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage

Abstract: Evaluating Multimodal LLMs (MLLMs) in Chinese Cultural Heritage (CCH) requires fine-grained reasoning over visual, textual, stylistic, and historical clues. However, existing CCH benchmarks mainly emphasize final-answer accuracy, while the accuracy and completeness of reasoning processes remain underexplored. To address this gap, we introduce CulMind and CulMind-R: a high-quality benchmark for multimodal CCH covering 50 tasks from collections of more than 100 museums, and a 24-task reasoning subset that adaptively defines task-specific dimensions for reasoning process evaluation. To evaluate reasoning quality, we propose ReaScore, a task-adaptive metric that evaluates reasoning by automatically weighting task-relevant dimensions. Experiments on 14 leading MLLMs reveal a substantial gap between answers and reasoning, especially on challenging tasks. Further analysis shows that task-adaptive dimension selection and weighting better align evaluation results with expert judgments. Overall, our benchmark and metric support a more expert-aligned assessment of CCH understanding and offer a transferable reference for broader evaluations of cultural heritage. We publicly release the data, code, and evaluation scripts at https://github.com/ZevTsao/CulMind to facilitate reproducible research.

• The paper introduces CULMIND, a comprehensive multimodal benchmark that evaluates both answer accuracy and detailed process-level reasoning in Chinese cultural heritage contexts.
• The paper constructs a dataset of 12,381 QA pairs across seven subdomains, using expert-written, metadata-instantiated, and LLM-assisted items with rigorous quality control.
• The paper validates the REASCORE metric through expert annotations, revealing that even answer-correct models often exhibit unreliable reasoning.

Expert Overview of "CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage" (2606.21618)

Benchmark Scope and Motivation

The paper establishes CULMIND—a comprehensive multimodal benchmark addressing the evaluation of Multimodal LLMs (MLLMs) in Chinese Cultural Heritage (CCH) contexts. Recognizing the unique multimodal complexity of CCH (text, image, style, historical context), the authors observe that existing benchmarks for MLLMs and CCH either lack fine-grained reasoning-process evaluation or exhibit limited domain/task coverage. CULMIND is constructed to balance canonical and long-tail cultural remains, covering seven subdomains with 50 tasks sourced from 100+ institutions. The accompanying CULMIND-R subset targets reasoning-intensive tasks, enabling process-level reasoning evaluation with adaptive dimensions rather than generic templates.

Dataset Construction and Schema Design

CULMIND contains 12,381 QA pairs (9,058 multiple-choice, 3,323 open-ended) spanning Ancient Books, Calligraphy, Painting, Seals, Decorative Arts, Cave Murals, and Ancient Scripts. Data generation integrates three sources: expert-written items (30%), metadata-instantiated questions (50%), and LLM-assisted instances (20%) with expert review. Rigorous quality control ensures answer correctness, image-question consistency, and metadata faithfulness; ambiguous or contentious items are filtered through structured group review.

CULMIND-R focuses on Calligraphy, Painting, and Decorative Arts—domains requiring integrative reasoning—comprising 6,032 QA pairs for 24 reasoning tasks. Process evaluation is structured via task-adaptive schemas: each task specifies which dimensions are targets, supporting, contextual, or excluded. This role assignment underpins precise evaluation of how models leverage textual, visual, stylistic, and provenance features.

Reasoning Evaluation: REASCORE

To address the inadequacy of answer-only metrics and generic reasoning evaluation, the authors introduce REASCORE. This automatic metric scores models' reasoning outputs against reference dimension pools with task-adaptive weights, derived from expert roles and further inferred via an auxiliary LLM (DeepSeek-V4-Pro). Scoring is type-aware (categorical/text/list/numeric), with partial credit for calibrated uncertainty to discourage unsupported specificity.

REASCORE aggregates dimension-level scores with adaptive weighting: the same dimension may have decisive impact for one task but be irrelevant for another. Only active dimensions contribute to the reasoning-process score, and missing fields or uncertainty are penalized according to calibrated constants.

Experimental Results: Model Performance and Analysis

Benchmarking 14 MLLMs (10 open-source, 4 closed-source), the authors assess both answer-only and process-level reasoning quality. The answer-only setting shows best overall accuracy from Gemini-3-Flash-Preview (0.507), with subdomain-specific strengths and non-monotonic trends across parameter scale. Decorative Arts is the most tractable category (0.749 average), and Ancient Scripts the most challenging (0.059). Fine-grained tasks relying on low-frequency scripts or knowledge-intensive reasoning remain difficult for all models.

However, CULMIND-R and REASCORE demonstrate a profound misalignment between final-answer correctness and reasoning reliability. Strong numerical insight: 90.6% of answer-correct instances manifest incorrect or weak reasoning. Even leading models (e.g., Gemini-3-Flash-Preview) show only 12.8% reliable reasoning among answer-correct cases. Task-adaptive structured reasoning prompts variably affect final-answer scores per model, with some improving and others degrading.

Dimension-set ablations confirm that task-adaptive selection achieves high recall and precision with minimal noise compared to exposing all metadata or only direct task dimensions. Task-related sets miss vast swaths of necessary supporting structure, while unrestricted metadata pools encourage unsupported specificity.

Metric Validation and Theoretical Implications

REASCORE's validity is independently confirmed via blinded expert annotation on Gemini-3-Flash-Preview outputs. Agreement with expert process scores (Spearman 0.9898) and quadratic weighted kappa (QWK 0.9186) surpasses uniform and fixed weighting variants, as well as answer-only correctness. Bootstrap resampling yields statistically significant correlation improvements for adaptive weighting over alternatives.

These results substantiate that structured reasoning evaluation via task-adaptive dimensions provides superior alignment with domain-expert judgment compared to generic criteria or outcome-based metrics. The finding that answer-correct models often exhibit unreliable reasoning underscores the necessity of process-level evaluation in complex cultural-heritage domains.

Practical and Theoretical Implications, Limitations, Future Outlook

CULMIND provides a systematic pipeline for evaluating multimodal cultural-heritage understanding, placing fine-grained, process-oriented demands on MLLMs. Practically, this aids benchmarking the efficacy of vision-language integration for museum curation, art analysis, provenance studies, and digital humanities, but cautions against deploying answer-only systems without process validation.

Theoretical advances include the operationalization of task-adaptive process evaluation in heterogeneous multimodal domains and the quantification of reasoning reliability via structured dimension pools. The extensibility of CULMIND schemas and REASCORE enables adaptation to other heritage contexts and open-ended reasoning forms with appropriate expert collaboration and schema revision.

Limitations center on the domain specificity and current focus on calligraphy, painting, and decorative arts; structured reasoning may require schema redesign for other domains (e.g., highly ambiguous historical or ritual cases). Further research should include retrieval-augmented reasoning, uncertainty-aware scoring, and multiple expert references.

Anticipated developments include the expansion of CULMIND to other cultural-heritage fields as more digitized resources and expert annotations become available, and the adoption of REASCORE-like metrics in robust, expert-aligned AI evaluation protocols. These directions are essential for trustworthy deployment of MLLMs in knowledge-intensive real-world scenarios.

Conclusion

CULMIND and CULMIND-R, with the REASCORE metric, establish a high-resolution, process-oriented evaluation framework for multimodal CCH understanding and reasoning in MLLMs. The rigorous separation between answer accuracy and reasoning reliability is empirically validated and reflects both practical and theoretical needs for expert-aligned multimodal model assessment. The benchmark and metric provide a foundation for future research and evaluation in computational cultural heritage and broader vision-language reasoning domains.