HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understanding

Abstract: Automated fine-grained perception of calligraphy styles--a task vital to cultural heritage preservation--remains a critical challenge for Large Vision-LLMs (LVLMs), largely constrained by existing datasets that suffer from modal mixture and flattened labels. To bridge this gap, we introduce HCSU, the first comprehensive dataset tailored for fine-grained Historical Calligraphy Style Understanding. HCSU comprises 39,307 meticulously curated character images from 49 historically prominent calligraphers across 10 dynasties, systematically decoupling authentic ink manuscripts (Tie) from stone rubbings (Bei) to resolve the long-standing modal mixture problem. Moving beyond conventional flattened labels, HCSU provides hierarchical expert-written aesthetic descriptions, enabling two rigorous evaluation protocols: fine-grained style discrimination and interpretable aesthetic reasoning. Extensive evaluations reveal a persistent gap between calligraphy-related knowledge and visually grounded style perception: state-of-the-art LVLMs show non-trivial performance but remain sensitive to script-level, textual, and source-specific cues, and often struggle to ground aesthetic judgments in fine-grained brushwork evidence. Ultimately, the HCSU benchmark exposes fundamental limitations in current multimodal architectures, aiming to inspire the evolution of expert-level visual reasoning for cultural heritage preservation. The dataset is available at https://huggingface.co/datasets/Tongji209/HCSU.

• The paper presents a novel dataset, HCSU, with 39,307 images from 49 calligraphers across 10 dynasties, enabling fine-grained author and style discrimination.
• It details an innovative hierarchical annotation schema and rigorous preprocessing techniques to isolate stylistic cues and reduce imaging artifacts.
• Empirical results reveal that state-of-the-art models struggle with accurate style attribution and aesthetic description, highlighting gaps in visual versus conceptual reasoning.

HCSU: A Fine-Grained Dataset and Benchmark for Historical Calligraphy Style Understanding

Motivation and Challenges in Fine-Grained Calligraphy Perception

Accurate computational analysis of historical calligraphy style is a technically demanding vision-language problem with direct implications for cultural heritage preservation and computational art history. Prior datasets and benchmarks—CASIA-HWDB, HisDoc1B, MCCD, and CalliNet—fundamentally conflate visual modalities (e.g., stone rubbings versus ink manuscripts), flatten style labels, and neglect hierarchical structure, thus limiting the capability of modern LVLMs to justify or discriminate individual authors' brushwork at fine granularity. The HCSU benchmark directly mitigates modal mixture confounders and label impoverishment by providing an expertly curated, multi-domain, and richly annotated dataset, facilitating rigorous analysis of both discriminative and generative capabilities of state-of-the-art LVLMs in the context of Chinese calligraphic traditions.

Dataset Construction and Hierarchical Annotation Schema

HCSU contains 39,307 character images systematically curated from 49 prominent calligraphers across 10 dynasties, distributed over three domains: fully processed ink manuscripts (Tie), processed stone rubbings (Bei), and unconstrained raw images (Wild). The curation protocol strongly emphasizes canonical exemplars for each author-script pair, with strict quality controls for structural completeness and minimal background noise. Image preprocessing pipelines combine polarity normalization, geometric alignment, and semantic masking to ensure that structural and ink-related cues are preserved while reducing acquisition artifacts and background noise.

Figure 1: HCSU spans 10 dynasties and five scripts with annotations decomposing style into ink, stroke, and structure as well as high-level aesthetic judgments.

The metadata schema is hierarchical and multi-modal: every character instance is annotated for character content, author, dynasty, script type, stroke style, ink style, structural attributes, and expert-written natural language aesthetic descriptions adhering to classical art-historical critique lexica. This enables supervised tasks not only in distinguishing authorative brush signatures but also in interpretable textual reasoning—supporting evaluation of both low-level visual perception and high-level aesthetic articulation.

Benchmark Design and Evaluation Protocols

HCSU introduces two key evaluation tracks motivated by challenges uniquely prominent in calligraphy:

1. Fine-Grained Style Discrimination: Unlike coarse writer/script identification, this task isolates the visual style signature irrespective of glyph or script structure. An 8-way candidate selection protocol requires LVLMs to select the author-style from highly confusable distractors, a setup tuned for robust granularity based on empirical $K$-way calibration.
2. Interpretable Aesthetic Reasoning: Beyond classification, models must generate natural-language descriptions that employ correct professional terminology as used in art history and relate directly to observable features (e.g., ink moisture, brush velocity, spatial rhythm). Evaluation combines BERTScore and an LLM-judge on expert-level terminology usage and visual detail richness.
3. Robustness and Description Ablation: Comprehensive ablations interrogate effect of data domain (raw vs. processed), canonicalization protocol, and descriptive text quality (random, GPT-generated, or expert).

Empirical Results: Perception, Reasoning, and Model Limits

Discriminative Performance

Despite strong technical advances, state-of-the-art proprietary models (Doubao-1.5-Vision-Pro-250328, GPT-5.2, Claude, Gemini) deliver only 29–37% accuracy on 8-way selection (random is 12.5%), and large open-weight models peak at 30.25%, confirming persistent difficulty even for models of 200B+ scale. Model scaling boosts performance but does not suffice for closing the gap between generalist LVLMs and the expert-level discrimination required in calligraphy.

Domain Sensitivity

A counterintuitive domain bias is observed: models perform better on Bei (high-contrast stone rubbings) than Tie (ink manuscripts), contradicting the expectation that ink dynamics in Tie confer clearer cues. This result suggests current CNN/ViT-based encoders are disproportionately sensitive to geometric border contrast, yet insufficiently attuned to subtle continuous ink/texture variations that define brushwork style.

Generative Reasoning

The best discriminative models often underperform in aesthetic articulation. For example, Doubao-1.5-Vision-Pro-250328, despite leading discrimination, yields notably low scores in description richness and terminology. By contrast, GPT-5.2 and Claude models slightly surpass expert references in employing technical terminology, but in some cases do so by regurgitating plausible but “ungrounded” stylistic descriptors not always supported by direct visual evidence.

Figure 2: The challenge of disentangling style from content, as visually distinct renderings of the same glyph must be mapped to artist style, not character class.

Data and Description Ablation

Robustness experiments show that raw Wild images offer higher discrimination performance than processed canonical images—likely reflecting the availability of incidental cues (paper aging, seals, mounting) as well as finer stroke textures. However, these cues may include spurious correlates as well as genuine stylistic signals, raising open challenges for ensuring style attribution is grounded in proper evidence. Description source ablation indicates expert annotations boost accuracy for Bei, but gains are modest due to vocabulary misalignment with model pretraining.

Figure 3: Domain-wise distribution of script types in the Wild subset, capturing the dataset's breadth along calligraphic and historical axes.

Figure 4: Data-driven calibration of benchmark, showing selection of optimal $K$ for candidate discrimination difficulty.

Figure 5: Example of raw Tie sample pre- and post-canonical processing, demonstrating denoising and structure preservation.

Implications and Theoretical Considerations

HCSU exposes a systematic gap between language/knowledge and visual grounding in contemporary LVLMs. Discriminative matching of style remains sensitive to visual shortcuts (geometric boundary, background cues), while generative reasoning can produce plausible but visually ungrounded terminology. The persistent superiority of proprietary models over open-weights and the Bei–Tie domain inversion underscore unresolved limitations in multimodal alignment, visual encoder architectures, and the integration of art-historical context within model training.

Practically, these results urge caution for deploying LVLMs in expert-level cultural heritage interpretation and suggest that current contrastive pretraining regimes insufficiently capture micro-level visual attributes central to creative authorship. From a theoretical perspective, the dual-task design provides an analytically fertile ground for investigating the tension between “knowledgeable but unperceptive” architectural paradigms and the requirements of fine-grained, evidence-driven aesthetic reasoning.

Future Directions

Addressing these challenges may require:

• Pretraining/fine-tuning on art-historical corpora with explicit grounding in brushwork-relevant features;
• Architecture modifications to improve sensitivity to continuous texture and ink distribution, potentially through hybrid convolutional-transformer or denoising diffusion backbones;
• Incorporation of hierarchical annotation schemas in both multi-task learning and prompt engineering;
• Cross-modal alignment objectives specifically targeting the disentanglement of content and style at the visual-conceptual boundary.

Conclusion

HCSU establishes the first domain-precise, interpretable benchmark for fine-grained historical calligraphy style analysis, directly targeting the failure modes of modern LVLMs in visually and conceptually complex arts domains. By isolating authentic ink perception and integrating expert-level, hierarchical annotations with comprehensive evaluation tracks, it provides essential infrastructure for future research into expert-level visual reasoning and lays the groundwork for advancing AI systems capable of genuine cultural heritage understanding (2607.04147).