AncientDoc: Chinese Ancient Document Benchmark
- AncientDoc is a benchmark for Chinese ancient documents that evaluates multimodal tasks including OCR, translation, reasoning-based QA, knowledge-based QA, and linguistic variant QA.
- The benchmark covers over 3,000 pages from diverse historical periods, predominantly from the Ming and Qing dynasties, and challenges models with vertical layouts and degraded text formats.
- Empirical results reveal that current vision-language models struggle with the complexity of ancient scripts and semantic interpretation, highlighting the need for specialized pretraining and evaluation techniques.
AncientDoc is a benchmark for Chinese ancient documents designed to evaluate vision-LLMs across a spectrum that runs from page transcription to semantic interpretation and historical reasoning. It is introduced as the first benchmark specifically designed for Chinese ancient documents, with five tasks—page-level OCR, vernacular translation, reasoning-based QA, knowledge-based QA, and linguistic variant QA—spanning 14 document types, over 100 books, and about 3,000 pages (Yu et al., 10 Sep 2025). In a broader computational-philology perspective, it belongs to a research trajectory that no longer treats ancient materials as merely scanned artifacts or isolated OCR targets, but as historically layered, semantically structured, and computationally explorable corpora (Hedges et al., 2016).
1. Definition and research setting
AncientDoc addresses a practical problem in the digitization of Chinese historical materials. Large-scale library and museum digitization has produced many page images, but image capture alone does not provide machine-actionable access to the linguistic, historical, and domain knowledge embedded in those sources. The benchmark is motivated by the claim that current vision-LLMs struggle with the visual and linguistic complexity of Chinese ancient documents, while existing document benchmarks concentrate on English printed material or simplified Chinese and therefore fail to test this domain adequately (Yu et al., 10 Sep 2025).
The benchmark’s difficulty profile is explicitly tied to the properties of the source material. Visually, many pages are vertically typeset from right to left; they may include interlinear notes, comments, small characters, postscripts, and degraded printing or scanning. Linguistically, ancient Chinese often lacks punctuation, contains high polysemy, and requires interpretation of compressed and historically layered expression. Some tasks also require historical, cultural, or domain knowledge, including medical terminology, place names, persons, and classical concepts (Yu et al., 10 Sep 2025).
This positioning places AncientDoc within a wider family of ancient-document research, but with a distinctive scope. Semantic-web editing of gnomologia emphasizes intertextual graphs rather than page understanding (Hedges et al., 2016); Perseus 6 and ATLAS emphasize stand-off annotations, CTS alignment, and an integrated reading environment (Crane et al., 2024); AthDGC emphasizes diachronic parsing and multilingual alignment for Greek (Lavidas et al., 13 Jun 2026). AncientDoc differs by using page images of Chinese ancient books as the primary input and asking whether a multimodal model can both read and understand them (Yu et al., 10 Sep 2025).
2. Corpus composition and historical coverage
AncientDoc is built from real page images, mainly from digitized Chinese ancient document holdings in the Harvard Library. The dataset focuses on representative printed Chinese ancient documents from the Qing dynasty and earlier, spanning major dynasties from the Warring States/Qin/Han through Tang, Song, Ming, and Qing. The abstract gives the scale as 14 document types, over 100 books, and about 3,000 pages; the detailed corpus analysis specifies 2,973 pages (Yu et al., 10 Sep 2025).
The chronological distribution is uneven and historically informative. The Ming dynasty contributes 1,148 pages and the Qing dynasty 778, together about 65% of the corpus, followed by Song with 540 pages and Tang with 208. Genre coverage is broad: biography, Confucianism, philosophy of war, collected works, medical authors, astronomy and mathematics, masters of unofficial history, general anthologies, eclectics, Chuci-style poetry, leishu, art, literary criticism of poetry and prose, and genealogies and catalogues. The largest categories by page count include astronomy and mathematics with 238 pages, art with 234, and Confucianism with 232 (Yu et al., 10 Sep 2025).
Script style is overwhelmingly regular. About 97% of the pages are regular script and 3% are cursive script. This means the benchmark is historically broad but paleographically selective. A plausible implication is that AncientDoc is strongest as a benchmark for printed, relatively legible book culture rather than highly cursive or manuscript-heavy settings. The paper does not report train/validation/test splits in the provided text, so benchmark partitioning cannot be specified more precisely (Yu et al., 10 Sep 2025).
3. Task architecture and annotation workflow
AncientDoc organizes evaluation into five tasks that move from visual extraction toward higher-level interpretation.
| Task | Primary output | QA pairs per page |
|---|---|---|
| Page-level OCR | Full-page text in correct reading order | 1 |
| Vernacular translation | Modern vernacular Chinese rendering | 1 |
| Reasoning-based QA | Inferred answer from page content | 2 |
| Knowledge-based QA | Objective knowledge grounded in the page | 2 |
| Linguistic variant QA | Answer about style, rhetoric, or variant expression | 2 |
Page-level OCR is explicitly framed as full-page OCR without reliance on the traditional pipeline of character detection, segmentation, and recognition. It tests whether a model can read vertical right-to-left columns, resolve line order, and decide how to treat annotations and small inserted text. Vernacular translation is intralingual translation from ancient Chinese into modern Chinese and therefore evaluates semantic interpretation rather than transcription alone. Reasoning-based QA targets implicit information such as facts, causality, or semantic relations. Knowledge-based QA targets objective knowledge expressed in ancient form, including time, places, objects, or medical terms. Linguistic variant QA is the most distinctive task, targeting language style, rhetorical methods, and stylistic features rather than only propositional content (Yu et al., 10 Sep 2025).
The annotation pipeline combines LLM pre-annotation with human revision. Qwen2.5-VL-72B is used to generate initial questions and answers aligned with each task and document category. For each page, the pipeline creates one QA pair for OCR, one for vernacular translation, and two QA pairs for each of the remaining three tasks. All generated results then undergo comprehensive manual review and revision, including OCR structure correction, translation quality improvement, and cross-task consistency checking; some LLM-generated QA pairs are filtered out during review (Yu et al., 10 Sep 2025).
The benchmark therefore encodes a particular view of document understanding. It is not limited to line transcription or visual layout parsing. Instead, it assumes that meaningful evaluation of ancient-document models must join page perception, classical-language interpretation, and historically grounded reasoning. This is consistent with broader work on ancient corpora in which annotation layers, linked data, and linguistic structure are treated as integral to scholarly use rather than optional post-processing (Celano, 2024).
4. Evaluation methodology and benchmark logic
AncientDoc uses different evaluation schemes for OCR and higher-level understanding. For page-level OCR, it reports character-level metrics: character error rate, character precision, character recall, and character F1. The appendix gives the following definitions for the first three:
The appendix also prints formulas for CharF1 and BS-F1 in malformed form, a detail that matters for careful reuse of the benchmark (Yu et al., 10 Sep 2025).
For vernacular translation, reasoning-based QA, knowledge-based QA, and linguistic variant QA, the benchmark uses CHRF++, BERTScore F1, and a human-aligned LLM judge. To calibrate the judge, the paper compares Qwen2.5-VL-72B, Gemini, Doubao, Qwen-Plus, and GPT-4o against human ratings on 50 QA pairs from each task, 250 in total, scored on a 0–10 scale. Alignment is assessed using Pearson, Spearman, Kendall, MSE, MAE, and Bias, and GPT-4o is selected as the final evaluator because it is closest to human scoring overall. In the reported comparison, GPT-4o has Pearson 0.846, Spearman 0.837, MSE 2.939, MAE 1.32, and the smallest bias magnitude among the listed candidates (Yu et al., 10 Sep 2025).
This evaluation design reflects a methodological judgment: exact string overlap is too brittle for ancient-language understanding, especially for vernacular translation where multiple acceptable modern renderings may exist. That judgment parallels a broader pattern in ancient-text computation. Text restoration work such as Pythia evaluates ranked restoration hypotheses because a single exact output can be too narrow a target (Assael et al., 2019). AncientDoc applies a comparable principle to multimodal page understanding by combining overlap-based metrics with a human-aligned semantic judge (Yu et al., 10 Sep 2025).
5. Empirical performance profile of current VLMs
The benchmark’s main empirical finding is that current VLMs remain far from robust on Chinese ancient documents, and that the ranking of models varies sharply by task. On page-level OCR, Gemini2.5-Pro is the strongest overall model, with CER 32.03, Character Precision 17.73, Character Recall 18.53, and Character F1 18.12. Doubao-V2 achieves a high Character F1 of 16.00 and the best GPT-4o score of 6.27, but its CER of 71.95 indicates weak character-faithful transcription. Among open models, Qwen2.5-VL-7B is notable with CER 35.47, Character F1 12.85, and GPT-4o 6.37, outperforming Qwen2.5-VL-72B on OCR (Yu et al., 10 Sep 2025).
On vernacular translation, Gemini2.5-Pro again leads by automatic semantic score, with BS-F1 72.5 and GPT-4o score 4.72. Qwen-VL-Max reaches BS-F1 71.03, and Qwen2.5-VL-72B 69.87. Yet the absolute judged quality remains low across the board: even the top model averages only 4.72 out of 10, and GPT-4o’s own translation score is reported as 0.92. The paper interprets this as evidence that classical Chinese understanding and punctuation restoration remain difficult, even for strong models (Yu et al., 10 Sep 2025).
On reasoning-based QA, Gemini2.5-Pro records the highest GPT-4o score at 7.76, while Qwen2.5-VL-72B reaches the highest BS-F1 at 71.4. Qwen2.5-VL-7B is also strong for its size, with BS-F1 69.96 and GPT-4o 6.44. On knowledge-based QA, GPT-4o leads by BS-F1 at 70.01, while Doubao-V2 and Gemini2.5-Pro tie for the highest GPT-4o-judged score at 7.36. On linguistic variant QA, GPT-4o reaches the highest BS-F1 at 64.58; Gemini2.5-Pro follows at 62.06 and has the top GPT-4o score among the main closed-source models at 5.92. A notable anomaly is InternVL2.5-2B, which reaches BS-F1 62.24 and exceeds many larger open models on this task (Yu et al., 10 Sep 2025).
The aggregate picture is not that one model dominates all dimensions, but that AncientDoc exposes separable competencies. OCR strength does not imply knowledge QA strength; reasoning performance does not guarantee stylistic sensitivity; and scaling alone does not resolve ancient-language variation. A plausible implication is that ancient-document understanding is better modeled as a bundle of coupled but distinct abilities rather than as a single DocVQA-style skill.
6. Position within the wider ancient-document research landscape
AncientDoc belongs to a much larger field of computational work on ancient materials, but its role is specific. It is a page-level benchmark for Chinese ancient books, whereas several adjacent projects address single-character evolution, decipherment, restoration, or corpus infrastructure.
For oracle-bone and ancient-character research, EVOBC provides a six-stage diachronic dataset of 229,170 images across 13,714 categories, linking oracle bone, bronze, seal, Spring and Autumn, Warring States, and clerical forms (Guan et al., 2024). OBSD treats oracle-bone decipherment as conditional image generation from OBS to modern Chinese characters, using localized structural sampling and zero-shot refinement (Guan et al., 2024). OracleSage instead proposes a cross-modal framework that combines hierarchical visual understanding with graph-based semantic reasoning, alongside the semantically enriched OracleSem dataset (Jiang et al., 2024). CipherOBS reframes oracle-bone decipherment from closed-set classification to dictionary-based retrieval over a synthetic OBS-style lexicon, achieving strong zero-shot Top- results on unseen characters (Wu et al., 1 Apr 2026).
AncientBench occupies a closer conceptual neighborhood. It evaluates comprehension of ancient characters across excavated and transmitted corpora using four dimensions—glyph, pronunciation, meaning, and contextual comprehension—and ten tasks including radical, phonetic radical, homophone, cloze, and translation (Zhou et al., 19 Dec 2025). AncientDoc differs in two ways: it uses full page images rather than character-centric prompts, and it evaluates multimodal models across OCR, translation, reasoning, knowledge, and stylistic variation (Yu et al., 10 Sep 2025).
This suggests that “AncientDoc” should not be conflated with all ancient-Chinese AI. It is one benchmark inside a layered ecosystem that includes digitization, glyph encoding, restoration, semantic annotation, translation, and decipherment. In that ecosystem, AncientDoc primarily tests the page-understanding layer.
7. Limits, misconceptions, and research implications
Several limitations are explicit or strongly implied in the benchmark design. The source images come mainly from Harvard Library holdings, so source diversity is narrower than the full universe of Chinese ancient documents. The corpus is weighted toward Ming and Qing printed materials and is about 97% regular script, which underrepresents more difficult handwritten or highly cursive sources. The annotation pipeline relies on LLM pre-annotation plus manual proofreading, but the paper does not report inter-annotator agreement, detailed annotator expertise, or transcription normalization rules. Evaluation relies on GPT-4o as a model judge, which is human-aligned but still not equivalent to direct expert adjudication (Yu et al., 10 Sep 2025).
A common misconception is that AncientDoc is simply an OCR benchmark for old books. The task design directly contradicts that reading. Only one of its five tasks is page-level OCR, and the benchmark’s distinctive contribution lies in vernacular translation, reasoning-based QA, knowledge-based QA, and linguistic variant QA (Yu et al., 10 Sep 2025). Another misconception is that strong modern document models should transfer automatically. The results show the opposite: models that perform well on mainstream modern benchmarks can degrade sharply when confronted with vertical layout, rare/traditional characters, absence of punctuation, and historically specific semantics (Yu et al., 10 Sep 2025).
The benchmark also implies several research directions, though these are better phrased as interpretations. Ancient-language-specific pretraining is likely relevant, given that AnchiBERT shows that continued pretraining on ancient Chinese corpora improves both understanding and generation over general Chinese BERT (Tian et al., 2020). Ancient-modern translation resources are also likely complementary, since clause-aligned corpora of 1.24M bilingual pairs have already shown utility for historical modernization tasks (Liu et al., 2018). More broadly, corpus infrastructures such as Opera Graeca Adnotata, Perseus 6, and AthDGC suggest that the long-term value of ancient-document benchmarks increases when they interoperate with standoff annotations, citation layers, and reusable philological pipelines (Celano, 2024).
In that sense, AncientDoc is best understood not as a terminal benchmark but as a diagnostic instrument. It demonstrates that Chinese ancient-document AI cannot be reduced to OCR, and that the central problem is multimodal historical understanding: reading complex pages, reconstructing classical semantics, accessing domain knowledge, and reasoning about variation in language and style.