ProHist-Bench: Historical Reasoning Benchmark

• ProHist-Bench is a research-grade benchmark designed to assess professional historical reasoning using expert-curated questions and detailed evidentiary rubrics.
• It organizes evaluation into four task families—term interpretation, fact QA, historical reasoning, and classical essay writing—anchored in Chinese Imperial Examination practices.
• The benchmark combines deep interdisciplinary collaboration and a rigorous multi-stage review process to ensure high-quality, citation-grounded historical analysis.

Searching arXiv for the specified paper to ground the article and citation. ProHist-Bench is a research-grade benchmark for evaluating whether LLMs can perform the kinds of professional historical reasoning that characterize expert practice. Rather than testing superficial recall or lexical competence, it targets evidentiary reasoning, sourcing, contextualization, synthesis, argumentation, historiographical judgment, academic expression, classical writing, and temporal reframing within a coherent historical domain: the Chinese Imperial Examination, or Keju. Anchored in an institutional complex that operated across roughly 1,300 years, ProHist-Bench comprises 400 expert-curated questions across eight dynasties and 10,891 fine-grained evaluation rubrics, and was introduced in “Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination” (Gao et al., 27 Apr 2026).

1. Conceptual orientation

ProHist-Bench is framed as a benchmark for “doing history” rather than merely recognizing historical facts. Its central premise is that professional historical inquiry requires the adjudication of conflicting sources, reconstruction of institutional change across periods, differentiation of meanings as terms shift over time, and the marshalling of claims with explicit evidentiary grounding. In this formulation, the benchmark is designed to operationalize the distinction between general historical knowledge and professional historical reasoning (Gao et al., 27 Apr 2026).

The benchmark is explicitly positioned against two clusters of existing history-related LLM benchmarks. One cluster consists of breadth-oriented knowledge tests, including HiST-LLM, C-Eval, and CMMLU. The other consists of lexical or language-focused tasks, including ACLUE, C-CLUE, WYWEB, ChroniclingAmericaQA, and M5HisDoc. These benchmarks are treated as useful for gauging general historical knowledge or ancient language processing, but as poor proxies for the core practices of historical scholarship. The paper emphasizes that even advanced LLMs still hallucinate and fail to resolve contradictory records, and ProHist-Bench responds by centering evaluation on evidentiary support, historiographical synthesis, and period-sensitive reasoning rather than answer surface form alone (Gao et al., 27 Apr 2026).

The Chinese Imperial Examination is selected because it is a well-documented, evolving institutional complex with ramifications for politics, society, economy, transportation, officialdom, and intellectual culture across East Asia. This makes it narrow enough to admit rigorous criteria while remaining broad enough to exercise concept definition, fact organization, cross-temporal comparison, and evidence-backed narrative construction. A plausible implication is that the benchmark’s domain specificity is meant to increase construct validity for historian-like reasoning, even at the cost of breadth.

2. Historical substrate and task taxonomy

The temporal scope spans eight dynasties in which Keju or analogous civil-service examinations were salient: Sui, Tang, Song, Liao, Jin, Yuan, Ming, and Qing. The benchmark characterizes this span as capturing institutional origins in the Sui–Tang period, codification and elaboration in the Song, distinctive adaptations in the non-Han regimes of Liao, Jin, and Yuan, and mature standardization and reform in the Ming–Qing period (Gao et al., 27 Apr 2026).

ProHist-Bench organizes evaluation into four task families. Term Interpretation (T1) concerns the precise definition of technical terms and institutional concepts. Fact QA (T2) concerns structured organization of the factual record. Historical Reasoning (T3) concerns comparative analysis, synthesis of viewpoints, and evidence-based argumentation. Celun (T4) concerns classical policy essay generation in eight-legged essay (baguwen) format under dynastic conventions and taboos. These tasks jointly cover concept definition, fact organization, historical comparison, evidentiary reasoning, comprehensive evaluation, viewpoint integration, academic expression, classical writing, and temporal reframing.

The dataset contains 400 expert-curated questions across these four tasks, covering 13 main topics and 31 sub-topics. Registration and Process accounts for 31.24% of items, Definition and Evolution for 12.38%, and Entry into Officialdom and Appointments for 11.43%. By task type, T2 is the largest at approximately 43%, followed by T3 at approximately 35% and T1 at approximately 22%; T4 is intentionally smaller at 10% because of its nuanced writing constraints. Difficulty is annotated as General versus Hard. Historical Reasoning has the highest proportion of hard items at 35.0%, whereas Term Interpretation has fewer hard items at 25.6%. Answers are long-form and source-grounded, with detailed citations, and penalty rubrics explicitly penalize missing citations when quoting classical texts (Gao et al., 27 Apr 2026).

The Celun component is especially distinctive because it imposes both stylistic and temporal constraints. A sample question asks the model to assume the role of a Qing dynasty candidate in the 46th year of Qianlong, write a baguwen, avoid taboo characters by using pinyin where necessary, limit the output to 700 words, and provide only the essay. The associated gold-standard outline requires canonical citations with accurate book and chapter references, Qing orthodoxy, policy prescriptions grounded in Qing institutional constraints, and avoidance of taboo terms.

3. Dataset construction and quality control

ProHist-Bench was produced through deep interdisciplinary collaboration among historians, sinologists, and NLP researchers. Six historians drafted 100 questions each from 125 authoritative sources, including ancient texts and archival compilations such as Qingdai Zhujuan Jicheng and Qingdai Keju Kaoshi Shulu, monographs by leading scholars such as Benjamin Elman, and CSSCI-indexed journal articles. LLMs were used to refine phrasing and stress-test prompts, but all reference answers were ultimately written sentence-by-sentence by historians because of model deficiencies in classical text completion, term interpretation, and event mechanisms (Gao et al., 27 Apr 2026).

Quality control proceeds through a four-stage pipeline. First, every item receives at least two independent reviews. Second, disagreements are adjudicated by a senior historian. Third, there is a global check by two senior historians. Fourth, repeated 5% random sampling is performed until the sampled batch reaches 100% accuracy, with any error triggering a full recheck. No inter-annotator agreement coefficients such as Cohen’s $\kappa$ or Krippendorff’s $\alpha$ are reported; instead, the benchmark relies on multi-stage human review, adjudication, and repeated sampling checks.

This construction strategy reflects the benchmark’s commitment to sentence-level reference quality and rubric precision. The emphasis on historian-authored answers, citation discipline, and repeated review indicates that annotation is treated not as a lightweight labeling task but as a form of domain-expert scholarly production. This suggests that ProHist-Bench is intended less as a broad crowdsourced benchmark than as a high-specificity instrument for expert-level evaluation.

4. Rubric system and scoring formalism

The benchmark’s core methodological novelty lies in its rubric framework: 10,891 question-specific criteria, or 27.23 per question on average, distributed across nine dimensions. These dimensions are R1 Concept Definition (2 points), R2 Fact Organization (3 points), R3 Historical Comparison (3 points), R4 Evidentiary Reasoning (4 points), R5 Comprehensive Evaluation (1 point), R6 Viewpoint Integration (5 points), R7 Academic Expression (5 points), R8 Classical Writing (3 points; baguwen structural completeness), and R9 Temporal Reframing (9 points total: 4 style + 5 composition, with major penalties for period violations) (Gao et al., 27 Apr 2026).

Rubric density varies by task. T1 has 1,650 criteria with an average of 18.33 per question; T2 has 3,963 with an average of 26.42; T3 has 4,325 with an average of 36.04; and T4 has 953 with an average of 23.83. Fact Organization (R2) is the largest category at 59.24%, followed by Academic Expression (R7) at 10.72% and Concept Definition (R1) at 9.20%. This distribution reflects the benchmark’s view that structured factual narration is central to historical practice.

Penalty rubrics encode errors treated as unacceptable across tasks. These include fabricated sources ($-5$), missing citations when quoting classical texts ($-3$), chronological conversion errors ($-3$), core concept errors ($-5$), inappropriate academic formulation ($-3$), and non-academic style ($-1$). In the abbreviated sample rubric, R1 requires accurate definition of Gongshi in the Ming–Qing context and distinction from Juren or Jinshi where applicable; R2 requires chronological narration of the transition from Shengshi to Huishi standardization with dynastic anchors and procedures; R4 requires at least two concrete citations to primary or archival sources in dynastic context.

Scoring is based primarily on Rubric Score (RS), which aggregates weighted rubric hits and penalties, normalizes by total possible positive points, and clips at zero:

$$RS = \max \left(0,\frac{\sum (I_b \cdot w_b)+\sum (I_p \cdot w_p)}{\sum w_b}\right)$$

where $I_b$ and $\alpha$0 are binary indicators for bonus and penalty criteria and $\alpha$1 denotes per-criterion weights. The paper also reports BLEU, ROUGE, and BERTScore for T1–T3, together with macro-average and normalized weighted score formulas, but RS is treated as the primary measure of professional historical capability.

5. Evaluation design and empirical findings

Evaluation uses an LLM-as-a-Judge pipeline. Judge selection is based on a preliminary consistency study against historian annotations on a 50-instance sample, comparing six candidate judges using the Pearson correlation coefficient at both rubric level and answer level. DeepSeek-R1 achieved the highest average consistency, approximately $\alpha$2, and was chosen as the judge. Reproducibility is supported through fixed prompts and deterministic inference, with hyperparameters fixed and randomness eliminated. Standardized prompts are used across models, and strategy variants include historian role-playing, “professional” prompting, chain-of-thought, and retrieval-augmented generation (Gao et al., 27 Apr 2026).

Eighteen LLMs are evaluated. The closed-source set includes Anthropic Claude-Sonnet-4.5-Thinking, OpenAI GPT-5.2, GPT-5.2-Thinking, GPT-o3-2025-04-16, Google DeepMind Gemini-3-Pro-Preview and Gemini-3-Pro-Preview-Thinking, and Alibaba Qwen3-Max. The open-source set includes Meta Llama-4-Scout-17B-16E, OpenAI gpt-oss-120b and gpt-oss-20b, Moonshot AI Kimi-K2-Thinking, Zhipu GLM-4.6-Thinking, Alibaba Qwen3-14B/32B/235B-A22B-Thinking, and DeepSeek V3.2/V3.2-Thinking/R1-0528.

Aggregate results on T1–T3 show a substantial proficiency gap. The top RS cluster consists of Gemini-3-Pro-Preview at approximately 29.29, Qwen3-235B-A22B-Thinking at approximately 28.14, DeepSeek-R1-0528 at approximately 26.87, GLM-4.6-Thinking at approximately 24.32, and Kimi-K2-Thinking at approximately 22.79; most other models lie below 15. Representative model results include Gemini-3-Pro-Preview with RS approximately 29.29 and BERTScore approximately 73.97, GPT-o3 with RS approximately 14.66, GPT-5.2-Thinking with RS approximately 14.08, Claude-Sonnet-4.5-Thinking with RS approximately 12.99, Qwen3-Max with RS approximately 17.71, DeepSeek-V3.2 with RS approximately 18.77, Llama-4-Scout-17B-16E with RS approximately 2.72, and gpt-oss-120b with RS approximately 10.75. BERTScore values are often in the 71–75 range, but these correlate weakly with RS and do not adequately capture evidentiary rigor or historiographical synthesis.

Human baselines sharpen the interpretation. Closed-book historians average approximately 29.33, and open-book historians average approximately 60.67. Top LLMs, such as Gemini-3-Pro-Preview at approximately 34.17 and Qwen3-235B-Thinking at approximately 33.97, surpass closed-book humans on average but remain far below open-book experts, especially on precision-critical tasks T1 and T2. On T4, RS is the only reported metric because stylistic and temporal fidelity are central; Qwen3-Max is reported as the top performer, while some models fail entirely because of taboo violations or baguwen structure noncompliance.

Capability analysis identifies systematic weaknesses. Viewpoint Integration (R6) has the lowest positive hit rates, typically 0.02–0.12 and with maxima around 0.23. Historical Comparison (R3) and Evidentiary Reasoning (R4) are also weak, with frequent conflation of institutional differences across dynasties and incomplete evidentiary chains. By contrast, R8 Classical Writing and R5 Comprehensive Evaluation achieve relatively higher hit rates, indicating comparatively stronger rule compliance and coherent narrative generation. Prompting strategies also matter: “professional” prompting and historian role-playing consistently outperform chain-of-thought, while retrieval-augmented generation underperforms, likely because high-quality ancient historical data in common retrieval corpora are scarce and noisy. Varying the number of retrieved documents, with $\alpha$3, shows sensitivity to noise, and additional documents often degrade reasoning.

6. Release, applications, limitations, and significance

ProHist-Bench is released at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench. The released artifacts include the questions, reference answers, rubrics, topic taxonomy, and evaluation scripts for RS and text-generation metrics. Licensing is restricted to non-commercial academic use, and the accompanying ethical guidance emphasizes the interpretive nature of historical materials and responsible use. The benchmark does not specify train/dev/test splits; in practice, the full set is commonly treated as test-only for evaluation, with subset splits reserved for internal development (Gao et al., 27 Apr 2026).

Potential uses include evaluation of domain-specific reasoning, training or fine-tuning specialized historical models, including classical Chinese studies and Keju-specific RAG, and the advancement of computational history through a structured testbed for evidentiary and historiographical reasoning. Recommended baselines include closed-book prompting, professional role prompts, and cautious RAG experiments with attention to retrieval corpus quality.

The benchmark also states several limitations. Coverage is biased toward the Qing because of documentation richness. There are language-specific issues involving Classical Chinese versus modern forms. Rubric subjectivity is inherent, though mitigated through multi-stage review and adjudication. Future work may expand to other historical domains, enrich retrieval corpora with curated primary sources, and explore multimodal artifacts.

Taken together, ProHist-Bench marks a shift from breadth-of-knowledge testing toward rubric-driven evaluation of historical research practice. Its primary contribution is not simply a new question set, but a formalized framework for scoring evidentiary support, period fidelity, comparison, and historiographical integration inside a historically coherent domain. This suggests that its broader significance lies in making professional historical reasoning legible to benchmark-based evaluation, while also showing that current LLMs remain far from autonomous research standards when explicit source-grounding, multi-source synthesis, and temporal fidelity are required.