Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

Abstract: While LLMs have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.

• The paper introduces a novel benchmark, ProHist-Bench, with 10,891 rubric criteria that rigorously operationalizes historiographical skills over 1,300 years of Chinese history.
• It employs a multi-dimensional, rubric-driven methodology and an LLM-as-a-Judge paradigm to reveal significant gaps in evidence-based reasoning and context-sensitive argumentation.
• Experimental results show that even state-of-the-art LLMs significantly underperform compared to human historians in complex, integrative research tasks.

Evaluating LLMs as Professional Historians: A Critical Appraisal of ProHist-Bench

Introduction

"Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination" (2604.24690) introduces ProHist-Bench, a domain-specific, fine-grained benchmark designed to rigorously assess the professional historical research abilities of modern LLMs. Grounded in the context of the Chinese Imperial Examination (Keju)—a complex socio-political institution with over 1,300 years of influence—this work systematically operationalizes historiographical skills, evaluates 18 models, and critically examines existing methodological gaps in LLM evaluation protocols for humanistic domains.

Limitations of Existing Historical Benchmarks

Current benchmarks for LLM historical performance largely test basic factual recall, lexical comprehension, or translation of historical texts. They fail to operationalize higher-order skills such as evidentiary reasoning, context-dependent argumentation, integration of scholarly perspectives, and historical comparison under shifting conceptual frameworks. This deficit leads to systematic overestimation of LLMs’ professional competence and fails to expose persistent issues such as hallucinations or inability to resolve conflicting sources.

ProHist-Bench: Scope and Methodology

ProHist-Bench is positioned as a multi-dimensional, rubric-driven evaluation pipeline. Through close collaboration between AI researchers and domain historians, the benchmark delivers 400 manually constructed questions spanning Term Interpretation, Fact QA, Historical Reasoning, and "Celun" (historically faithful policy essay generation) (T1–T4), with coverage across eight dynasties and over 1,300 years of Chinese history.

A core methodological advance is the introduction of 10,891 task-specific, fine-grained rubric criteria, systematically distributed across nine macro-capabilities (e.g., concept definition, evidentiary reasoning, temporal reframing, academic expression, etc.). This rubric framework is explicitly designed to assess not just static knowledge, but also the inferential, comparative, and narrative-building skills foundational to professional history as an academic discipline.

Figure 1: Overview of the expert-driven dataset construction pipeline and validation stages for ProHist-Bench.

The dataset’s structure ensures comprehensive topic and period coverage, as illustrated by hierarchical taxonomies and fine-grained topic distribution.

Figure 2: The hierarchical taxonomy of the ProHist-Bench topic framework, decomposing 13 main topics into 31 sub-topics for systematic coverage.

Automated Expert-Level Metric Design

A critical innovation is the deployment of an LLM-as-a-Judge paradigm with empirical validation against professional historian annotation. DeepSeek-R1 was found to correlate highly (Pearson correlation ~0.77) with expert rubric-level and answer-level evaluations, thereby enabling reproducible, efficient model assessment with minimized human bias.

Figure 3: Consistency of LLM judge models with manually crafted (human) ground truth, establishing the reliability of DeepSeek-R1 for automated scoring.

Rubric Score (RS) is introduced as the central metric, aggregating weighted positive and penalty rubric items. The rubric schema penalizes hallucinations, core factual errors, non-academic style, and period-inappropriate formulations. This metrics pipeline allows for nuanced capability decomposition and diagnostic error analysis.

Experimental Results and Capability Analysis

General Findings: Even the strongest LLMs underperform by substantial margins compared to human scholars. State-of-the-art models (Gemini-3-Pro, Qwen3-235B) rarely exceed RS scores of 30 on research tasks, with most remaining well below 15, exposing a significant gap between general LLM competence and professional historian proficiency.

Figure 4: Main results across T4 (Celun) research tasks—most LLMs cluster at low Rubric Scores despite large parameter counts and closed-source training.

Performance in factual tasks (e.g., Term Interpretation, Fact QA) shows moderate correlation with scale and Chinese corpus training, but deep reasoning and evidence-based argumentation remain fundamentally weak. Advanced prompting (role-play, professional identity cues) yields modest gains, while chain-of-thought and retrieval strategies offer little improvement in low-resource or highly specialized topics due to lack of high-quality retrieval context and excessive noise.

Capability-Level Heatmap: The disaggregated rubric analysis reveals pronounced weaknesses in R3 (comparison), R4 (evidentiary reasoning), and especially R6 (viewpoint integration), with hit rates rarely exceeding 0.23 even among top-tier models.

Figure 5: Left—Positive rubric hit rate across capabilities; green cells are rare. Right—Penalty rubric hit rate, measuring systematic defects such as hallucinations and factual mistakes.

Human-Like Reasoning Gaps: Side-by-side evaluation with closed-book and open-book human experts (n ≈ 20) decisively demonstrates the remaining shortfall. While closed-book historians can be outperformed on basic recall, open-book experts dominate in any context requiring multi-source synthesis, evidence chaining, or nuanced reinterpretation—skills where LLMs’ inability to resolve contradictory evidence and lack of actual research grounding is exposed.

Figure 6: Performance comparison between open/closed-book human experts and SOTA LLMs: SOTA models approach closed-book human recall but cannot match open-book evidence-based research.

Difficulty and Temporal Transfer: LLM performance collapses on "hard" tasks and in low-resource historical periods (e.g., Liao, Jin), indicating substantial limitations in scaling historical inference and domain adaptation.

Figure 7: Heatmap showing severe performance drops for non-mainstream dynasties, correlating with data scarcity and limited corpus coverage.

Qualitative Error Analysis: Case studies confirm that LLMs hallucinate plausible but incorrect content, struggle with the evolution of terms/concepts across dynasties, and frequently violate hard constraints (e.g., taboo names in Celun tasks)—all hallmarks of non-professional reasoning (Figure 8).

Figure 8: LLM hallucinations in specialized historical contexts, persistent even among advanced models.

Impact of Prompt Engineering and RAG

Prompt engineering (role or expert identity) helps prime LLMs for more formalized output but does not materially shift capabilities in critical, evidence-based reasoning. The RAG pipeline, even with carefully curated retrievals from processed Chinese Wikipedia and external sources, generally introduces more noise than value for tasks governed by low-prevalence, expert-level context—a finding robust across retrieval depths.

Theoretical and Practical Implications

The study convincingly demonstrates that state-of-the-art LLMs, even at trillion-parameter scale and with sophisticated Chinese-centric pretraining, cannot presently match the professional workflow of historical research. Their evidence synthesis is shallow, their comparative reasoning is brittle, and their handling of controversial or interpretively open questions remains unsatisfactory.

On the theoretical front, ProHist-Bench provides a model for operationalizing professional domain evaluation via large-scale, fine-grained rubrics—translatable to other low-resource, evidence-driven knowledge work such as legal or scientific reasoning. Practically, it provides a baseline for LLM developers, highlighting the necessity of integrating explicit temporal/contextual awareness, interpretive evidence mapping, and perhaps genuine knowledge-grounded reasoning architectures or hybrid neural-symbolic methods to bridge current gaps.

Future Prospects

Advances will require LLM architectures and training regimes that focus on deep multi-source grounding, explicit ambiguity modeling, temporal and cultural alignment, and robust mechanism design for hallucination avoidance. ProHist-Bench sets a high bar for such innovations: progress on this benchmark will likely be indicative of broader gains in machine-based scholarly reasoning in complex, human-constructed domains.

Conclusion

"Can LLMs Act as Historians?" establishes that, despite advances in parameter count and corpus coverage, current LLMs are not professional historians: they cannot consistently execute the core tasks of evidence-based, context-sensitive, and interpretive reasoning that undergird academic history. ProHist-Bench stands as both a diagnostic and a challenge benchmark, providing the field with a rigorous tool for measuring, and eventually enabling, the emergence of genuinely research-capable AI systems in the humanities.