Historical Character Preservation Rate (HCPR)

• Historical Character Preservation Rate (HCPR) is a metric that measures the accurate preservation of period-specific graphemes in 18th-century Russian Civil font OCR outputs.
• It complements general OCR metrics like CER and WER by isolating temporal bias, such as modernization or archaic insertions, enabling precise historical fidelity assessment.
• The evaluation framework pairs HCPR with Archaic Insertion Rate (AIR) to support rigorous diplomatic transcription and model comparisons for historical texts.

Historical Character Preservation Rate (HCPR) is a targeted fidelity metric for historical OCR introduced for evaluating LLM-based transcription of 18th-century Russian Civil font texts. It quantifies whether a system preserves the specific graphemes characteristic of the target historical period, addressing a failure mode that conventional metrics such as Character Error Rate (CER) and Word Error Rate (WER) do not isolate: temporal bias, including modernization of period-specific letters and insertion of characters from earlier orthographic stages. In the reported framework, HCPR is paired with Archaic Insertion Rate (AIR) and embedded in a broader evaluation methodology for contamination control, stability testing, and diplomatic-transcription assessment (Levchenko, 8 Oct 2025).

1. Definition and evaluative purpose

HCPR is introduced as a metric for historical OCR that directly measures historical fidelity rather than overall edit distance. In the 18th-century Russian Civil font setting, the principal risk is that multimodal LLMs either modernize away period-specific letters or “over-historicize” by inserting graphemes associated with earlier, pre-Petrine or Church Slavonic stages. The metric therefore focuses on whether historically salient letters in the ground truth survive in the OCR output after alignment (Levchenko, 8 Oct 2025).

The framework does not provide an explicit LaTeX formula for HCPR. Instead, HCPR is described conceptually as the proportion of ground-truth instances of period-specific characters that appear correctly in the OCR output after character-level alignment. The published tables report “Hist. Char. Error (%)” rather than a formal HCPR expression; HCPR is best understood as the complement of that error rate, so higher HCPR corresponds to fewer errors on historical characters (Levchenko, 8 Oct 2025).

This design is motivated by the requirements of diplomatic transcription. In that regime, exact preservation of orthography, spacing, hyphenation, punctuation, and original typographic errors is required. Period-specific graphemes are therefore not peripheral details but core objects of evaluation. A transcription may achieve an acceptable CER while still erasing the letters that define the orthographic period, and HCPR was proposed to make that discrepancy measurable.

2. Character inventory and temporal error taxonomy

The HCPR framework is tied to an explicit historical character inventory. For the 18th-century Russian Civil font domain examined in the study, the core set tracked by HCPR is stated as the period-specific characters “(і/ї, Ñ£, ъ).” Additional Civil-font characters listed in prompts as “special characters” to preserve include “ѳ, ѵ, and case,” but the HCPR definition itself is described as focusing on “і/ї, Ñ£, ъ” (Levchenko, 8 Oct 2025).

The complementary error category is the insertion of archaic characters that should not appear in Civil-font prints of this period. The prompting context explicitly warns models about the “absence of Church Slavonic letters (Ñ¡, ѧ, ѱ, etc.)” in Civil-font output. The error-analysis table further lists top archaic insertions such as “ѧ (little yus), Ñ¡ (omega), ꙋ (monograph uk), Ñ¿ (ot), ꙗ (iotified a), Ñ¥ (iotified e).” The evaluation treats clear Church Slavonic-only letters, including “ѧ, Ñ¡, ꙋ, Ñ¿,” as anachronistic insertions (Levchenko, 8 Oct 2025).

The transcription regime preserves case and punctuation, and the study reports case-insensitive CER/WER separately while also introducing Case Error Rate (CaseER). On that basis, the framework indicates that HCPR is case-sensitive for the tracked graphemes unless otherwise stated. Diacritics and combining marks hallucinated by models are treated as errors when absent in Civil-font sources; no separate diacritic-normalization regime is specified for HCPR.

A central distinction in the framework is therefore between two forms of temporal bias. Low HCPR reflects loss, substitution, deletion, or modernization of legitimate Civil-era graphemes. High AIR reflects insertion of excluded earlier-era forms. Together they operationalize period fidelity in a way aggregate edit-distance measures do not.

3. Computation and operational interpretation

The computation procedure is described at a high level. Inputs are the ground-truth diplomatic transcription and the OCR output. Character-level alignment is performed using Levenshtein-style dynamic programming, as in CER/WER computation, so that insertions, deletions, and substitutions become explicit. All occurrences in the ground truth of the period-specific set used in HCPR, “{і, ї, Ñ£, ъ},” are extracted and evaluated (Levchenko, 8 Oct 2025).

A ground-truth historical character counts as preserved if the aligned OCR character at that position is exactly the same character, including case. Substitutions, deletions, and modernizations are counted as non-preservation. The study gives explicit examples of non-preservation, including “ї→і,” “ъ→ь,” and “Ñ£→е.” It reports the result as “Hist. Char. Error (%)” by model and document rather than as a standalone HCPR score.

An illustrative worked example is given using the ground truth line “... съѣздъ и съѣлъ ...” and OCR output “... сездъ и съелъ ...”. In that example, the HCPR set contributes four tracked ground-truth instances: two occurrences of “ъ (ъ)” and two of “Ñ£ (ѣ).” Only the final terminal “ъ” is preserved. Preserved historical characters are therefore 1 out of 4, yielding a conceptual HCPR of 25%, while “Hist. Char. Error (%)” is 75% (Levchenko, 8 Oct 2025).

The framework does not provide pseudocode or a confusion matrix. Error behavior is instead summarized through qualitative examples and a table of common substitutions and archaic insertions. This suggests that HCPR is intended as a focused diagnostic layered on top of standard alignment-based OCR evaluation rather than as a replacement for CER/WER.

4. Relation to CER, WER, AIR, and diplomatic transcription

The main methodological claim associated with HCPR is that CER and WER are insufficient in this setting. CER/WER conflate period-specific preservation with general orthographic correctness. The study documents cases in which models achieve low CER but substantially higher historical-character error, indicating poor fidelity to 18th-century orthography despite good aggregate recognition performance (Levchenko, 8 Oct 2025).

AIR was introduced to expose the complementary failure mode of over-historicization. It measures insertion of obsolete pre-Petrine characters from an explicitly excluded archaic set. The study does not provide a formal LaTeX formula for AIR either, but operationalizes it by detecting archaic letters and reporting their insertion prevalence. One reported finding is that “GPT-4o insert[s] archaic characters in 59% of files,” making AIR a document-level indicator of anachronistic output (Levchenko, 8 Oct 2025).

The framework therefore distinguishes three evaluative dimensions:

Metric	What it captures	What it can miss alone
CER/WER	General edit distance accuracy	Temporal bias and period fidelity
HCPR / “Hist. Char. Error (%)”	Preservation of Civil-era graphemes	Archaic insertions outside the tracked set
AIR	Insertion of excluded archaic letters	Loss or modernization of legitimate historical characters

The study’s examples of dominant substitution errors further clarify the diagnostic role of HCPR. Reported error arrows include:

• “ї~$\rightarrow$~і,\ ъ~$\rightarrow$~ь,\ т~$\rightarrow$~ш”
• “ъ~$\rightarrow$~Ъ,\ ї~$\rightarrow$~і,\ ь~$\rightarrow$~ъ”
• “ї~$\rightarrow$~і,\ т~$\rightarrow$~п,\ ъ~$\rightarrow$~ь”

Not all listed substitutions are themselves historical-character substitutions in the narrow HCPR sense, but they show the interaction between period-specific grapheme confusion and general OCR confusions. This suggests that temporal bias is only partly reducible to standard recognition noise; some errors are specifically period-indexed.

5. Empirical behavior across multimodal LLMs

The reported experiments evaluate 12 multimodal LLMs on 18th-century Russian Civil font documents, with “Hist. Char. Error (%)” summarized for full-page mode. Interpreting HCPR as the complement of that error yields the following model-level values (Levchenko, 8 Oct 2025).

Model	Hist. Char. Error (%)	Conceptual HCPR
Gemini-2.5-Pro	9.83	≈ 90.17%
Gemini-2.5-Flash	12.86	≈ 87.14%
Claude-3.5	15.24	≈ 84.76%
Claude-3.7	15.29	≈ 84.71%
Qwen-2.5-VL	16.40	≈ 83.60%
GPT-4.1	16.94	≈ 83.06%
OpenAI o4-mini	18.38	≈ 81.62%
GPT-4o	20.70	≈ 79.30%
Llama-4 Maverick	22.33	≈ 77.67%
Gemini-2.0	32.00	≈ 68.00%
Llama-4 Scout	42.23	≈ 57.77%

A general observation reported in the study is that “All models showed higher error rates for historical character preservation than for general character recognition.” One example is Gemini-2.5-Pro in full-page mode, which achieves CER $= 3.36\%$ while still showing “Hist. Char. Error (%)” $\rightarrow$0. This indicates that historically critical letters are materially more error-prone than average characters (Levchenko, 8 Oct 2025).

The study also identifies model-specific insertion patterns. GPT-4o and GPT-4.1 show archaic insertions such as “Ñ¡, ѧ, ꙋ, Ñ¿, ꙗ,” with common errors “ї~$\rightarrow$1~і, ъ~$\rightarrow$2~ь, т~$\rightarrow$3~ш.” The model “o4-mini” is associated with “ѧ, Ñ­, Ñ¥” and common errors “ї~$\rightarrow$4~і, ъ~$\rightarrow$5~ь, Ñ£~$\rightarrow$6~е.” Gemini-2.5-Flash shows “ѧ, Ñ¥” with common errors “ї~$\rightarrow$7~і, т~$\rightarrow$8~ш, ъ~$\rightarrow$9~ь.” By contrast, Claude-3.7 and Qwen2.5 are described as having minimal archaic insertions, though they still exhibit period-specific confusions such as “ъ~$\rightarrow$0~Ъ” for Claude and “т~$\rightarrow$1~п” for Qwen (Levchenko, 8 Oct 2025).

Over-historicization is most explicit in the OpenAI models. The statement that GPT-4o inserts archaic characters in 59% of files shows that acceptable or competitive OCR performance can coexist with substantial AIR. This is the central empirical rationale for reporting HCPR and AIR jointly rather than relying on CER/WER alone.

6. Experimental protocol, robustness, and limitations

The HCPR framework is embedded in a broader protocol designed to reduce contamination risk and support metric reliability. The dataset was built from “previously unpublished, limited-access Civil-font prints kept strictly offline during known LLM pretraining periods.” Ground truth was produced through a disciplined pipeline consisting of layout detection, fine-tuned TrOCR for initial transcriptions, and “100% manual correction to diplomatic guidelines.” Audited accuracy on a stratified sample is reported as “line-level exact match 98.6% and character-level accuracy 99.93%” (Levchenko, 8 Oct 2025).

The corpus consists of “1,029 pages from 428 unique books (1752–1801), Civil font prints across fiction, religion, history, science, etc.,” with low-resolution images below 150 ppi excluded. Ground truth comprises “28,657 lines, 146,690 words,” prepared to diplomatic standards preserving orthography, hyphenation, spacing, and original errors. The study notes that this corpus design makes HCPR meaningful because the ground truth contains many instances of the tracked historical characters, while diverse page conditions stress OCR systems in ways that can depress HCPR (Levchenko, 8 Oct 2025).

Stability testing was performed on “20 documents processed daily over seven days.” Daily variation was summarized by coefficient of variation (CV) of word accuracy. Gemini-2.5-Pro had the highest reported stability, with “CV 0.037” and mean word accuracy “0.9620,” and “No model’s daily performance deviated by more than one standard deviation from the previous day.” This supports the use of repeated measurements when comparing HCPR and AIR across non-deterministic LLM runs (Levchenko, 8 Oct 2025).

The study also evaluates post-OCR correction and reports that it “degraded rather than improved performance in text-only mode,” while even image+text correction did not exceed direct OCR quality. In historical-fidelity terms, the reason given is that models “re-decoded” rather than applied minimal edits, often increasing substitutions or modernizations that reduce HCPR and increasing anachronistic insertions that raise AIR. A common misconception that post-correction is automatically beneficial is therefore not supported in this setting (Levchenko, 8 Oct 2025).

Several limitations are explicit. The dataset is specific to “second-half 18th-century Russian Civil-font print,” so HCPR behavior may not generalize to other scripts or periods without adjusting the historical set. Rare annotation errors may occur in visually ambiguous regions, and scan degradation or typography variability can confound recognition, including confusions such as “т vs ш.” LLM non-determinism remains a validity threat despite the reported stability measurements.

Practitioner guidance follows directly from these results. The recommendations are to measure historical-character error or HCPR and AIR in addition to CER/WER, prefer models with both strong overall accuracy and low historical-character error, avoid text-only post-OCR correction workflows, use period-aware prompts listing allowed and disallowed characters, and pilot on “10–20 pages” to compile a collection-specific grapheme inventory before scaling (Levchenko, 8 Oct 2025). The paper does not define explicit HCPR acceptance thresholds. This suggests that HCPR functions primarily as a comparative and diagnostic metric whose interpretation depends on corpus, script, and period-specific grapheme inventory rather than on a universal cutoff.