Archaic Insertion Rate (AIR) in Historical OCR

• Archaic Insertion Rate (AIR) is a historical-fidelity metric that measures the rate of anachronistic character insertions in OCR outputs of 18th-century Russian texts.
• It complements standard OCR metrics by specifically identifying the erroneous inclusion of pre-Petrine characters, which standard measures like CER/WER might miss.
• Experimental comparisons show models such as GPT-4o frequently over-insert archaic characters, highlighting the need for period-aware evaluation in digital humanities.

Searching arXiv for the AIR source paper and closely related OCR-evaluation context. Archaic Insertion Rate (AIR) is a historical-fidelity metric for evaluating OCR output on historical documents, introduced in the context of LLM-based OCR for 18th-century Russian Civil font texts. It is designed to detect a failure mode that standard OCR metrics miss: the insertion of characters that are historically “too old” for the target document period. In the setting studied in "Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities" (Levchenko, 8 Oct 2025), AIR measures the rate at which model predictions contain archaic character insertions relative to the period-specific ground truth inventory, thereby operationalizing the phenomenon the paper describes as “over-historicization.”

1. Definition and scope

AIR is one of the paper’s two proposed “historical fidelity” metrics, paired with Historical Character Preservation Rate (HCPR) (Levchenko, 8 Oct 2025). Its purpose is narrowly defined: it captures the insertion of obsolete, pre-Petrine characters into OCR output for texts that belong to the 18th-century Russian Civil font tradition.

The metric is motivated by a period-specific mismatch. The corpus consists of post-reform Civil print, but still pre-1918 orthography. Accordingly, the expected character inventory is neither medieval Church Slavonic nor modern post-1918 Russian. AIR therefore targets errors in which a model introduces characters from an earlier orthographic layer than the source text actually uses, rather than merely making a generic transcription mistake (Levchenko, 8 Oct 2025).

A central point in the source paper is that AIR is historically aware. CER and WER are based on Levenshtein distance and treat errors more or less uniformly; AIR instead isolates a temporally specific class of errors. This makes it a diagnostic for historical bias rather than a general-purpose accuracy score (Levchenko, 8 Oct 2025).

2. Historical inventory and target period

AIR is measured against the 18th-century Russian Civil font period, specifically books published roughly 1752–1801, with most material from the 1780s and 1790s (Levchenko, 8 Oct 2025). The metric is meaningful only because the target historical inventory is fixed and period-specific.

Within that inventory, the paper identifies characters such as ѣ, і and/or ї, ѳ, ѵ, and terminal ъ as period-specific forms that may legitimately occur in the target corpus (Levchenko, 8 Oct 2025). By contrast, AIR is concerned with the appearance of forms that are not part of the expected Civil-font inventory for this period, including ѡ, ѧ, ѯ, ѱ, ѫ, and other pre-Petrine or Church Slavonic glyphs (Levchenko, 8 Oct 2025).

This distinction is essential. AIR does not treat all nonmodern characters as errors. Rather, it distinguishes between historically appropriate older orthography and anachronistic forms imported from an earlier textual layer. In that sense, AIR is not a modernization metric and not a paleographic similarity metric; it is a period-boundary metric. A plausible implication is that the metric is only transportable to other corpora if the evaluator can specify an equally precise inventory for the target historical stage.

3. Computation and operational interpretation

The paper does not provide a standalone symbolic formula for AIR in the excerpt, and no explicit LaTeX equation for AIR is shown (Levchenko, 8 Oct 2025). The metric is instead defined conceptually as the rate of insertion of obsolete, pre-Petrine characters in OCR output.

The computation is described procedurally. First, one chooses the target historical inventory for the corpus; in the paper, this is the 18th-century Russian Civil font inventory. Second, the page or line is transcribed with the model. Third, the prediction is scanned for archaic insertions, meaning characters that appear in the model output but are not appropriate for the target period, especially pre-Petrine or Church Slavonic letters. Fourth, the count is normalized, because AIR is reported as a proportion or percentage rather than a raw count (Levchenko, 8 Oct 2025).

The paper is explicit that AIR is computed over OCR output at the level of the historical character inventory in the transcription, not merely at the level of words or pages. The evaluation may be line-based or page-based depending on mode, but the metric itself concerns character occurrences in the predicted transcription (Levchenko, 8 Oct 2025).

An important nuance is that AIR is about insertion, not general mismatch. If a model replaces one valid 18th-century character with another valid 18th-century character, that is not AIR. AIR specifically captures the model adding character types from the wrong historical layer (Levchenko, 8 Oct 2025). Because the excerpt does not print the exact denominator, any stronger formalization would exceed the supplied evidence.

4. Relation to HCPR, CER, WER, and edit distance

AIR is designed to complement, not replace, standard OCR metrics. The paper pairs it with HCPR under the heading of “Historical Fidelity Metrics” (Levchenko, 8 Oct 2025). The two metrics are explicitly complementary: HCPR measures how well the model preserves legitimate period-specific characters, whereas AIR measures how often the model inserts illegitimate archaic characters.

This complementarity is clearest in cases where a model achieves acceptable aggregate OCR performance while still distorting the historical register of the source. A model can have decent CER but poor AIR if it transcribes most letters correctly while nevertheless over-inserting archaic forms (Levchenko, 8 Oct 2025). AIR therefore adds an axis of evaluation that CER and WER do not capture.

The paper also distinguishes AIR from edit distance and from case-related metrics. Edit distance aggregates insertions, deletions, and substitutions; AIR isolates one specific edit class, namely insertions of archaic characters. Case-insensitive CER/WER and Case Error Rate address casing behavior, whereas AIR concerns historical character inventory mismatches rather than capitalization (Levchenko, 8 Oct 2025).

This suggests that AIR functions most effectively as a targeted error decomposition tool. It does not summarize all historical inaccuracy, but it identifies a pattern that is especially consequential for diplomatic transcription and corpus building in digital humanities settings.

5. Error mode captured: over-historicization

The principal phenomenon AIR is intended to detect is “over-historicization,” which the paper defines through systematic insertion of archaic Slavonic characters that had already been eliminated by Peter the Great’s reforms (Levchenko, 8 Oct 2025). In the paper’s analysis, LLM-based OCR systems do not merely commit random visual confusions; some models drift into an orthographic regime older than that of the source document.

The paper emphasizes several dimensions of this failure mode. First, it captures temporal conflation: models may conflate pre-Petrine Church Slavonic, 18th-century Civil font, and modern Russian (Levchenko, 8 Oct 2025). Second, it captures a historically meaningful error pattern that standard OCR metrics can obscure. A transcription may be close in edit distance yet historically misleading if it introduces obsolete characters that change the period identity of the text.

For diplomatic transcription, this is not a minor issue. The objective is faithful preservation of the source rather than normalization. AIR therefore acts as an indicator of whether OCR output remains within the correct historical inventory boundary. A plausible implication is that AIR is especially relevant when OCR outputs are later used for philological, bibliographic, or corpus-linguistic analysis, where anachronistic character insertions could contaminate downstream inferences.

6. Experimental findings, limitations, and interpretation

The source paper does not provide a dedicated numerical AIR table in the excerpt, but it reports qualitative and model-specific archaic insertion patterns (Levchenko, 8 Oct 2025). OpenAI and Gemini models are described as especially prone to archaic insertions, and GPT-4o is described as inserting archaic characters in 59% of files. Claude-3.7 and Qwen2.5 are reported to have minimal archaic insertions relative to others. The paper also lists model-associated insertion tendencies, including forms such as ѡ, ѧ, ѫ, ѿ, and ѵ-related behaviors, depending on the model (Levchenko, 8 Oct 2025).

These findings should be interpreted with the metric’s assumptions and limitations in view. AIR depends on a carefully defined period inventory; it is corpus-specific; it does not capture deletions or substitutions of legitimate period forms; degraded scans and typographic variability can complicate annotation; and non-deterministic model outputs may cause slight variation in AIR across runs (Levchenko, 8 Oct 2025). The metric assumes that the ground truth is diplomatically accurate, that the target historical orthography is known and fixed, and that the designated archaic forms are unambiguously outside the expected Civil-font inventory for the dataset (Levchenko, 8 Oct 2025).

The recommended interpretation is correspondingly narrow and technical. Low AIR means the model respects the period boundary better. High AIR means the model tends to over-archaize the text. AIR should be interpreted together with CER/WER and HCPR: CER/WER indicate overall transcription proximity, HCPR indicates preservation of valid historical forms, and AIR indicates introduction of invalid historical forms (Levchenko, 8 Oct 2025).

In that sense, AIR is best understood as a period-fidelity diagnostic for historical OCR. It formalizes a specific kind of temporal bias in model outputs and provides a way to evaluate whether an OCR system not only transcribes historical text accurately, but also keeps that text within the correct historical orthographic layer.