Open Korean Historical Corpus (OKHC)

• Open Korean Historical Corpus is a large-scale, diachronic collection of Korean texts spanning over 1,300 years and diverse genres and scripts.
• It supports quantitative linguistic analysis and NLP applications by tracking script transitions and language evolution from Idu and Hanja to modern Hangul.
• Standardized metadata and extensive annotations facilitate precise historical language studies and digital humanities research.

The Open Korean Historical Corpus (OKHC) is a large-scale, multi-genre, diachronic collection of digitized Korean texts spanning over 1,300 years, with a primary focus on pre-modern, early modern, and modern Korean, as well as adjacent languages and scripts that have historically shaped linguistic practice in Korea. Designed as a foundational resource for NLP, historical linguistics, and digital humanities, OKHC and related datasets consolidate materials formerly scattered across isolated institutional archives, covering a range of writing systems (Hanja, Hangul, mixed script, Idu, Japanese) and genres (annals, diaries, literature, news). This corpus enables the quantitative study of language change, script transition, and named entity recognition across centuries, while also supporting the training and evaluation of LLMs tailored for under-represented or morphologically complex periods of Korean (Song et al., 28 Oct 2025).

1. Composition and Structure of the Open Korean Historical Corpus

OKHC incorporates texts from 19 primary sources, yielding more than 17.6 million documents and 5.1 billion tokens. Its chronological range extends from the 7th century to 2025, and its coverage spans Middle Korean (10th–16th c.), Early Modern Korean (17th–19th c.), Modern Korean (late 19th c.–present), Classical Chinese (Hanja), North Korean corpora (post-1945), and selected Japanese texts from the colonial period. The primary genres include royal annals, daily diaries, literary collections, government documents, modern newspapers, and scholarly works.

Key writing systems represented are: Korean-style Sinitic (Idu, Gugyeol, Hyangchal), Sino-Korean mixed script (Hanja + Hangul), Old Hangul (archaic jamo), Modern Hangul (standardized), and Japanese kana (Song et al., 28 Oct 2025). File formats are provided as JSON Lines, with entries containing text plus exhaustive metadata (year, source, language, script, copyright, and analytics fields).

The table below summarizes several major sources:

Abbreviation	Corpus Name (English)	Publication Years	Documents	License
AJD	Annals of the Joseon Dynasty	1392–1928	413,131	CC0/Public
DRS	Diaries of the Royal Secretariat	1623–1910	1,792,187	CC0/Public
KLC	Korean Literary Collections	886–1933	652,405	CC0/Public
NNL	Naver News Library	1920–1999	13,536,494	CC BY-NC/PD
KNA	Korean Newspaper Archive	1883–1952	364,409	CC0/Public

Restricted or copyrighted sources are included only with URLs and metadata (Song et al., 28 Oct 2025), ensuring legal compliance.

2. Data Acquisition, Licensing, and Processing

Texts are acquired via a combination of direct bulk downloads, institutional data partnerships, and large-scale web scraping with frameworks such as BeautifulSoup and HTTPX. Pre-1963 materials are generally public domain, while later texts are distributed under Creative Commons BY-NC 4.0 or via research-only institutional licenses (Yoo et al., 2022, Song et al., 28 Oct 2025). OCR transcriptions, UTF-8 normalization (Unicode NFKC), whitespace regularization, removal of control characters, and boilerplate cleaning form the standard preprocessing pipeline. Documents are filtered for high-noise or corrupted encodings based on document-level heuristics.

All entries are normalized and provided in machine-parseable JSON, with fields for document ID, text, year, language, script, source, corpus, copyright/licensing status, URL, and metadata, facilitating corpus querying by period, language variant, or script (Song et al., 28 Oct 2025).

3. Historical Corpora: Key Subsets and Task-Oriented Datasets

Three major high-value corpus subsets have driven recent historical-Korean NLP research:

1. Annals of the Joseon Dynasty (AJD): 1392–1897, 27 royal reigns, ≈230k entries, extensively annotated for dates, major/minor topics, and named entities (PERSON, LOCATION) by the Institute for the Translation of Korean Classics (Yoo et al., 2022).
2. Diaries of the Royal Secretariat (DRS/SeungJeongWon Ilgi): 1623–1910, ≈1.9 million diary entries, annotated both for entity classes and phrase/sentence markers by professional historians, with parallel "marked" (punctuated) and "unmarked" (plain) UTF-8 text versions (Kim et al., 2023).
3. Daily Records of the Royal Court and Important Officials (DRRI): 18th–20th centuries, ≈426k daily records, supports summary retrieval, chronological attribution, and zero-shot NER experimentation (Yoo et al., 2022).

OKHC unifies these with large literary, administrative, and modern news materials to support diachronic and cross-domain studies (Song et al., 28 Oct 2025).

4. Quantitative Linguistic Analyses Enabled by OKHC

OKHC enables high-resolution quantitative analysis of script and lexical change over centuries. Three main analyses are presented in Song et al. (2025) (Song et al., 28 Oct 2025):

• Idu Usage Trends: Using Aho–Corasick dictionary-based matching, the Idu (Korean-style Sinitic) marker ratio, $$\text{count}_{\text{Idu markers}}/\text{count}_{\text{Hanja characters}}$$, is computed by period and document length. Peak Idu usage occurred in the 1860s, declined sharply post-1890 in response to the Kabo Reform (1894) and rise of Hangul orthography.
• Hanja-to-Hangul Script Shift: Character-level proportions $P_\mathrm{Hanja}(t)$ and $P_\mathrm{Hangul}(t)$ are calculated from Unicode-regex counting. A logistic model fit characterizes the transition, with the midpoint $t_0\approx1890$, and Hangul surpassing 93% share by 1980. Circa 1890–1920, a distinct mixed-script phase (>10% each script) is observable.
• North–South OOV Divergence: Tokenizer OOV rates vary up to 51-fold between North Korean (KCNA) and South Korean web text under KLUE-BERT and KcBERT vocabularies, directly attributable to divergent loanword orthography and lexical drift in modern North Korea.

These analyses leverage the unprecedented scale and temporal granularity of OKHC to reveal language systematics not detectable in smaller corpora.

5. Annotation Protocols, Metadata Standards, and Benchmark Tasks

OHKC and associated projects implement machine-interpretable metadata schemas encompassing dataset name, intended NLP task (e.g., POS tagging, NER, language modeling, evaluation), provider, documentation status, license, volume (tokens/sentences/docs), and explicit language/script labels (Cho et al., 2020, Song et al., 28 Oct 2025).

Annotation is performed at the entity (PERSON, LOCATION, BOOK/title), phrase, and document level. The HUE benchmark (Yoo et al., 2022) offers four supervised tasks over AJD/DRS: Chronological Attribution, Topic Classification, Named Entity Recognition (BIO tagging), and Summary Retrieval. Gold annotations are produced by professional historians and, in the case of SeungJeongWon, are further enriched with structural and note markers (Kim et al., 2023).

Model performance benchmarks demonstrate that continued pretraining on era-specific corpora (e.g., AnchiBERT+AJD/DRS) reduces unknown-token rate from ≈0.8% to 0.04% and yields best-in-class results across all HUE tasks (CA F1: 79.3%, TC F1: 88.3%/78.1% for major/minor labels, NER F1: 90.4%) (Yoo et al., 2022).

6. Access, Usage, and Downstream Applications

OKHC is distributed via GitHub (https://github.com/seyoungsong/OKHC) under a non-commercial Creative Commons license for most sources, with processing code (MIT license) included. API-level access as a HuggingFace dataset is supported, and recommended workflows include year/script-based filtering and adaptive tokenizer retraining for morphologically distinct or archaic subcorpora (Song et al., 28 Oct 2025).

Primary downstream applications include:

• Pre-training and finetuning of diachronic or year-aware LLMs.
• Diachronic NER, automated dating, topic modeling, and digital lexicography.
• Text normalization and script conversion between Idu, Old Hangul, mixed script, and modern orthography.
• Machine translation for pre-modern Korean and Classical Chinese sources.
• Digital-humanities research on lexical innovation, discourse shifts, and the sociopolitical impact of orthographic reform (Song et al., 28 Oct 2025, Yoo et al., 2022, Kim et al., 2023).

7. Best Practices for Historical Corpus Construction and Community Infrastructure

Song et al. (2025) (Song et al., 28 Oct 2025) and Cho et al. (2020) (Cho et al., 2020) recommend:

• Adoption of detailed, machine-readable metadata for all corpora and subcorpora.
• Maintaining living registry documents (on arXiv or equivalent) plus synchronized, community-editable registries (GitHub) for tracking releases and updates.
• Use of permissive open licenses where possible, with clear redistribution and modification terms.
• Providing minimal but sufficient documentation and sample code to accelerate adoption.
• Updating model and data benchmarks regularly to reflect new annotation quality or improved OCR/transcription standards.
• Engaging historiographical expertise for annotation, especially for diachronic or genre-diverse corpora.

These protocols have established OHKC as the reference architecture for open diachronic corpora in Korean NLP and have catalyzed both quantitative linguistics and cross-disciplinary digital scholarship (Song et al., 28 Oct 2025, Yoo et al., 2022, Cho et al., 2020).