OracleCS: Ancient Chinese Corpus & Encoding

• OracleCS is a unified corpus that standardizes excavated oracle bone inscriptions and rare glyphs using the InteChar encoding system.
• It integrates expert-curated transcriptions, instruction-style augmented samples, and classical texts to support precise representation learning for historical NLP.
• The dataset improves performance in tasks like cloze completion, translation, and retrieval by replacing unknown tokens with consistent glyph encodings.

Oracle Corpus Set (OracleCS) is an ancient Chinese corpus introduced to make language modeling feasible on excavated oracle bone inscriptions (OBI), a domain long constrained by sentence-level data scarcity and by the absence of a unified text encoding for many unearthed characters not included in Unicode. Built on InteChar, a unified, Unicode-compatible character list that integrates previously unencoded oracle bone characters with traditional and modern Chinese, OracleCS combines expert-curated OBI transcriptions, instruction-style augmented samples, and selected classical texts to support representation learning and downstream historical NLP (Diao et al., 12 Aug 2025).

1. Definition, motivation, and research scope

OracleCS is designed around two obstacles that had traditionally prevented progress on historical language modeling for oracle bone inscriptions: the scarcity of historical language samples and the absence of a unified text encoding for many excavated characters. The corpus is therefore not merely a collection of transcriptions; it is part of a representation-and-modeling pipeline in which character standardization precedes corpus construction.

The corpus is centered on Chinese oracle bone inscriptions and targets both the digitization problem and the modeling problem. The digitization problem concerns turning image-only glyphs into machine text. The modeling problem concerns making low-frequency, undeciphered, and rare characters learnable by LLMs. OracleCS addresses these jointly by pairing a character inventory, InteChar, with a task-rich corpus that includes expert-annotated data and LLM-assisted augmentation.

A central practical consequence is that oracle glyphs, including previously unencoded and undeciphered ones, can be treated as textual units rather than as images alone. This permits character-level modeling, controlled benchmarking, and direct integration with modern NLP pipelines. The paper further presents OracleCS as a foundation for robust modeling of ancient scripts and for historical language understanding tasks such as cloze completion, commentary-to-text retrieval, translation, polysemous word matching, and word parsing (Diao et al., 12 Aug 2025).

2. InteChar as the representational substrate

OracleCS depends on InteChar, which the paper describes as the prerequisite for the corpus. InteChar is built in four stages. First, the authors initialize with the official Unicode character set, including more than 90,000 CJK Unified Ideographs, to guarantee compatibility with modern NLP systems. Second, they integrate previously encoded ancient glyphs from paleographic font resources, notably the Zhongjian Library’s historical fonts, while retaining only glyphs actually attested in the curated corpus and de-duplicating across fonts. Third, for oracle glyphs absent from Unicode and historical fonts, they construct new characters through a semi-automated radical-based pipeline consisting of image preprocessing, radical recognition, component standardization, expert verification, vectorization, and code point assignment. Fourth, they perform expert-guided proofreading and de-duplication, using Siamese-network glyph similarity to surface candidate duplicates (Diao et al., 12 Aug 2025).

InteChar contains 11,288 characters in total and ships with a TrueType font for consistent rendering. Each newly constructed glyph receives an internal Unicode-style code point, and mappings to modern equivalents are recorded when available. The design is explicitly Unicode-compatible and extensible, so newly discovered characters can be added through the same pipeline.

This representational layer changes the modeling interface. In the reported experiments, models are adapted by replacing the original character embedding layer with a newly initialized embedding matrix aligned to the InteChar vocabulary, while keeping the pretrained backbone frozen during embedding training. This realizes a character-level tokenization interface tied directly to the oracle inventory. A plausible implication is that the main benefit of OracleCS is inseparable from the encoding discipline imposed by InteChar: the corpus is useful partly because the character inventory makes previously unstable glyph identities computationally addressable.

3. Data composition, sources, and annotation schema

OracleCS draws on three principal source classes. The first is excavated oracle bone texts, manually selected and annotated by paleography and historical Chinese experts from archaeological literature and oracle rubbings, including materials associated with the cited site portal. The second is selected classical and pre-Qin texts from ctext.org, including the Analects, Spring and Autumn Annals, Mencius, and Xunzi, which provide sentence-level contexts and support cross-era semantic alignment with commentaries. The third is lexical material from Shuowen Jiezi and the Great Chinese Dictionary, which supply radical decompositions and definitions; undeciphered characters have semantic fields left blank (Diao et al., 12 Aug 2025).

The corpus statistics reported in the paper are as follows.

Component	Size	Notes
Vocabulary	approximately 11,288	Matches InteChar
Total annotated samples	173,459	Expert-annotated and augmented
Cloze dataset	15,416	12,416 train, 3,000 eval
Commentary-to-text retrieval	896 queries	12,141 candidate sentences
Ancient Chinese Translation	15,868 train	10,578 test
Polysemous Word Matching	33,380 train	22,253 test
Word Parsing	81,929 train	54,619 test

OracleCS records multi-level annotations. At the character level, entries include an InteChar code point or ID, radical decomposition, and dictionary-backed definitions when a character is deciphered. At the sentence or passage level, records include Oracle transcription using InteChar characters, normalized or transcribed forms when available, mappings to modern Chinese equivalents, and task-specific labels such as source-target pairs, binary labels, or choice labels. Source references are also retained at a high level.

The paper does not report per-source counts, inscription identifiers, artifact IDs, excavation numbers, per-piece site or period fields, or a formal artifact-level schema. It also does not provide an exact breakdown of expert-curated versus augmented data quantities. Those omissions matter for provenance-sensitive philology, but the released description is sufficient to characterize OracleCS as a multi-source corpus engineered for low-resource historical NLP rather than as a fully enumerated archaeological catalog.

4. Augmentation pipeline and task formulation

To mitigate sparsity and long-tailed character distributions, OracleCS includes instruction-style augmented samples. The augmentation pipeline uses task descriptions and input-output exemplars for translation, synonym substitution, glyph structure analysis, character decomposition, and semantic prediction. Generated outputs are filtered by aligning them back to InteChar, discarding malformed sequences or illegal IDs, and enforcing task-specific output constraints. Domain experts verify sensitive cases, especially rare or undeciphered glyphs, correct incorrect mappings, and leave undeciphered items semantically unlabeled when necessary (Diao et al., 12 Aug 2025).

The paper emphasizes that augmentation expands scale and task diversity, but it does not report separate augmented-data counts, acceptance rates, or automatic quality metrics. This suggests that OracleCS should be understood as a hybrid corpus in which augmentation is operationally important but not separately quantified.

Two evaluation regimes are defined. The first is embedding-based evaluation with frozen backbones and trained InteChar-aligned embeddings. This regime includes oracle-inscription cloze completion, evaluated with NDCG@k and MRR@k, and commentary-to-text retrieval, also evaluated with NDCG@k and MRR@k. The second regime is LoRA-based fine-tuning for three downstream tasks: Ancient Chinese Translation, Polysemous Word Matching, and Word Parsing, all evaluated with accuracy.

For generative modeling, the paper states the standard language-model objectives

$L = -\frac{1}{N} \sum_{i=1}^{N} \log p_\theta(x_i)$

and

$$PP = \exp\bigg( -\frac{1}{N} \sum_{i=1}^{N} \log p_\theta(x_i) \bigg).$$

However, perplexity is not reported in the experiments; the empirical analysis relies on ranking metrics and accuracy rather than generative evaluation.

5. Modeling protocol, baselines, and training configuration

The experimental study covers 10 representative systems: BERT, GPT-2, Llama-3-8B, MiniRBT, guwenBERT-base, sikuBERT, Qwen-7B-Chat, GLM-4-9B, XunziALLM, and TongGu-LLM. The experimental logic is deliberately controlled. In embedding evaluation, the original character embedding layer is replaced with a new InteChar-aligned embedding matrix, the backbone is frozen, and embeddings are trained on OracleCS. In fine-tuning, LoRA-based parameter-efficient adaptation is used while freezing the backbone and training low-rank adapters and task heads (Diao et al., 12 Aug 2025).

The reported hardware is 8× HUAWEI Ascend-D910b NPUs, and the implementation uses PyTorch. Embedding training runs for 10 epochs with batch size 32, AdamW, learning rate $3 \times 10^{-5}$, and early stopping on NDCG@10. Fine-tuning also runs for 10 epochs with batch size 32, AdamW, learning rate $1 \times 10^{-5}$, CrossEntropy loss, and selection of the best checkpoint by validation score.

This setup is important methodologically because the paper frames the comparison as a clean ablation on the character inventory. The only structural change is replacing the original character list and embedding layer with InteChar while keeping the backbone and training regime fixed. The resulting differences are therefore attributed to unified encoding and oracle-glyph coverage rather than to architecture changes.

6. Empirical performance and interpretive significance

The reported results show consistent improvements across both evaluation regimes and all model families when switching from the original character list to InteChar trained on OracleCS. On oracle-only cloze completion, Qwen-7B-Chat improves from NDCG@10 0.302 to 0.842 and from MRR@10 0.254 to 0.736. GPT-2 improves from MRR@10 0.168 to 0.534 and from NDCG@20 0.224 to 0.643. BERT improves from MRR@10 0.134 to 0.375 and from NDCG@10 0.167 to 0.515. The paper states that gains are universal across all 10 models and across the ranking metrics at $k=10$ and $k=20$ (Diao et al., 12 Aug 2025).

On commentary-to-text retrieval, GLM-4-9B improves from MRR@500 0.176 to 0.285 and from NDCG@500 0.432 to 0.595. Qwen-7B-Chat improves from NDCG@500 0.418 to 0.579 and from MRR@500 0.168 to 0.272. The improvements are described as consistent across all models at $k=400$ and $k=500$.

On LoRA-based downstream tasks, TongGu-LLM improves from Translation 94.12 to 94.84, Parsing 92.06 to 92.65, and Average 92.21 to 92.92. Qwen-7B-Chat improves in average accuracy from 92.32 to 93.06, and guwenBERT-base improves from 90.62 to 91.29. The paper states that improvements hold for all three tasks and all models tested.

Several empirical interpretations are explicitly suggested by the reported findings. First, consistent encoding for rare and undeciphered glyphs avoids unknown-token style losses and enables discriminative character embeddings. Second, radical-informed construction and expert validation reduce inventory noise. Third, Unicode compatibility and a TTF font allow excavated glyphs to be treated as first-class textual units. The especially large cloze gains, such as the Qwen-7B-Chat jump from 0.302 to 0.842 on NDCG@10, indicate that character inventory design is not a marginal preprocessing choice but a primary determinant of performance in oracle-bone language modeling.

7. Limitations, deployment considerations, and terminological context

The paper identifies several limitations. Oracle bone inscriptions remain sparse and context-limited, and selection from published rubbings may over-represent better-preserved or famous inscriptions. Site and period coverage are not quantified. LLM-based augmentation may propagate anachronisms or hallucinated mappings, and neither acceptance rates nor augmentation error profiles are reported. Undeciphered characters remain semantically unlabeled, and rare characters may still be underrepresented. Open questions include how best to integrate radical structure into language-model pretraining objectives, whether formal Unicode proposals for oracle blocks would reduce workflow friction, and how to design provenance-aware evaluation by site or period (Diao et al., 12 Aug 2025).

The paper also states several practical usage recommendations. Oracle transcriptions should be normalized to InteChar IDs. Character tables should be keyed by InteChar ID and retain Unicode links where available, vector glyphs, radical decomposition, and dictionary glosses. Oracle passages should use character-level tokenization tied to InteChar, with a fresh embedding matrix aligned to the inventory. Distinct oracle glyphs should not be collapsed into a single unknown token. For undeciphered characters, semantic fields should remain blank while radicals and positional context are preserved. The reported training advice includes early stopping, small learning rates, and parameter-efficient adaptation such as LoRA; a curriculum from classical texts to oracle inscriptions is suggested when training from scratch.

Several metadata items are explicitly not reported: a public download link or DOI, licensing terms, directory layout, file formats, inter-annotator agreement, augmentation acceptance statistics, and artifact-level provenance fields. For archival, philological, or benchmark governance purposes, those omissions are substantial.

Within the supplied literature, OracleCS is explicitly named as the ancient Chinese corpus introduced with InteChar, while the same label also appears as an editorial framing for oracle-centered research problems in exact partition recovery with same-cluster oracles (DePavia et al., 2023), extractive oracle summary enumeration (Hirao et al., 2017), fault-tolerant graph distance oracles (Dey et al., 2024), and quantum Grover oracles for subset sum (Benoit et al., 2024). This suggests a broader cataloging use of the term in oracle-related research, but the concrete dataset bearing the name is the historical Chinese corpus. In that specific sense, OracleCS is best understood as a corpus-and-encoding infrastructure for excavated oracle inscriptions: a resource that makes previously unencoded glyph inventories tractable for modern NLP while establishing benchmark tasks for ancient Chinese language understanding.