GuwenBERT: Pretrained Model for Classical Chinese

• GuwenBERT is a classical Chinese pretrained encoder optimized for simplified texts, used for literary analysis and sequence-labeling tasks.
• It builds on BERT/RoBERTa architectures and is primarily trained on simplified-character corpora, achieving top scores on NER, sentence breaking, and punctuation benchmarks.
• Its application in Tang-poetry origin prediction shows the model can capture regional linguistic nuances, though character n-gram features sometimes perform competitively.

Searching arXiv for GuwenBERT and related classical Chinese LLM papers to ground the article in current literature. arxiv_search(query="^^^^^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^^^^^", max_results=^^^^4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^, sort_by="^^^^^^^^4relevance4 Searching arXiv for the two cited papers by title and related ancient-text model work. arxiv_search(query="^^^^^^^^4ti:\4 Poets' Origins from Verse\" OR ^^^^^^^^4ti:\4 and GujiGPT\"", max_results=^^^^4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^, sort_by="^^^^^^^^4relevance4 GuwenBERT is a classical-Chinese pretrained encoder used for guwen understanding tasks and, in later work, as a transformer backbone for computational literary analysis. In the literature considered here, the name refers both to an earlier classical Chinese BERT primarily trained on simplified-character corpora from Dai Zhi Ge and to a public pretrained model treated as “a RoBERTa for literary Chinese.” Its documented roles are twofold: as a strong simplified-script baseline for sequence-labeling tasks on ancient texts, and as a domain-matched pretrained model for testing whether contextual encoders can detect regional linguistic fingerprints in Tang poetry (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&, &&&^^^^4relevance4^^^^&&&).

^^^^4relevance4^^^^. Definition, lineage, and scope

GuwenBERT is described as a family of BERT-based models trained for classical Chinese. In "GujiBERT and GujiGPT: Construction of Intelligent Information Processing Foundation LLMs for Ancient Texts" (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&), it is treated as a strong existing baseline for simplified-character classical Chinese, in contrast to GujiBERT, which is introduced as a newer suite covering traditional, simplified, and mixed-character settings. In "Predicting Poets' Origins from Verse: A Computational Analysis of Regional Linguistic Fingerprints in the Complete Tang Poems" (&&&^^^^4relevance4^^^^&&&), GuwenBERT is the authors’ transformer backbone for classical Chinese and is explicitly characterized as “a RoBERTa for literary Chinese,” with citation to the public repository "GuwenBERT: A Pre-trained LLM for Classical Chinese" by Yan (^^^^4ti:\4GuwenBERT classical Chinese BERT ancient texts4ti:\4GuwenBERT classical Chinese BERT ancient texts4^^^^).

The two papers jointly delimit the model’s practical identity. It is not presented as a generic modern-Chinese encoder, but as a pretrained model for premodern literary language. This reduces domain mismatch for tasks on Tang poetry and other ancient corpora relative to modern-Chinese BERT baselines (&&&^^^^4relevance4^^^^&&&). At the same time, the available descriptions remain incomplete: neither paper reports exact layer counts, hidden sizes, parameter count, or exact vocabulary size for GuwenBERT.

A recurring source of confusion is that "GuwenBERT" functions both as a model name and as a category label for simplified-focused classical-Chinese encoders. The literature therefore treats it less as a single fully specified architecture than as a recognized pretrained backbone within the BERT/RoBERTa family for guwen NLP.

^^^^4ti:\4^^^^. Architectural characterization and pretraining domain

The available evidence places GuwenBERT within the encoder-only Transformer line derived from BERT and RoBERTa. The ^^^^4ti:\4 OR ti:\4GuwenBERT classical Chinese BERT ancient texts4^^^^7 study situates it among BERT-class models for ancient-text understanding, while the ^^^^4ti:\4GujiBERT and GujiGPT: Construction of Intelligent Information Processing Foundation LLMs for Ancient Texts4GuwenBERT classical Chinese BERT ancient texts4^^^^6 study uses it as a RoBERTa-family encoder for poet-origin classification (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&, &&&^^^^4relevance4^^^^&&&). No architectural modification beyond that family assignment is specified.

The pretraining domain is classical Chinese rather than modern Mandarin. This is important because both papers operate on corpora with archaic lexicon, sparse punctuation, and orthographic phenomena poorly aligned with general-purpose modern-Chinese checkpoints. In the comparative study of ancient-text foundation models, GuwenBERT is described as having been primarily trained on simplified-character corpora from Dai Zhi Ge and as being especially strong on simplified tasks (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&). In the Tang-poetry study, this domain match motivates its use as the transformer baseline for literary Chinese (&&&^^^^4relevance4^^^^&&&).

Tokenization and vocabulary are only partially documented. The Tang-poetry experiments are conducted at the character level, and the model is used through token-level hidden states with masked pooling, but tokenization specifics are not stated (&&&^^^^4relevance4^^^^&&&). A plausible implication is that the model is compatible with character-oriented processing, but the paper does not provide a formal tokenizer specification. Likewise, the fragment length of ^^^^4ti:\4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^ characters in the poetry experiments is reported, whereas sequence-length capacity is not.

^^^^4 OR ti:\4^^^^. Use in Tang-poetry origin prediction

In the Tang-poetry study, GuwenBERT is used to test whether a pretrained LLM can detect regional linguistic fingerprints in verse and whether it adds predictive signal beyond simple lexical statistics (&&&^^^^4relevance4^^^^&&&). The task is poet-origin prediction at three granularities: binary South vs. North, a three-way macro grouping, and multi-class classification across the ten Tang administrative circuits. The underlying corpus aggregates every poem attributed to each author in the Quan Tang Shi and links poets to circuit-of-origin through the China Biographical Database; after filtering for poets with at least five surviving poems and clear attribution, the dataset contains ^^^^4 OR ti:\4^^^^57 poets across ^^^^4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^ circuits.

The study first applies naive fine-tuning on fragments. Each poet’s corpus is sliced into ^^^^4ti:\4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^-character fragments, fragments are classified with a standard transformer classifier head, and fragment-level probabilities are averaged to obtain a poet-level decision. This reaches PRESERVED_PLACEHOLDER_^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^ accuracy under grouped cross-validation, but the setup is considered “unfair” because TF-IDF features summarize the entire poet corpus in one vector while the fragment-based transformer never jointly sees the whole oeuvre (&&&^^^^4relevance4^^^^&&&).

To address that asymmetry, the paper introduces a hierarchical frozen-encoder representation. Each fragment PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^ is encoded by GuwenBERT, denoted PRESERVED_PLACEHOLDER_^^^^4ti:\4^^^^, with encoder weights frozen. Token hidden states PRESERVED_PLACEHOLDER_^^^^4 OR ti:\4^^^^ are reduced by masked mean pooling over non-padding positions:

$$h_i = \left( \frac{1}{\sum_t m_{it}} \right) \sum_t m_{it} z_{it},$$

where $m_{it} \in \{0,1\}$ is the attention mask. The poet-level representation is then the mean of fragment embeddings:

$H = \left( \frac{1}{N} \right) \sum_{i=1}^{N} h_i.$

A simple classifier head predicts origin:

$\hat{y} = \mathrm{softmax}(W H + b),$

optimized with standard cross-entropy loss,

$$\mathcal{L} = -\sum_i \sum_c y_{i,c} \log \hat{y}_{i,c}.$$

The paper reports no partial unfreezing and no adapter tuning. In effect, the entire corpus is represented by averaging fragment embeddings, each itself formed by averaging token states (&&&^^^^4relevance4^^^^&&&).

4. Empirical performance across tasks

On ancient-text sequence-labeling benchmarks, GuwenBERT is strongest in simplified-script settings. In the Shiji NER evaluation, it is the best simplified model with $F1 = 93.73$ and PRESERVED_PLACEHOLDER_^^^^4relevance4GuwenBERT classical Chinese BERT ancient texts4^^^^, PRESERVED_PLACEHOLDER_^^^^4relevance4relevance4^^^^. In simplified sentence breaking on the Twenty-Four Histories, it is again best with PRESERVED_PLACEHOLDER_^^^^4relevance4ti:\4^^^^. In simplified automatic punctuation on ^^^^4ti:\4^^^^5 pre-Qin canonical texts, it is also best with PRESERVED_PLACEHOLDER_^^^^4relevance4 OR ti:\4^^^^. On the simplified Zuozhuan word-segmentation and lexical/POS tagging task, it reaches PRESERVED_PLACEHOLDER_^^^^4relevance44^^^^, very close to the best score of PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^5 from GujiBERT_jian (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&). These results establish GuwenBERT as a high-performing simplified classical-Chinese encoder rather than merely a historical baseline.

Its behavior is different in poet-origin prediction. For South-vs.-North classification, the best classical model built on character PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^6-gram TF-IDF together with interpretable domain features reaches PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^7 accuracy, well above the PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^8 majority baseline. Under the grouped fragment fine-tuning setup, GuwenBERT reaches PRESERVED_PLACEHOLDER_^^^^4relevance4^^^^9 accuracy. Under the hierarchical frozen-encoder setup, it reaches PRESERVED_PLACEHOLDER_^^^^4ti:\4GuwenBERT classical Chinese BERT ancient texts4^^^^ accuracy under the same 5-fold protocol. Concatenating the hierarchical GuwenBERT vector with TF-IDF features yields no improvement over either alone (&&&^^^^4relevance4^^^^&&&).

The circuit-level results supply further context. Performance is above chance, Jiangnan is the most identifiable circuit with recall PRESERVED_PLACEHOLDER_^^^^4ti:\4relevance4^^^^, and central circuits near Chang’an and Luoyang—especially Guannei and Henan—are frequently confused (&&&^^^^4relevance4^^^^&&&). The paper interprets this pattern as consistent with court-driven homogenization. GuwenBERT therefore participates in a broader empirical finding: regional signal exists, but the signal appears to be largely recoverable from lexical and imagistic features without additional predictive gain from the pretrained encoder.

5. Interpretation, limitations, and historically meaningful errors

The poet-origin study argues that GuwenBERT matches but does not beat simple TF-IDF because character PRESERVED_PLACEHOLDER_^^^^4ti:\4ti:\4^^^^-grams already capture the available regional signal (&&&^^^^4relevance4^^^^&&&). The paper proposes several likely contributing factors. First, in a logographic script, the character is the natural unit, so lexical statistics can already encode regional variation through imagery, lexical choice, and functional characters. Second, the hierarchical experiment freezes the encoder, which limits task adaptation, while the labeled dataset is small enough that full end-to-end tuning may be underpowered or prone to overfitting. Third, mean-pooling over tokens and fragments is a blunt long-context aggregation strategy, which may obscure motif co-occurrence, stylistic cadence, or tonal structure. Fourth, although GuwenBERT is pretrained on classical Chinese, it may not be specifically tailored to Tang poetic registers such as regulated verse and idiomatic imagery.

These limitations are not merely technical. The study treats some model errors as literary-historical evidence. In the Early Tang, every misclassification is a southern poet read as northern, which the authors connect to the prestige of the northern court idiom (&&&^^^^4relevance4^^^^&&&). This indicates that GuwenBERT’s mistakes can reflect historically meaningful homogenization rather than arbitrary noise. The same study reports that South/North separability is at chance in the High Tang at PRESERVED_PLACEHOLDER_^^^^4ti:\4 OR ti:\4^^^^ accuracy and strongest in the Late Tang at PRESERVED_PLACEHOLDER_^^^^4ti:\44^^^^, while linguistic distance between circuits grows with geographic distance, yielding a Mantel statistic of PRESERVED_PLACEHOLDER_^^^^4ti:\4^^^^5 with PRESERVED_PLACEHOLDER_^^^^4ti:\4^^^^6 over nine sufficiently populated circuits. The result is described as suggestive and driven by the distinctiveness of the Jiangnan idiom (&&&^^^^4relevance4^^^^&&&).

A common misconception is that pretrained literary Transformers necessarily dominate interpretable lexical models in authorship or provenance tasks. The GuwenBERT results do not support that generalization. In this setting, interpretable machine learning and character PRESERVED_PLACEHOLDER_^^^^4ti:\4^^^^7-grams remain fully competitive, and the pretrained encoder contributes representation quality without clear downstream superiority.

6. Position within ancient-text NLP and related model families

Within the broader ecosystem of ancient-Chinese LLMs, GuwenBERT occupies the role of an earlier simplified-focused encoder. The ^^^^4ti:\4GuwenBERT classical Chinese BERT ancient texts4ti:\4 OR ti:\4^^^^ comparative study positions GujiBERT as a successor-style alternative with broader script coverage: traditional-only, simplified-only, and mixed-font variants, continued pretraining from established checkpoints, and vocabularies expanded to include “all the Chinese characters that can be displayed in UTF-8” (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&). In that comparison, GuwenBERT remains excellent for simplified-only tasks, especially sentence breaking and punctuation, while GujiBERT and GujiRoBERTa are preferable on traditional and many mixed-script evaluations.

The same study also notes that Koichi Yasuoka’s RoBERTa-classical-Chinese-base-char further extends GuwenBERT to handle both simplified and traditional through incremental training, and that this model performs best on mixed sentence breaking (&&&^^^^4GuwenBERT classical Chinese BERT ancient texts4^^^^&&&). This places GuwenBERT in a lineage rather than at an endpoint: it is a strong simplified classical-Chinese encoder whose limitations become most visible when script coverage broadens or when downstream tasks require long-context aggregation beyond sentence- or sequence-labeling regimes.

Future directions are stated only implicitly in the Tang-poetry paper, but they are technically clear. More sophisticated hierarchical architectures, including attention-weighted pooling, limited unfreezing or adapter-based fine-tuning, longer-context Transformers, and domain-adaptive pretraining on Tang poetic corpora are identified as plausible ways to test whether pretrained encoders can surpass character PRESERVED_PLACEHOLDER_^^^^4ti:\4^^^^8-grams when given richer capacity to model stylistic nuance (&&&^^^^4relevance4^^^^&&&). The present evidence, however, supports a narrower conclusion: GuwenBERT is a strong domain-matched model for classical Chinese, especially in simplified-script understanding tasks, but its advantages are task-dependent and can disappear when the target signal is already well captured by interpretable lexical statistics.