ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations (1911.00720v1)

Abstract: The pre-training of text encoders normally processes text as a sequence of tokens corresponding to small text units, such as word pieces in English and characters in Chinese. It omits information carried by larger text granularity, and thus the encoders cannot easily adapt to certain combinations of characters. This leads to a loss of important semantic information, which is especially problematic for Chinese because the language does not have explicit word boundaries. In this paper, we propose ZEN, a BERT-based Chinese (Z) text encoder Enhanced by N-gram representations, where different combinations of characters are considered during training. As a result, potential word or phase boundaries are explicitly pre-trained and fine-tuned with the character encoder (BERT). Therefore ZEN incorporates the comprehensive information of both the character sequence and words or phrases it contains. Experimental results illustrated the effectiveness of ZEN on a series of Chinese NLP tasks. We show that ZEN, using less resource than other published encoders, can achieve state-of-the-art performance on most tasks. Moreover, it is shown that reasonable performance can be obtained when ZEN is trained on a small corpus, which is important for applying pre-training techniques to scenarios with limited data. The code and pre-trained models of ZEN are available at https://github.com/sinovation/zen.

Overview of "ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations"

This paper introduces ZEN, a novel approach to pre-trained text encoding, specifically designed for the Chinese language by leveraging n-gram representations. ZEN stands for Chinese text encoder enhanced by N-gram, building on BERT (Bidirectional Encoder Representations from Transformers) but incorporating n-gram information to better handle the intricacies of Chinese, which lacks explicit word boundaries.

Key Contributions

• N-gram Integration: ZEN integrates n-gram representations into the traditional character-based encoder paradigm, addressing potential boundary issues and semantic loss in the Chinese language.
• Architecture: The ZEN model is structured with BERT as the backbone, enhanced by an n-gram encoder that represents combinations of characters, aligning with the multi-layer strategy of transformers.
• Experimental Validation: With evaluations conducted on various Chinese NLP tasks, ZEN demonstrates state-of-the-art performance, often surpassing existing models trained on larger datasets.

Methodology

The methodology involves two primary steps in the n-gram process:

1. N-gram Extraction: An n-gram lexicon is crafted using unsupervised techniques, which identifies combinatory chunks of text that appear salient within the language context. This n-gram lexicon serves as the foundation upon which potential word and phrase boundaries are explicitly represented.
2. N-gram Encoding: In ZEN, n-grams are encoded through a multi-layer transformer, paralleling BERT’s architecture but without assigning sequential order to n-grams. This choice empowers the model to capture semantic units effectively without disrupting downstream tasks.

Results

ZEN was evaluated across seven key NLP tasks, including Chinese Word Segmentation (CWS), Named Entity Recognition (NER), and Document Classification (DC), using commonly accepted datasets like MSRA and CTB5. The encoder achieved impressive results, often outperforming existing models. Highlights include achieving F1-scores that surpass BERT with whole-word masking strategies and large models like ERNIE 2.0 without relying on external or significantly larger datasets.

Implications and Future Directions

The implications of this research are multifaceted:

• Efficiency in Resource Utilization: ZEN's ability to perform well with limited data presents a significant advantage for specialized applications where access to large corpora is restricted.
• Complementarity: Given its architectural design, ZEN is complementary to other approaches that utilize weak supervision or entity-level masking. Future work could explore hybrid models incorporating ZEN's n-gram encoding with other semantic extraction techniques.
• Language Generalization: While ZEN is tailored for Chinese, its foundational principles may extend to other languages with a similar lack of explicit token boundaries, paving the way for broader adoption and adaptation.

In conclusion, the ZEN model presents a robust alternative to existing pre-trained text encoders for Chinese, showcasing that explicit n-gram integration can enhance semantic understanding and boundary resolution effectively. The research contributes notably to the domain of language-specific NLP model design and opens up avenues for further refinement and cross-linguistic application.