ZEN 2.0: N-gram Enhanced Text Encoder

• ZEN 2.0 is an n-gram enhanced text encoder that scales BERT-large and refines integration with frequency-based weighting and whole n-gram masking.
• It fuses an auxiliary n-gram encoder with a BERT-like character Transformer via layer-wise weighted addition for enriched lexical representation.
• The model demonstrates strong cross-lingual transfer on Chinese and Arabic tasks, leveraging explicit n-gram signals to boost performance.

Searching arXiv for the ZEN 2.0 paper and closely related work. {"query":"ZEN 2.0 Continue Training and Adaption for N-gram Enhanced Text Encoders arXiv", "max_results": 5} ZEN 2.0 is an n-gram-enhanced pre-trained text encoder that extends the earlier ZEN design by scaling to BERT-large size, refining n-gram integration with frequency-based weighting, masking whole n-grams during masked language modeling, and adopting relative positional encoding in the character encoder. It is designed for settings in which vanilla character/subword encoders underutilize larger-granularity signals such as words, phrases, and spans, a limitation that is especially consequential in languages without explicit word boundaries, such as Chinese, and in morphologically rich languages, such as Arabic. The architecture combines a character-based Transformer backbone with an auxiliary n-gram encoder and demonstrates strong cross-lingual and cross-domain transfer on a broad set of downstream tasks (Song et al., 2021).

1. Conceptual basis and scope

ZEN 2.0 addresses a specific limitation of pre-trained text encoders: single-stream character or subword models such as BERT can encode local context through self-attention, but they do not explicitly represent larger lexical units discovered from corpora. ZEN 1.0 addressed this limitation by explicitly encoding n-grams via an auxiliary encoder and integrating them into a character-based Transformer. ZEN 2.0 preserves that two-stream premise while revising the training and fusion mechanisms and scaling the model to larger regimes (Song et al., 2021).

The central design choice is explicit n-gram encoding rather than reliance on masking strategy alone. Prior models compared in the paper, including BERT, RoBERTa, ERNIE, MacBERT, and NEZHA, are described as mostly relying on masking strategies such as whole-word masking while keeping a single-stream architecture. ZEN 2.0 instead encodes and fuses n-gram representations at every layer, which provides controllable knowledge insertion through the lexicon and is reported to yield stronger gains, especially for Chinese, where character-only models omit critical chunking information (Song et al., 2021).

This framing places ZEN 2.0 at the intersection of lexicalized pretraining and Transformer-based representation learning. A plausible implication is that the model is best understood not merely as a masking variant, but as an architectural intervention that changes the representational substrate available to the encoder.

2. Architectural organization

ZEN 2.0 consists of two encoders with layer-wise fusion. The character encoder is a BERT-like Transformer stack with relative positional encoding and serves as the main backbone used for downstream tasks. The n-gram encoder is a Transformer stack, inherited from ZEN 1.0 and used without positional encoding, that encodes all n-grams detected in the input via a lexicon. Its outputs are injected into every layer of the character encoder through weighted addition (Song et al., 2021).

For each input sequence, an n-gram lexicon is used to enumerate all n-grams that occur in the sequence. Each character position $i$ is associated with the set of n-grams that span it. In ZEN 1.0, the fusion was unweighted:

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$

ZEN 2.0 replaces this with frequency-weighted fusion. For character $i$ and its $k$-th associated n-gram, the model computes a frequency-based weight

$p_{i,k} = \frac{c_{i,k}}{\sum_k c_{i,k}},$

where $c_{i,k}$ is the corpus frequency of that n-gram. The fused representation then becomes

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k p_{i,k} \cdot \mu_{i,k}^{(l)}.$$

In this formulation, more salient frequent n-grams are given greater weight when n-gram signals are injected into character representations. The paper characterizes this as a refinement of n-gram integration rather than a change to the overall two-encoder topology (Song et al., 2021).

Relative positional encoding is also incorporated into the character encoder using a Transformer-XL-style mechanism. With per-token inputs $H$, learned projections $W_q$, $W_k$, $$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$0, and sinusoidal relative positional vectors $$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$1, the model defines

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$2

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$3

The relative attention score for one head is

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$4

and the head output is

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$5

This architecture preserves standard Transformer blocks while augmenting them with explicit lexical pathways. The paper notes that the design eases integration into existing pipelines because the character encoder remains the main path at inference time, even though n-gram lookup and encoding add overhead.

3. Pretraining objectives and training procedure

ZEN 2.0 is trained with standard BERT-style masked language modeling and next sentence prediction, with the joint objective

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$6

For MLM, the model uses Whole N-gram Masking rather than random character masking:

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$7

where $$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$8 is the set of masked character positions. The NSP loss is binary classification over sentence pairs:

$$\vartheta_i^{(l)*} = \vartheta_i^{(l)} + \sum_k \mu_{i,k}^{(l)}.$$9

These are the only pretraining objectives reported in the paper (Song et al., 2021).

Whole N-gram Masking is implemented by first segmenting text into character n-grams using an off-the-shelf segmenter and then merging adjacent n-grams if the merged span exists in the lexicon. Selected spans are masked jointly so that 15% of characters are covered; within the covered spans, 80% are replaced with [MASK], 10% with random tokens, and 10% remain unchanged. The purpose is to enforce span-level prediction during MLM and to align the masking strategy with the model’s explicit n-gram pathway (Song et al., 2021).

The training configuration is large scale. For Chinese, the cleaned and filtered corpus contains 268.2M sentences and 8.40B tokens, drawn from Wikipedia, News, Baike, Webtext, the Chinese side of Zh–En parallel data, Comments, and Zhihu. For Arabic, the corpus contains 231.1M sentences and 7.29B tokens from Wikipedia, crawled News, AraCorpus, Abu El-Khair Corpus, OSCAR, Tashkeela, and the Arabic side of UN Parallel. Optimization uses Adam, a peak learning rate of $i$0 with linear decay, and 36K warmup steps. Batch size is 24,576 for the base model and 8,192 for the large model. The Chinese model is trained for 600K steps and the Arabic model for 800K steps on a large GPU cluster with hundreds of NVIDIA Tesla V100 GPUs (Song et al., 2021).

The model sizes are explicitly specified. ZEN 2.0 base uses a character encoder of $i$1 heads, 768 hidden$i$2 and an n-gram encoder of $i$3 heads$i$4. ZEN 2.0 large uses a character encoder of $i$5 heads, 1024 hidden$i$6 and an n-gram encoder of $i$7 heads$i$8.

4. Lexicon construction, tokenization, and language adaptation

A defining component of ZEN 2.0 is the explicit n-gram lexicon. For Chinese, the backbone is character-based. Candidate n-grams of length $i$9 are extracted from large raw text using pointwise mutual information, with PMI threshold 3 and frequency threshold 15. The final Chinese lexicon contains 261K n-grams. For Whole N-gram Masking, WMSeg is used to segment raw text into character n-grams, and adjacent pieces are merged into larger n-grams if the merged span exists in the lexicon before selecting spans to mask (Song et al., 2021).

For Arabic, the procedure is adapted rather than replaced. The BERT tokenizer segments text into word pieces, and these word pieces are treated as “characters” for the character-based encoding and for n-gram construction. The n-gram lexicon again uses lengths $k$0 to $k$1, but with PMI threshold 10 and frequency threshold 20, yielding a final lexicon size of 194K Arabic n-grams (Song et al., 2021).

This cross-lingual setup is important because the paper presents ZEN 2.0 not as a Chinese-specific segmentation model but as a more general architecture for explicit larger-granularity lexical encoding. The same architecture and training recipe are reported to apply to typologically distinct languages. In Chinese, the combination of a character-level backbone and explicit n-grams addresses the absence of whitespace boundaries and leverages frequent multi-character phrases. In Arabic, treating word pieces as “characters” while retaining n-gram lexicons yields strong gains across POS tagging, NER, NLI, and MRC. This suggests that the transfer mechanism is architectural rather than language idiosyncratic (Song et al., 2021).

A practical dependency remains: lexicon quality and segmentation quality matter. The paper explicitly identifies dependence on segmentation for Whole N-gram Masking and notes that poor segmentation can reduce the efficacy of WNM.

5. Empirical performance across Chinese and Arabic benchmarks

ZEN 2.0 is evaluated on Chinese across nine tasks and ten datasets, and on Arabic across six tasks and ten datasets. The reported results emphasize the large configuration (Song et al., 2021).

Language	Task / Dataset	ZEN 2.0 large
Chinese	MSR-CWS test F1	98.66
Chinese	CTB5 POS dev/test acc	97.84 / 97.09
Chinese	MSRA NER dev/test F1	96.68 / 96.20
Chinese	THUCNews dev/test acc	98.26 / 97.93
Chinese	ChnSentiCorp dev/test acc	96.25 / 96.50
Chinese	LCQMC dev/test acc	89.33 / 88.81
Chinese	BQ Corpus dev/test acc	87.11 / 85.99
Chinese	XNLI dev/test acc	83.25 / 83.09
Chinese	CMRC2018 dev EM/F1	73.00 / 89.92
Chinese	NLPCC-DBQA dev/test MRR/F1	96.04 / 85.69 and 96.11 / 86.47
Arabic	ATB POS dev/test acc	96.67 / 96.69
Arabic	AQMAR NER dev/test F1	79.24 / 80.26
Arabic	ANERCorp test F1	85.47
Arabic	SANAD AR-5 / AB-7 / KH-7 acc	98.92 / 96.58 / 99.23
Arabic	ASTD test acc	75.17
Arabic	XNLI dev/test acc	82.89 / 83.09
Arabic	Arabic-SQuAD dev/test EM/F1	46.99 / 63.91 and 46.09 / 62.37
Arabic	ARCD test EM/F1	38.32 / 70.12

On Chinese, the model improves over ZEN 1.0 large on CWS, POS, NER, document classification, sentiment analysis, semantic pair matching, NLI, MRC, and open-domain QA. The paper reports that it surpasses MacBERT large and ERNIE 2.0 large on XNLI test accuracy, and exceeds ERNIE 2.0 large on MSRA NER test F1. On Arabic, it improves over Arabic BERT large across POS, AQMAR NER, ASTD sentiment, XNLI, Arabic-SQuAD, and ARCD, with especially large gains reported for NLI and MRC (Song et al., 2021).

The cross-task pattern is central. Results are reported not as isolated task-specific gains but as broad improvements across sequence labeling, classification, inference, machine reading comprehension, and open-domain question answering. This suggests that the architecture’s benefit is not narrowly tied to a single supervision format.

6. Ablation findings and interpretive analysis

The ablation results isolate three principal contributors: Whole N-gram Masking, relative positional encoding, and the explicit n-gram representation pathway (Song et al., 2021).

For Chinese, Whole N-gram Masking with WMSeg outperforms both Jieba-based WNM and character masking. On XNLI test accuracy, the results are 83.09 with WMSeg, 82.44 with Jieba, and 81.06 with character masking. On CMRC2018 dev F1, they are 89.92, 88.95, and 87.84. On LCQMC test accuracy, they are 88.81, 88.54, and 88.48. On THUCNews test accuracy, they are 97.93, 97.90, and 97.90. On MSRA NER test F1, they are 96.20, 96.18, and 96.08. The paper’s stated takeaway is that masking larger, lexicon-aligned spans is beneficial and that better segmentation further improves WNM effectiveness (Song et al., 2021).

Relative positional encoding is also reported to improve performance consistently on Chinese NLI and QA and on Arabic NER and NLI, with more than 2 F1 points on Arabic AQMAR NER. The paper interprets this as evidence for the importance of relative order modeling, particularly in morphologically rich Arabic.

Training-step analysis further shows that on Chinese NLI and MRC, ZEN 2.0 large outperforms both BERT large and ZEN 1.0 large consistently across training, with especially large margins early, before 100K steps. This is presented as evidence that the architectural and training enhancements accelerate and stabilize learning (Song et al., 2021).

The paper also includes a t-SNE analysis of n-gram representations, in which learned n-gram embeddings cluster semantically; examples such as “USA,” “China,” “UK,” “Germany,” and “Japan” co-cluster. The reported interpretation is that the n-gram encoder captures meaningful phrase-level semantics even though the n-grams are automatically discovered. A plausible implication is that the auxiliary encoder is not merely providing frequency priors, but learning a structured lexical semantic space.

7. Limitations, reproducibility, and position within encoder design

Several limitations are explicitly noted. Whole N-gram Masking depends on the off-the-shelf segmenter, and WMSeg outperforms Jieba. The frequency-based fusion weights are heuristic because they use raw corpus frequencies; the paper identifies more principled, context-aware gating or attention over n-grams as a possible improvement. Lexicon construction depends on PMI- and frequency-thresholded mining, so lexicon coverage and quality affect performance. The model retains NSP and does not introduce an explicit n-gram prediction loss. Finally, the two-encoder design and whole-span masking over billions of tokens require substantial compute (Song et al., 2021).

At the implementation level, the paper reports that code and models are released at https://github.com/sinovation/ZEN2, with base and large checkpoints for Chinese and Arabic. The reported reproduction procedure is to build the n-gram lexicon via PMI on large raw text with $k$2 and language-specific thresholds, preprocess corpora with cleaning and filtering, pretrain with MLM using Whole N-gram Masking plus NSP, enable relative positional encoding in the character encoder, and fine-tune with standard BERT-style task heads and metrics (Song et al., 2021).

Within the broader landscape of pre-trained encoders, ZEN 2.0 occupies a distinct position. It shares with BERT-style models the Transformer backbone, MLM, NSP, and standard task heads, but differs by making larger lexical units first-class representational objects through an auxiliary n-gram encoder fused at every layer. The paper’s empirical results indicate that this explicit lexical pathway is effective across languages and domains. This suggests that ZEN 2.0 is best viewed as a lexicon-grounded Transformer pretraining framework for scenarios in which character-only or subword-only encoders fail to adequately exploit phrase-level regularities.