Bert-Kyrgyz-Tokenizer: Kyrgyz Subword Tokenization

• Bert-Kyrgyz-Tokenizer is a tailored subword segmentation framework designed for Kyrgyz, addressing agglutinative morphology using a custom WordPiece approach.
• It optimizes vocabulary size to 30,522 tokens by preserving interpretable morpheme boundaries and maintaining an OOV rate below 0.5%.
• Integration with KyrgyzBERT shows competitive downstream performance, enhancing tasks like sentiment analysis for this low-resource language.

Bert-Kyrgyz-Tokenizer is a subword segmentation and vocabulary construction framework tailored for the Kyrgyz language, developed as the lexical backbone of KyrgyzBERT—the first monolingual BERT-based LLM trained specifically for Kyrgyz. Addressing the challenges of highly agglutinative Turkic morphology and extreme vocabulary sparsity, Bert-Kyrgyz-Tokenizer implements a custom WordPiece approach, optimized to preserve interpretable morpheme boundaries while maintaining a compact, efficient token inventory. Its design principles, training algorithms, quantitative properties, and integration strategies exemplify best practices for tokenization in low-resource, morphologically rich languages (Metinov et al., 25 Nov 2025). Alternative and complementary approaches for adapting existing BERT-based tokenizers to Kyrgyz, including continued BPE training and leaf-based vocabulary pruning, are also described in recent literature (Purason et al., 3 Dec 2025).

1. Linguistic Motivation and Design Principles

Kyrgyz is characterized by highly agglutinative morphology, where lexical stems combine with a variety of productive affixes, resulting in a combinatorial explosion of surface forms. A word-level vocabulary under such conditions leads to infeasibly large lexicons and extreme out-of-vocabulary (OOV) rates, while character-level models fail to delineate meaningful morpheme boundaries critical for syntactic and semantic understanding. Bert-Kyrgyz-Tokenizer employs a data-driven subword tokenization strategy (WordPiece), trained from scratch on a large corpus of Kyrgyz text, to optimally balance morphologically meaningful unit discovery and vocabulary compactness. This enables common stems (e.g., “жүр”, “бил”) and productive suffixes (“##ген”, “##дер”, “##би”) to be encoded as discrete subwords, greatly reducing OOV frequency and facilitating model interpretability for downstream linguistic analysis (Metinov et al., 25 Nov 2025).

2. Token Generation and Vocabulary Construction Algorithms

The WordPiece algorithm, as realized in Bert-Kyrgyz-Tokenizer, operates via a bottom-up agglomerative procedure analogous to Byte-Pair Encoding (BPE). Its key steps are as follows:

• Initialization: Start with a symbol inventory $V$ containing all Unicode characters present in the training corpus $C$, augmented by five special tokens ([PAD], [UNK], [CLS], [SEP], [MASK]).
• Corpus encoding: Each word is first mapped to its constituent characters (with an optional start-of-word symbol).
• Merge loop: Until the desired vocabulary size $V_{target}$ is met (30,522 tokens), the following iterates:
  1. For every pair of adjacent symbols $(a,b)$ in the current corpus segmentation, count frequencies: $p(a,b) = \frac{count(a,b)}{\sum_{(x,y)}count(x,y)}$.
  2. Select the most frequent pair $(a^*,b^*) = \arg\max_{(a,b)} p(a,b)$.
  3. Merge $a^*,b^*$ globally, appending the new symbol to $V$ and updating all sequences in the corpus.

• Output: The finalized vocabulary $V$ and the deterministic merge order (Metinov et al., 25 Nov 2025).

Pseudocode formalizing this process:

V = {all Unicode chars in C} ∪ {[PAD], [UNK], [CLS], [SEP], [MASK]} while len(V) < V_target: # Count all adjacent pairs p = get_adjacent_pair_probabilities(C) (a_star, b_star) = argmax(p) V.add(merge(a_star, b_star)) C = apply_merge(C, a_star, b_star)

The merge objective ensures that the discovered subwords maximize data compression by lexical frequency, leading to high coverage for both stems and affixes crucial in Kyrgyz.

3. Vocabulary Size, Token Selection, and OOV Handling

Bert-Kyrgyz-Tokenizer’s vocabulary is strictly capped at 30,522 tokens, determined by the ranked frequency of discovered subwords plus the five required special symbols. The subword selection criterion ensures that only the most frequent morpheme-sized units and productive affixations are retained. OOV handling is greedy and multi-level: unknown words are first greedily segmented into the longest matching subwords, defaulting to single characters when necessary; any residual out-of-inventory symbols are mapped to [UNK]. In practice, this design yields a negligible OOV rate (<0.5%) over 1.5 million pre-training sentences, with >99.5% of tokens fully decomposed to known subwords (Metinov et al., 25 Nov 2025).

Token Type	Example Kyrgyz Token	Function / Note
Root	мектеп	High-frequency lexical stem
Suffix	##ке	Morphological dative affix
Special	[CLS], [SEP], [MASK]	Sequence, separation, masking

4. Morphologically Faithful Tokenization: Illustrative Examples

The tokenization process preserves morpheme demarcations, as demonstrated by the following examples:

• Sentence: Мен мектепке бардым
  • Here, “мектеп” (root: school) and “##ке” (dative suffix) are individuated, allowing morpheme-specific embedding learning.
• Sentence: Китептеримди үйгө алып келем
  • “Китеп” (book), plural “##тер”, 1SG possessive “##им”, accusative “##ди”; “үй” (house) and dative “##гө” are each tokenized as discrete morphemes (Metinov et al., 25 Nov 2025).

This approach allows the model to construct compositional embeddings for any valid morphological sequence, significantly enhancing its ability to cope with Kyrgyz word formation processes.

5. Integration into BERT Architectures

Tokens generated by the Bert-Kyrgyz-Tokenizer are mapped to embeddings in a fixed token-to-vector matrix of shape (30,522 × 512). Positional embeddings for positions 0–511 and standard BERT segment embeddings are summed with token embeddings before input to the six-layer, eight-head BERT transformer. Special tokens serve their BERT-standard roles ([CLS] for sequence-level context, [SEP] for separation, [MASK] for masked language modeling, [PAD]/[UNK] for padding or unknowns). Although only masked language modeling (MLM) is pre-trained in KyrgyzBERT, full BERT sentence A/B infrastructure is retained for architectural compatibility. Thus, pre-processing consists of whitespace tokenization, WordPiece segmentation, and augmentation with special symbols to produce model-ready input (Metinov et al., 25 Nov 2025).

6. Quantitative Evaluation and Downstream Impact

While no standalone intrinsic tokenizer evaluation (e.g., subword sequence perplexity, explicit token entropy) is reported, empirical performance demonstrates the effectiveness of the approach:

• OOV tokenization rate is <0.5% on a 1.5M-sentence corpus.
• In downstream sentiment classification (kyrgyz-sst2), KyrgyzBERT with this tokenizer yields an F1-score of 0.8280, very close to mBERT’s 0.8401 (with only 20% of the parameters), and outperforms baseline multilingual WordPiece models in capturing Kyrgyz morphology (Metinov et al., 25 Nov 2025).

A plausible implication is that morphology-aware subword tokenization confers competitive downstream efficacy with significantly reduced model footprint, a critical factor for low-resource language processing.

7. Extensions: Adaptation and Controlled Vocabulary Growth

Recent research proposes methods for efficiently adapting existing English-oriented BERT tokenizers to Kyrgyz. Continued BPE training extends the existing merge sequence with Kyrgyz-specific data, ensuring all newly introduced tokens are guaranteed reachable by the merge rules. Embedding weights for new subwords are initialized by Fast Vocabulary Transfer—averaging embeddings of their decompositions under the original vocabulary. Leaf-based vocabulary pruning further improves efficiency by removing redundant leaf tokens (tokens with no descendants in the merge graph), guided by usage frequency. These techniques systematically optimize compression, minimize unreachable tokens, and enable controlled specialization of existing tokenizers for Kyrgyz, with practical implementation available as open-source tooling (Purason et al., 3 Dec 2025).