---
title: Kyrgyz-SST2 Benchmark for Sentiment Analysis
url: https://www.emergentmind.com/topics/kyrgyz-sst2-benchmark
type: topic
---

# Kyrgyz-SST2 Benchmark for Sentiment Analysis

The Kyrgyz-SST2 benchmark is a binary sentiment analysis corpus for Kyrgyz, constructed to facilitate the evaluation and development of language models and sentiment classifiers for Kyrgyz, a morphologically rich, low-resource Turkic language. It is derived from the Stanford Sentiment Treebank (SST-2) and provides standardized training, development, and gold-standard test splits with closely matched class distributions to the English original. Kyrgyz-SST2 was introduced alongside KyrgyzBERT— the first publicly available monolingual BERT-based model for the language—and is instrumental for reproducible experiments in Kyrgyz NLP [2511.20182].

## 1. Dataset Construction Protocol

Kyrgyz-SST2 is anchored on the SST-2 benchmark, which provides binary sentiment annotations (positive, negative) for short movie-review sentences. All English sentences in the original training (67,349 instances) and development (872 instances) sets were translated to Kyrgyz using a Transformer-based neural machine translation (NMT) system employing beam search, trained on parallel Kyrgyz–English data. The original SST-2 sentiment labels were directly transferred to their respective Kyrgyz translations for these splits.

For the test split, 1,821 English sentences were translated using the same NMT pipeline. Native Kyrgyz speakers subsequently conducted manual annotation following explicit guidelines: sentences were assigned a “positive” label if the sentiment was favorable, “negative” otherwise. Rare instances of neutral or mixed sentiment were adjudicated by discussion until consensus. Further, ambiguous cases underwent re-evaluation. To ensure annotation reliability, a random 10% subset of test sentences was double-annotated by an independent native speaker, yielding an inter-annotator agreement (Cohen’s κ) exceeding 0.92, demonstrating high annotation consistency [2511.20182].

## 2. Dataset Composition and Quantitative Characteristics

Kyrgyz-SST2 preserves the structural properties and label distributions of English SST-2, supporting controlled cross-linguistic evaluation. The split sizes and sentiment class ratios are nearly identical to the source, minimizing distributional shift. The following summarizes the data breakdown:

| Split  | N_samples | % positive | % negative |
|--------|-----------|------------|------------|
| Train  | 67,349    | 51.7%      | 48.3%      |
| Dev    | 872       | 51.6%      | 48.4%      |
| Test   | 1,821     | 51.6%      | 48.4%      |

Kyrgyz exhibits agglutinative morphology, with words often incorporating multiple affixes. The dataset exhibits an average of ~20 subword tokens per sentence (standard deviation ≈ 5) following application of the custom WordPiece tokenizer. The mean sentence length pre-tokenization is 12.5 words (σ ≈ 3), reflecting the complexity and nuance of Kyrgyz morphological structures in the domain of sentiment expression [2511.20182].

## 3. Tokenization and Data Preprocessing

A WordPiece tokenizer (‘bert-kyrgyz-tokenizer’) was trained from scratch on a 1.5 million-sentence Kyrgyz corpus to address the language’s morphological complexity and frequent out-of-vocabulary phenomena. The resulting vocabulary contains 30,522 tokens and includes [PAD], [UNK], [CLS], [SEP], and [MASK] as special tokens. The tokenizer is optimized to learn subword representations that typically correspond to lexical stems combined with affixes, mitigating the sparsity induced by agglutination.

Preprocessing entails:

1. Unicode normalization (NFC)
2. Lowercasing all text
3. WordPiece tokenization
4. Mapping to input IDs and appending [CLS] and [SEP] consistent with BERT input conventions

This approach aligns with best practices for BERT-style pretraining and finetuning regimes, allowing models to effectively capture the underlying syntactic and semantic properties of Kyrgyz sentences [2511.20182].

## 4. Evaluation Metrics and Formal Definitions

Model performance on Kyrgyz-SST2 is primarily assessed using weighted F₁-score and accuracy. Let TP, TN, FP, FN denote true positives, true negatives, false positives, and false negatives, respectively. The following definitions are used:

**Accuracy:**
\[
\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}
\]

**Precision:**
\[
\mathrm{Precision} = \frac{TP}{TP + FP}
\]

**Recall:**
\[
\mathrm{Recall} = \frac{TP}{TP + FN}
\]

**F₁-score:**
\[
F_{1} = 2 \times \frac{\mathrm{Precision}\;\times\;\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}
\]

Weighted F₁-score serves as the primary metric due to potential implications of slight label imbalance and its heightened sensitivity to both precision and recall [2511.20182].

## 5. Model Benchmarking and Experimental Results

Kyrgyz-SST2 was used to benchmark both monolingual and multilingual transformer-based models. KyrgyzBERT (35.9M parameters) and multilingual BERT (mBERT, 177M parameters) were finetuned on the training set for 3 epochs with a learning rate of 2e-5. Additionally, zero-shot performance was evaluated using XLM-RoBERTa and mBERT without Kyrgyz-specific finetuning. Summarized results:

| Model                    | Params (M) | F₁-score | Accuracy |
|--------------------------|------------|----------|----------|
| KyrgyzBERT (finetuned)   | 35.9       | 0.8280   | 0.8320   |
| mBERT (finetuned)        | 177.0      | 0.8401   | 0.8453   |
| XLM-RoBERTa (zero-shot)  | 270.0      | 0.3221   | 0.3364   |
| mBERT (zero-shot)        | 177.0      | 0.3509   | 0.3642   |

KyrgyzBERT achieves an F₁ of 0.8280, only 1.2% below finetuned mBERT, despite being 5× smaller. Both multilingual models perform near chance in the zero-shot setting, highlighting the necessity of task-specific finetuning and a high-quality Kyrgyz evaluation set. This suggests that monolingual models are competitive in resource-constrained settings and that cross-lingual transfer is non-trivial for Kyrgyz [2511.20182].

## 6. Data Release, Licensing, and Usage Scenarios

The full Kyrgyz-SST2 dataset, tokenizer, pretrained KyrgyzBERT, and finetuned checkpoints are publicly accessible via the Hugging Face Hub (https://huggingface.co/metinovadilet), distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. The dataset is available in JSON/CSV format, with each record consisting of:

- `sentence`: Kyrgyz movie-review sentence (string)
- `label`: sentiment label (0=negative, 1=positive)

Sample loading code using `datasets` (v1.18+):

```python
from datasets import load_dataset

ds = load_dataset("metinovadilet/kyrgyz-sst2")
print(ds["train"][0]) # {'sentence': 'бул кино жакшы...', 'label': 1}
```

This artifact constitutes a reliable, reproducible baseline for research targeting Kyrgyz sentiment classification and the development or benchmarking of Kyrgyz-specific NLP resources [2511.20182].

Source: https://www.emergentmind.com/topics/kyrgyz-sst2-benchmark