---
title: 'BERTić: South Slavic Electra Transformer'
url: https://www.emergentmind.com/topics/bertic-model
type: topic
---

# BERTić: South Slavic Electra Transformer

BERTić is a monolingual Electra-style transformer language model specifically pre-trained on over eight billion tokens of web-crawled text in Bosnian, Croatian, Montenegrin, and Serbian. Utilizing a dedicated regional corpus and an Electra-style replaced token detection pre-training objective, BERTić achieves state-of-the-art performance across multiple morphosyntactic, semantic, and geolocation tasks, surpassing both multilingual and existing monolingual transformer baselines on a diversity of benchmarks relevant to South Slavic languages [2104.09243].

## 1. Model Architecture

BERTić implements the “base” transformer encoder design, following the architectural configuration from Devlin et al. (2019) and Clark et al. (2020) but with the Electra generator/discriminator pre-training regime. Key characteristics include:

- Discriminator (“BERTić proper”): 12 transformer encoder layers, hidden (model) dimension $d=768$, 12 self-attention heads (each of size 64), intermediate feed-forward layer dimension $d_{ff}=3072$, with a total parameter count of approximately 110 million.
- Generator: A smaller auxiliary network, typically one quarter the width of the discriminator, used during pre-training for proposing replacements for masked tokens, as in the Electra-base configuration.
- No architectural modifications are introduced beyond substituting BERT's MLM objective with Electra’s replaced token detection (RTD).

## 2. Tokenization and Vocabulary

BERTić employs the WordPiece tokenization strategy as in the original BERT, with the following properties:

- Vocabulary size: 32,768 tokens.
- Special tokens: [CLS], [SEP], [PAD], [MASK], [UNK].
- Vocabulary training utilized 10 million randomly sampled paragraphs from the combined corpus; unlike BERT’s original pipeline for English, BERTić retains all Unicode codepoints, ensuring the preservation of diacritics and special characters present in South Slavic languages.

## 3. Pre-training Data and Procedure

The pre-training corpus comprises 8,387,681,518 words collected from multiple curated datasets: hrWaC (Croatian web crawl, 2011/2014), CLASSLA-hr (Croatian, 2019), cc100-hr (Croatian CommonCrawl), Riznica (Croatian literature and newspapers), srWaC, CLASSLA-sr, cc100-sr (Serbian), bsWaC and CLASSLA-bs (Bosnian), and cnrWaC (Montenegrin). All corpora underwent deduplication at the sentence level to remove 5–15% overlap.

Pre-training strictly follows the Electra framework:

- **Generator Objective (MLM):**
  $$
  \mathcal{L}_{\mathrm{MLM}}(\theta_G) = -\mathbb{E}_{x \sim \mathcal{D}}\sum_{i \in M} \log P_G(x_i \mid x_{\setminus M})
  $$
  where $M \subset \{1, \ldots, n\}$ marks masked positions.

- **Discriminator Objective (RTD):**
  $$
  \mathcal{L}_{\mathrm{RTD}}(\theta_D) = -\mathbb{E}_{x \sim \mathcal{D}} \sum_{i=1}^n \left[I_i \log D(\tilde{x})_i + (1 - I_i)\log(1 - D(\tilde{x})_i) \right]
  $$
  with $I_i=1$ if $\tilde{x}_i = x_i$, $0$ otherwise.

- Training parameters: batch size of 1,024, distributed across eight TPUv3 cores; two million total training steps (approximately 50 epochs); Adam optimizer ($\beta_1 = 0.9$, $\beta_2 = 0.999$); linear learning rate warm-up (first 10% of steps) and decay; Electra-base peak learning rate and weight decay.

## 4. Benchmark Evaluation and Results

BERTić was systematically evaluated against CLASSLA (a BiLSTM with static embeddings), mBERT (multilingual BERT), and cseBERT (CroSloEngual BERT, monolingual Croatian) on four task categories:

| Task & Dataset                  | CLASSLA | mBERT      | cseBERT    | BERTić     |
|----------------------------------|---------|------------|------------|------------|
| Morphosyntactic Tagging (μF₁, %) |         |            |            |            |
| hr500k (Croatian)                | 93.87   | 94.60      | 95.74      | **95.81**  |
| ReLDI-hr (Croatian Twitter)      | -       | 88.87      | 91.63      | **92.28**  |
| SETimes.SR (Serbian)             | 95.00   | 95.50      | 96.41      | 96.31†     |
| ReLDI-sr (Serbian Twitter)       | -       | 91.26      | 93.54      | **93.90**  |
| Named Entity Recognition (F₁, %) |         |            |            |            |
| hr500k                           | 80.13   | 85.67      | 88.98      | **89.21**  |
| ReLDI-hr                         | -       | 76.06      | 81.38      | **83.05**  |
| SETimes.SR                       | 84.64   | 92.41      | 92.28      | 92.02†     |
| ReLDI-sr                         | -       | 81.29      | 82.76      | **87.92**  |
| Social Media Geolocation (km)    |         |            |            |            |
| Median/Mean Error (lower=better) | 107.10/145.72|42.25/82.05|40.76/81.88|**37.96/79.30**|
| Commonsense Causal Reasoning     |         |            |            |            |
| COPA-HR Accuracy (%)             | 50.00†  | 54.12      | 61.80      | **65.76**  |

† Not a statistically significant difference vs. best baseline.

These results confirm BERTić’s advances over state-of-the-art models for part-of-speech tagging, NER, and geolocation, with especially notable gains on tasks requiring broader world knowledge (NER, geolocation, causal reasoning). For commonsense causal reasoning, COPA-HR (a Croatian translation of the COPA dataset) was introduced; BERTić achieved 65.76% accuracy, compared to 61.80% for cseBERT and 54.12% for mBERT.

## 5. Resource Release and Downstream Fine-tuning

BERTić is available via HuggingFace (https://huggingface.co/classla/bcms-bertic), released under a permissive research license. Recommended fine-tuning hyper-parameters for downstream tasks were:

- Token classification (POS, NER): Learning rate search in $[9 \times 10^{-6}, 1 \times 10^{-4}]$, epochs in $[3, 15]$, optimized with Bayesian search.
- COPA-HR: Learning rate $1 \times 10^{-5}$, batch size 16, 30 epochs yielded stable outcomes.
- Geolocation: Batch size 64, 40 epochs, dev-set median error checkpoint selection provided optimal performance.

A model card and reproducible recipes for further domain/task adaptation are provided.

## 6. Limitations and Observed Biases

Performance improvements are more pronounced on knowledge-rich tasks than surface morphosyntax, especially when the test data originate from a single low-diversity source (e.g., SETimes.SR). The web-crawl dominant training corpus introduces potential domain bias (news, forums, blogs), which may inhibit BERTić’s zero-shot generalization to underrepresented or highly specialized domains (e.g., legal, medical). Systematic assessment of gender, regional, or register-based biases in the model outputs has not been conducted.

## 7. Future Directions

Planned research areas include:

- Extension to additional South Slavic dialects and varieties.
- Domain-adaptive pre-training using corpora from legal and medical domains.
- Comprehensive audits of potential biases and fairness along gender and ethnic dimensions.
- Expanded integration with knowledge-centric tasks, such as fact verification and entity linking.

BERTić is the first transformer model explicitly pre-trained on a dedicated large-scale Bosnian/Croatian/Montenegrin/Serbian corpus, establishing new state-of-the-art benchmarks for linguistic and semantic tasks in South Slavic computational linguistics [2104.09243].

Source: https://www.emergentmind.com/topics/bertic-model