BERTić: South Slavic Electra Transformer

• BERTić is a monolingual Electra-style transformer pre-trained on over eight billion tokens from Bosnian, Croatian, Montenegrin, and Serbian content, establishing state-of-the-art benchmarks in South Slavic NLP.
• It utilizes a generator–discriminator architecture with WordPiece tokenization and Electra’s replaced token detection objective, achieving superior performance in morphosyntactic tagging, NER, and geolocation tasks.
• Resource release on HuggingFace and fine-tuning guidelines support domain adaptation, while future research aims to extend dialect coverage and address potential domain biases.

BERTić is a monolingual Electra-style transformer LLM specifically pre-trained on over eight billion tokens of web-crawled text in Bosnian, Croatian, Montenegrin, and Serbian. Utilizing a dedicated regional corpus and an Electra-style replaced token detection pre-training objective, BERTić achieves state-of-the-art performance across multiple morphosyntactic, semantic, and geolocation tasks, surpassing both multilingual and existing monolingual transformer baselines on a diversity of benchmarks relevant to South Slavic languages (Ljubešić et al., 2021).

1. Model Architecture

BERTić implements the “base” transformer encoder design, following the architectural configuration from Devlin et al. (2019) and Clark et al. (2020) but with the Electra generator/discriminator pre-training regime. Key characteristics include:

• Discriminator (“BERTić proper”): 12 transformer encoder layers, hidden (model) dimension $d=768$, 12 self-attention heads (each of size 64), intermediate feed-forward layer dimension $d_{ff}=3072$, with a total parameter count of approximately 110 million.
• Generator: A smaller auxiliary network, typically one quarter the width of the discriminator, used during pre-training for proposing replacements for masked tokens, as in the Electra-base configuration.
• No architectural modifications are introduced beyond substituting BERT's MLM objective with Electra’s replaced token detection (RTD).

2. Tokenization and Vocabulary

BERTić employs the WordPiece tokenization strategy as in the original BERT, with the following properties:

• Vocabulary size: 32,768 tokens.
• Special tokens: [CLS], [SEP], [PAD], [MASK], [UNK].
• Vocabulary training utilized 10 million randomly sampled paragraphs from the combined corpus; unlike BERT’s original pipeline for English, BERTić retains all Unicode codepoints, ensuring the preservation of diacritics and special characters present in South Slavic languages.

3. Pre-training Data and Procedure

The pre-training corpus comprises 8,387,681,518 words collected from multiple curated datasets: hrWaC (Croatian web crawl, 2011/2014), CLASSLA-hr (Croatian, 2019), cc100-hr (Croatian CommonCrawl), Riznica (Croatian literature and newspapers), srWaC, CLASSLA-sr, cc100-sr (Serbian), bsWaC and CLASSLA-bs (Bosnian), and cnrWaC (Montenegrin). All corpora underwent deduplication at the sentence level to remove 5–15% overlap.

Pre-training strictly follows the Electra framework:

• Generator Objective (MLM):

$$\mathcal{L}_{\mathrm{MLM}}(\theta_G) = -\mathbb{E}_{x \sim \mathcal{D}}\sum_{i \in M} \log P_G(x_i \mid x_{\setminus M})$$

where $M \subset \{1, \ldots, n\}$ marks masked positions.

• Discriminator Objective (RTD):

$$\mathcal{L}_{\mathrm{RTD}}(\theta_D) = -\mathbb{E}_{x \sim \mathcal{D}} \sum_{i=1}^n \left[I_i \log D(\tilde{x})_i + (1 - I_i)\log(1 - D(\tilde{x})_i) \right]$$

with $I_i=1$ if $\tilde{x}_i = x_i$, $0$ otherwise.

• Training parameters: batch size of 1,024, distributed across eight TPUv3 cores; two million total training steps (approximately 50 epochs); Adam optimizer ($\beta_1 = 0.9$, $\beta_2 = 0.999$); linear learning rate warm-up (first 10% of steps) and decay; Electra-base peak learning rate and weight decay.

4. Benchmark Evaluation and Results

BERTić was systematically evaluated against CLASSLA (a BiLSTM with static embeddings), mBERT (multilingual BERT), and cseBERT (CroSloEngual BERT, monolingual Croatian) on four task categories:

Task & Dataset	CLASSLA	mBERT	cseBERT	BERTić
Morphosyntactic Tagging (μF₁, %)
hr500k (Croatian)	93.87	94.60	95.74	95.81
ReLDI-hr (Croatian Twitter)	-	88.87	91.63	92.28
SETimes.SR (Serbian)	95.00	95.50	96.41	96.31†
ReLDI-sr (Serbian Twitter)	-	91.26	93.54	93.90
Named Entity Recognition (F₁, %)
hr500k	80.13	85.67	88.98	89.21
ReLDI-hr	-	76.06	81.38	83.05
SETimes.SR	84.64	92.41	92.28	92.02†
ReLDI-sr	-	81.29	82.76	87.92
Social Media Geolocation (km)
Median/Mean Error (lower=better)	107.10/145.72	42.25/82.05	40.76/81.88	37.96/79.30
Commonsense Causal Reasoning
COPA-HR Accuracy (%)	50.00†	54.12	61.80	65.76

† Not a statistically significant difference vs. best baseline.

These results confirm BERTić’s advances over state-of-the-art models for part-of-speech tagging, NER, and geolocation, with especially notable gains on tasks requiring broader world knowledge (NER, geolocation, causal reasoning). For commonsense causal reasoning, COPA-HR (a Croatian translation of the COPA dataset) was introduced; BERTić achieved 65.76% accuracy, compared to 61.80% for cseBERT and 54.12% for mBERT.

5. Resource Release and Downstream Fine-tuning

BERTić is available via HuggingFace (https://huggingface.co/classla/bcms-bertic), released under a permissive research license. Recommended fine-tuning hyper-parameters for downstream tasks were:

• Token classification (POS, NER): Learning rate search in $d_{ff}=3072$0, epochs in $d_{ff}=3072$1, optimized with Bayesian search.
• COPA-HR: Learning rate $d_{ff}=3072$2, batch size 16, 30 epochs yielded stable outcomes.
• Geolocation: Batch size 64, 40 epochs, dev-set median error checkpoint selection provided optimal performance.

A model card and reproducible recipes for further domain/task adaptation are provided.

6. Limitations and Observed Biases

Performance improvements are more pronounced on knowledge-rich tasks than surface morphosyntax, especially when the test data originate from a single low-diversity source (e.g., SETimes.SR). The web-crawl dominant training corpus introduces potential domain bias (news, forums, blogs), which may inhibit BERTić’s zero-shot generalization to underrepresented or highly specialized domains (e.g., legal, medical). Systematic assessment of gender, regional, or register-based biases in the model outputs has not been conducted.

7. Future Directions

Planned research areas include:

• Extension to additional South Slavic dialects and varieties.
• Domain-adaptive pre-training using corpora from legal and medical domains.
• Comprehensive audits of potential biases and fairness along gender and ethnic dimensions.
• Expanded integration with knowledge-centric tasks, such as fact verification and entity linking.

BERTić is the first transformer model explicitly pre-trained on a dedicated large-scale Bosnian/Croatian/Montenegrin/Serbian corpus, establishing new state-of-the-art benchmarks for linguistic and semantic tasks in South Slavic computational linguistics (Ljubešić et al., 2021).