BanglaBERT: Bengali Pretrained Language Model

• BanglaBERT is a family of monolingual Bengali pretrained language models built on BERT-Base and ELECTRA-Base architectures, optimized for low-resource natural language understanding.
• It leverages advanced pretraining techniques such as replaced token detection and masked language modeling to achieve state-of-the-art performance on varied tasks including sentiment analysis, fake news detection, and named entity recognition.
• The model is designed for flexible downstream adaptation with practical applications in social media analysis, environmental discourse, and online moderation, supported by extensive pretraining on diverse Bengali text corpora.

BanglaBERT is a family of monolingual Bengali (“Bangla”) pretrained LLMs adhering closely to the BERT or ELECTRA transformer architectures, optimized for natural language understanding tasks in the low-resource Bengali language. First introduced by the BUET NLP group (Bhattacharjee et al., 2021), BanglaBERT demonstrated state-of-the-art performance on a broad suite of core Bengali NLP benchmarks and was subsequently extended for diverse domains including social media depression detection, hyperpartisan news detection, fake news classification, environmental discourse, communal-violence moderation, and named entity recognition.

1. Model Architecture and Variants

BanglaBERT predominantly implements the BERT-Base or ELECTRA-Base architecture:

• Transformer Encoder Layers: 12
• Hidden Size: 768
• Self-Attention Heads: 12
• Intermediate (Feed-forward) Dimension: 3072
• Vocabulary: 30,000–32,000 WordPiece tokens (with support for romanized/codeswitched Bangla)
• Total Parameters: ≈110 million
• Maximum Sequence Length: 512 tokens (some finetunings extend to 768 tokens via padding/truncation)

Some variants, such as BanglaBERT-Large, employ 24 layers, a 1024-d hidden size, and 16 attention heads (Farhan et al., 30 Jan 2024). Most practical downstream models integrate only the base architecture, occasionally adding lightweight multi-layer perceptron heads for task-specific adaptation (Wasi et al., 22 Oct 2024).

The original release (Bhattacharjee et al., 2021) substituted BERT’s masked language modeling (MLM) and next-sentence prediction (NSP) objectives with ELECTRA’s replaced token detection (RTD), in which a generator predicts corrupted tokens and a discriminator classifies whether each token is “replaced” or original. Later domain-specific replications and baselines sometimes revert to standard BERT-Base (MLM+NSP) pretraining (Chowdhury et al., 14 Jan 2024, Hasan et al., 28 Jul 2025).

2. Pretraining Corpus Construction and Tokenization

The pretraining corpora for BanglaBERT models aggregate large-scale, diverse text sources to maximize language coverage:

• Raw Data Volume: 27.5 GB (“Bangla2B+”) in the original, 1.75 billion tokens in later variants (Bhattacharjee et al., 2021, Khondoker et al., 24 Jun 2025)
• Primary Sources:
  • Wikipedia snapshots
  • High-traffic Bangla news outlets (Prothom Alo, BDNews24, etc.)
  • Common Crawl/OSCAR Bengali slices
  • Blogs, social networks, and general web content
  • E-books, legal codes, and song lyrics
• Preprocessing Steps:
  • Unicode normalization and punctuation filtering
  • Document and near-duplicate removal
  • Language identification via FastText to filter non-Bangla text
  • Lowercasing and special handling for named entities and code-switching
• Tokenization:
  • WordPiece (BERT-standard) with vocabularies ranging between 30,000 and 102,025 (the latter in models employing broader subword morph segmentation (Chowdhury et al., 2023))
  • Subword units designed for rich Bangla morphology, compound words, and diacritics

3. Pretraining Objectives and Optimization

The dominant pretraining paradigm is ELECTRA’s replaced token detection (RTD):

$$\mathcal{L}_D = -\mathbb{E}_x \sum_{i=1}^{n} [ y_i \log P_D(y_i=1|\tilde x) + (1-y_i)\log P_D(y_i=0|\tilde x) ]$$

where the generator $G$ performs masked language modeling,

$$\mathcal{L}_G = -\mathbb{E}_x \sum_{i \in M} \log P_G(x_i|x_{[mask]})$$

and $y_i$ is an indicator for whether token $i$ is original or replaced.

Several follow-up works and benchmark replications employ classic BERT objectives—MLM:

$$\mathcal{L}_{\mathrm{MLM}} = -\sum_{t \in \mathcal{M}} \log p(x_t | x_{\setminus \mathcal{M}})$$

and NSP, optimizing over token-wise negative log-likelihood. Across all variants, pretraining employs the Adam or AdamW optimizer, with batch sizes between 64 and 256, learning rates $2 \times 10^{-4}$ (ELECTRA pretraining) or $2 \times 10^{-5}$ (standard BERT fine-tuning), and training durations up to 2.5 million steps (Bhattacharjee et al., 2021).

4. Downstream Tasks and Fine-tuning Methodologies

BanglaBERT models are routinely finetuned on a wide range of Bangla NLP applications:

4.1 Text Classification and Sentiment Analysis

• Depression Detection on Social Media: Fine-tuned on 28,000 Bengali Reddit/X posts (Bengali Social Media Depressive Dataset), achieving accuracy 0.8604 and F1 0.8625 for binary classification. SahajBERT (ALBERT-based) slightly outperforms BanglaBERT on this benchmark (Chowdhury et al., 14 Jan 2024).
• Sentiment Analysis: Three-way social media sentiment (SentNoB); BanglaBERT attains macro-F1 72.89, outperforming mBERT and XLM-R (Bhattacharjee et al., 2021).

4.2 Fake News and Hyperpartisan Detection

• Fake News Classification: Ensemble strategies blending data augmentation (masked-LM augmentation, back-translation, paraphrasing) yield accuracy up to 96% on test data; summarization helps only for articles well above 512 tokens (Chowdhury et al., 2023).
• Hyperpartisan News: Semi-supervised fine-tuning and LIME-based interpretability yield F1 0.9544, surpassing all conventional baselines. Pseudo-labeling of ~18,500 high-confidence unlabeled examples amplifies performance (Hasan et al., 28 Jul 2025).

4.3 Environmental and Climate News Analysis

• Dhoroni Dataset: Multi-class and multi-label tasks targeting news stance, authenticity, political influence, and more, with macro-F1 up to 0.465 on tasks with clear lexical cues. Model architecture comprises the base BanglaBERT encoder and a two-layer [128,128] MLP (Wasi et al., 22 Oct 2024). Distributional alignment and minority class oversampling are key.

4.4 Social Conflict and Violence Moderation

• Communal Violence Detection: Four-way classification over social media comments, with base BanglaBERT obtaining macro-F1 0.60 and ensemble wrappers up to 0.63. Cosine similarity and LIME highlight embedding overlap between communal and non-communal classes, pinpointing future adaptation needs (Khondoker et al., 24 Jun 2025).

4.5 Named Entity Recognition

• NER (MultiCoNER, Gazetteer-Enhanced): Base and large models, with embeddings feeding k-means–enhanced CRFs. BanglaBERT-Large + Gazetteer + CRF achieves macro-F1 0.8267 (test). ELECTRA-style pretraining and external gazetteers yield substantial sequence labeling gains (Farhan et al., 30 Jan 2024).

5. Empirical Benchmarking and Performance

Key empirical findings are summarized below:

Task / Dataset	BanglaBERT Results	Reference
Sentiment (SentNoB, macro-F1)	72.89	(Bhattacharjee et al., 2021)
NLI (BNLI, accuracy)	82.80	(Bhattacharjee et al., 2021)
NER (MultiCoNER, micro-F1)	77.78	(Bhattacharjee et al., 2021)
QA (TyDiQA, F1)	79.34	(Bhattacharjee et al., 2021)
Depression Detection (BSMDD, F1)	0.8625	(Chowdhury et al., 14 Jan 2024)
Hyperpartisan News (BNAD, F1)	0.9544	(Hasan et al., 28 Jul 2025)
Fake News (multiple test sets, accuracy)	0.96 (max)	(Chowdhury et al., 2023)
Comm. Violence (macro-F1, ensemble)	0.63	(Khondoker et al., 24 Jun 2025)
NER + Gazetteer + K-means CRF (macro-F1)	0.8267	(Farhan et al., 30 Jan 2024)

A salient empirical observation: monolingual BanglaBERT models consistently outperform mBERT and XLM-R in Bengali tasks, especially in low-data regimes (<1k training samples) (Bhattacharjee et al., 2021). Augmentation and semi-supervised learning can further bridge data scarcity.

6. Model Analysis, Limitations, and Interpretability

• Strengths:
  • Captures Bangla-specific morphology, syntax, and compounds more effectively than multilingual analogs (Bhattacharjee et al., 2021, Hasan et al., 28 Jul 2025).
  • Shows sample efficiency and robust transfer to diverse domains—news, social platforms, science, law.
  • Extensible: plug-in MLP heads, CRF post-processing, or domain-specific adapters.
• Weaknesses:
  • Performance can lag when labeled data are sparse for complex, world-knowledge–dependent tasks (e.g., fine-grained stance, environmental topic attribution) (Wasi et al., 22 Oct 2024).
  • Embedding proximity between semantically close classes (e.g., communal and non-communal terms) impedes fine-grained moderation (Khondoker et al., 24 Jun 2025).
  • ELECTRA-style discriminators do not supply generative NLG heads (Bhattacharjee et al., 2021).
  • Susceptibility to domain shift, artifacts from data augmentation, and annotation noise.
• Interpretability Approaches:
  • LIME widely used to uncover influential Bengali tokens/patterns for trust building and error analysis in classification tasks (Hasan et al., 28 Jul 2025, Khondoker et al., 24 Jun 2025).
  • Clustering hidden-state embeddings via k-means provides effective discretization for downstream sequence models (NER) (Farhan et al., 30 Jan 2024).
  • Error and ablation studies highlight importance of careful pretraining corpus construction and class balancing.

7. Research Directions and Future Extensions

Authors and the broader Bengali NLP community have identified several directions to advance BanglaBERT:

• Expanded Evaluation: Extending BLUB with syntactic, morphological, coreference, and information extraction tasks (Bhattacharjee et al., 2021).
• Multi-Task and Adapter-Based Learning: Simultaneous adaptation for multiple downstream perspectives, leveraging shared knowledge across overlapping problem domains (Wasi et al., 22 Oct 2024).
• Domain Adaptation: Continued pretraining on task- or domain-specific Bangla data for improved disambiguation and contextualization, as in communal violence or environmental news (Khondoker et al., 24 Jun 2025, Wasi et al., 22 Oct 2024).
• Low-Resource Transfer: Cross-lingual transfer (e.g., Hindi/Urdu, Multilingual BERT) and semi-supervised techniques to alleviate label scarcity (Hasan et al., 28 Jul 2025).
• NLG and Generative Models: Development of encoder–decoder architectures and curriculum-based pretraining to enhance generation capabilities and handle dialect/colloquial variation (Bhattacharjee et al., 2021).
• Deployment and Moderation: Operationalizing BanglaBERT for real-time moderation, early-warning in online spaces, and support for Bangla user safety.

BanglaBERT defines the de facto foundation for state-of-the-art Bengali language understanding and task transfer in low-resource settings (Bhattacharjee et al., 2021, Chowdhury et al., 14 Jan 2024, Hasan et al., 28 Jul 2025, Chowdhury et al., 2023, Wasi et al., 22 Oct 2024, Khondoker et al., 24 Jun 2025, Farhan et al., 30 Jan 2024).