---
title: 'BanglishBERT: Code-Mixed NLP Transformer'
url: https://www.emergentmind.com/topics/banglishbert-model
type: topic
---

# BanglishBERT: Code-Mixed NLP Transformer

BanglishBERT is a Transformer-based language model specifically designed for Natural Language Processing (NLP) tasks involving Bangla-English code-mixed text and Romanized Bangla (termed “Banglish”). Under the ELECTRA training paradigm, BanglishBERT functions as a discriminator for Replaced Token Detection (RTD), aiming to distinguish original input tokens from those generated as corruptions. Its architecture, pretraining regimen, tokenization, fine-tuning procedures, and empirical evaluation have yielded state-of-the-art results for sentiment analysis and multimodal classification in low-resource Bangla and code-mixed domains, as evidenced by deployments on e-commerce reviews and meme sentiment tasks [2401.09446, 2412.13161].

## 1. Model Architecture and Training

BanglishBERT is based on the BERT family of Transformer encoder models, adopting the ELECTRA framework for pretraining. Its specification typically follows:

- **Transformer configuration**: 12 encoder layers, hidden size of 768, 12 attention heads, feed-forward dimension of 3072. The parameter count is approximately 100 million for the base model.
- **Training paradigm**: ELECTRA-style pretraining leverages a small generator network to corrupt tokens, with BanglishBERT acting as the discriminator and learning to classify each token as either “real” (original) or “fake” (replaced).
- **RTD loss function**:
  $$
  \mathcal{L}_{\mathrm{RTD}} = - \sum_{i=1}^{n}\Big[ \mathbb{1}\{x_i = \tilde x_i\}\log P_{\theta}(d_i = \text{real}\mid \mathbf{h}_i) + \mathbb{1}\{x_i \neq \tilde x_i\}\log P_{\theta}(d_i = \text{fake}\mid \mathbf{h}_i) \Big]
  $$
  Masked Language Modeling (MLM) and Next Sentence Prediction objectives are *not* used during discriminator pretraining [2401.09446, 2412.13161].

## 2. Data Sources and Preprocessing

Pretraining data for BanglishBERT consists of extensive Bangla and English corpora, as well as code-mixed and transliterated text (“Banglish”), though precise corpus size and domains are not always reported. For the BanglishRev project, 1.747 million e-commerce text reviews with diverse linguistic scripts and code-mixing proportions were used [2412.13161].

**Tokenization and normalization pipeline:**
- Unified WordPiece tokenizer with a typical vocabulary of ~50,000 subwords and maximum input length of 256 tokens.
- For each review, preprocessing steps include:
  - Removal of emojis and punctuation,
  - Script recognition (Bangla, English, Banglish via regex and NLTK),
  - Romanized Bangla is converted to phonetic Bangla via the Avro remapper,
  - Whitespace normalization.
- This pipeline ensures mixed-script and “Banglish” tokens are mapped to a robust subword vocabulary [2412.13161].

## 3. Fine-Tuning Procedures

Sentiment classification is the main downstream task. Inputs consist of tokenized review text or meme captions, bounded by [CLS] and [SEP]. The final [CLS] hidden state is passed through a linear classifier:

- **Sentiment classification**:
  - For multiclass (e.g. memes): a 3-way softmax layer, cross-entropy loss.
  - For binary sentiment (e-commerce): sigmoid/softmax for two labels, cross-entropy loss.
- **Example fine-tuning hyperparameters**:
  - Batch size: 32 (meme sentiment), 128 (BanglishRev reviews),
  - Optimizer: Adam(W), learning rate typically $1 \times 10^{-5}$–$5 \times 10^{-5}$,
  - Epochs: 3–5 (with slight overfitting observed beyond 2–3 epochs for large, noisy datasets),
  - Hardware for scale: NVIDIA A100 GPU, 40 GB VRAM [2401.09446, 2412.13161].

## 4. Multimodal Fusion and Classification

BanglishBERT has been integrated for multimodal classification, particularly in meme sentiment analysis:

- **Architecture**: Textual features (\(\mathbf{h}_{\text{CLS}}\)) from BanglishBERT are paired with visual features (\(\mathbf{h}_{\text{image}}\)) from ResNet50 (pretrained on ImageNet).
- **Fusion pipeline**:
  - Each modality projective via 1-layer MLP (20 units, ReLU),
  - Embeddings concatenated to form a 40-dimensional input to final classifier,
  - Cross-entropy loss for final predictions [2401.09446].

**Performance metrics on held-out meme sentiment test set:**

| Model                  | Accuracy | Precision | Recall | F1-score |
|------------------------|----------|-----------|--------|----------|
| BanglishBERT (text)    | 0.66     | 0.66      | 0.66   | 0.66     |
| ResNet50 (image)       | 0.72     | 0.67      | 0.72   | 0.69     |
| BanglishBERT+ResNet50  | 0.74     | 0.69      | 0.74   | 0.71     |

Neutral sentiment remains challenging; multimodal and image-only models failed to classify neutral examples, while BanglishBERT correctly labeled only 6 of 58 [2401.09446].

## 5. Large-Scale Sentiment Classification Using BanglishRev

BanglishBERT, fine-tuned on BanglishRev’s 1.747 million reviews, achieved state-of-the-art results for binary sentiment analysis:

- **Labeling**: Ratings ≥4 classified as positive, ≤3 as negative.
- **Evaluation**: Tested against an out-of-domain, manually annotated dataset (78,000 reviews; Rashid et al. 2024).
- **Performance**:
  - Direct fine-tuning: 93% accuracy, F1=0.93 (negative-class F1=0.78).
  - BanglishRev pretraining: Up to 95% accuracy, F1=0.94 (negative-class F1=0.79).
  - Robustness to review length and code-mixing across scripts [2412.13161].
- After 1 epoch, performance peaks; further epochs led to slight overfitting.

This suggests the RTD paradigm and large-scale noisy supervision confer strong generalization to out-of-domain reviews and diverse code-mixed compositions.

## 6. Comparison to Multilingual and Distilled Baselines

BanglishBERT has been benchmarked against mBERT, XLM-R, and smaller distilled models such as Tri-Distil-BERT and Mixed-Distil-BERT [2309.10272].

**Table: Weighted F1-score on synthetic Banglish test corpus**

| Task                 | mBERT | XLM-R | BanglishBERT | Tri-Distil-BERT | Mixed-Distil-BERT |
|----------------------|-------|-------|--------------|-----------------|-------------------|
| Emotion detection    | 0.49  | 0.51  | 0.47         | 0.48            | 0.50              |
| Sentiment analysis   | 0.74  | 0.77  | 0.72         | 0.69            | 0.70              |
| Offensive detection  | 0.88  | 0.88  | 0.86         | 0.86            | 0.87              |

Mixed-Distil-BERT, with only 6 layers, closes more than 90% of the performance gap to XLM-R, outperforming BanglishBERT on specific emotion and offensive-language tasks while being substantially more compact. BanglishBERT exhibits competitive sentiment analysis accuracy despite its larger architecture. A plausible implication is that synthetic code-mixed pretraining benefits smaller models most, though larger ELECTRA-based models maintain an edge on authentic human-written code-mixed data.

## 7. Explainability and Analysis Techniques

Explainable AI methods such as LIME have been applied to interpret BanglishBERT decisions:

- LIME visualizations highlight evidence supporting or opposing classification outputs for both image and text models.
  - Image: green/red segmentation patches,
  - Text: orange, green, blue token spans for positive, negative, neutral sentiment.
- BanglishBERT accurately tags sentiment-bearing Bangla and English tokens.
- In case studies, the visual model often focuses on faces, neglecting textual sentiment cues, motivating the multimodal approach [2401.09446].

## 8. Limitations and Future Directions

Key limitations observed include:

- **Class imbalance** (majority-positive data in BanglishRev) and difficulty with short reviews.
- **Inferior neutral-class classification** in meme tasks, even under multimodal fusion.
- **Tokenization robustness**: The normalization and subword pipeline mitigates script and spelling variation without special tokenizers.
- **Overfitting**: Increasing epochs on large, noisy data can degrade generalization.

Prospective avenues include spam detection via time-stamped meta-data, leveraging image content in reviews, and pivoting to fine-grained emotion labeling as annotated corpora develop [2412.13161].

BanglishBERT represents a scalable ELECTRA-based solution for sentiment and emotion analysis across code-mixed Bangla-English domains, outperforming baseline models in both unimodal and multimodal scenarios, and providing rich interpretability for practical deployment.

Source: https://www.emergentmind.com/topics/banglishbert-model