---
title: 'UrduBERT: Transformer Model for Urdu NLP'
url: https://www.emergentmind.com/topics/urdubert
type: topic
---

# UrduBERT: Transformer Model for Urdu NLP

UrduBERT is a transformer-based pretrained language representation model tailored for the Urdu language and Roman-Urdu code-mixed text, building on the standard BERT-Base architecture. UrduBERT is designed to address the performance limitations of multilingual language models in low-resource Urdu NLP tasks, offering enhanced contextual embeddings for both native and mixed-script utterances. Its applications span robust natural language understanding (NLU), especially in Automatic Speech Recognition (ASR)-related domains, and classification tasks such as hope-speech detection in social media. Empirical studies demonstrate its superior accuracy and robustness to linguistic variability and noise, establishing UrduBERT as a primary resource in Urdu NLP research [2401.09354], [2512.22705].

## 1. Architecture and Pretraining Configuration

UrduBERT employs the standard BERT-Base transformer architecture comprising:

- **Encoder layers:** 12 transformer blocks
- **Embedding dimensionality:** 768 (hidden size)
- **Self-attention heads:** 12 per layer
- **Feed-forward inner-layer size:** 3,072
- **Vocabulary:** ~60,000 WordPiece tokens for native Urdu, Roman-Urdu, and frequent English subwords [2401.09354]; alternatives include ~30,000 tokens in certain code-mixed variants [2512.22705]

The pretraining corpus integrates approximately 50 million sentences from sources such as Urdu Wikipedia, national newswire corpora, Common Crawl (Urdu subset), and wide-coverage web-crawled Urdu text [2401.09354]. Code-mixed versions involve Roman-Urdu social media and newswire (~5–10 GB), English Wikipedia, and Common Crawl snippets (~20 GB), yielding ~100–150 million tokens [2512.22705]. Pretraining objectives follow the original BERT protocol: masked language modeling (MLM) with 15% random token masking, and next-sentence prediction (NSP).

## 2. Tokenization and Text Normalization

UrduBERT utilizes a WordPiece tokenizer trained on multilayered corpora to segment Urdu, Roman-Urdu, and English text. The tokenizer is engineered for robust handling of out-of-vocabulary words and code-mixed sentences by breaking tokens into subword units instead of discarding them. Text normalization for Urdu and Roman-Urdu involves:

- Removal of diacritics (zabar, zer, pesh)
- Standardization of punctuation to ASCII
- Lowercasing Roman-Urdu
- Collapsing character repetitions (e.g., “claasss” → “class”)
- Padded/truncated sequence length up to 128 subword tokens [2512.22705]

No explicit language identification tags are used during code-mixing; the unified vocabulary encodes both scripts.

## 3. Fine-Tuning Methodologies

### ASR-Robust NLU Adaptation

For transfer learning in smart-home command understanding [2401.09354]:

- **Data Augmentation:** Audiomentations library introduces pitch shifts (±2 semitones), time-stretching (0.9×–1.1×), and multiple noise profiles (white Gaussian noise, babble, environmental) with SNR from 20 dB to 0 dB.
- **ASR Error Simulation:** Empirical confusion matrix simulates character/word-level errors at WER regimes of 10%, 20%, and 30%.
- **Freeze-Fine-Tune Strategy:** Bottom N=8 transformer layers frozen while the top 4 layers plus a random classification head are fine-tuned on domain data, followed by unfreezing all layers and learning-rate decay.
- **Cross-Validation:** 5-fold stratified cross-validation for stable generalization statistics
- **Domain Adaptation:** Tested on restaurant-booking data without further fine-tuning to assess transferability

### Hope Speech Detection

For hope-speech classification, UrduBERT is integrated with the GHaLIB framework [2512.22705]:

- **Classification Head:** Dropout (rate 0.1–0.3) applied to [CLS] embedding (768), followed by a linear projection to C output classes (C=2 for binary, C=4 for multi-class) and softmax activation
- **Weighted Cross-Entropy Loss:** Positive (“hope”) class receives 1.5× weight to mitigate class imbalance
- **Optimization:** AdamW optimizer with learning rates 5×10⁻⁶–5×10⁻⁵ and batch sizes 4–16, tuned via Optuna across 30 trials
- **Regularization:** Dropout and weight decay in [0.0, 0.1]; early stopping on validation F₁ within 3–5 epochs

## 4. Training Objectives and Regularization

For intent classification and robust NLU [2401.09354]:

- **Cross-Entropy Loss:**  
  $$\mathcal{L}_{CE} = -\sum_{i=1}^{C}y_{i}\log\hat y_{i} $$
  where $C$ = number of classes, $y_i$ = ground-truth one-hot label, $\hat y_i$ = softmax output

- **Consistency Loss:**  
  $$\mathcal{L}_{cons} = \lVert h(x_{clean}) - h(x_{noisy}) \rVert_2^2 $$
  $h(x)$ = penultimate layer output; aligns representations of clean and noise-augmented inputs

- **Total Loss:**  
  $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{cons} $$
  with $\lambda = 0.5$ as regularization weight

For hope-speech classification [2512.22705], standard weighted cross-entropy is employed, and evaluation metrics are macro-averaged accuracy, precision, recall, and F₁-score.

## 5. Core Datasets and Task Coverage

### Smart-Home ASR NLU

- **Data Source:** "Home Automation System in Urdu Language" (Kaggle)
- **Size:** ~8,000 audio-transcript pairs
- **Intent Taxonomy:** 20 distinct smart-home intents
- **Entities:** device_type, location, numeric_value, mode
- **Speaker Diversity:** 50+; widespread urban-rural and regional accent variability
- **Noise Conditions:** ambient (fan, traffic); WER assessment: 15% (clean), 30% (medium noise), 50% (heavy noise)

### Hope Speech Detection

- **Source:** PolyHope-M 2025 shared task
- **Size:** ≈8,000 labeled samples
- **Labels:** Generalized Hope, Realistic Hope, Unrealistic Hope, Not Hope
- **Class Distributions:** Not Hope (~50%), Generalized Hope (~30%), Realistic Hope (~12%), Unrealistic Hope (~8%)
- **Splits:** train/val/test stratified 70/15/15%

## 6. Evaluation Protocols and Empirical Performance

| Condition                     | Acc (%) UrduBERT | Acc (%) baseline mBERT | ΔAcc vs. clean |
|-------------------------------|------------------|------------------------|----------------|
| Clean                         | 94.7             | 91.2                   | –              |
| Simulated ASR WER=10%         | 92.1             | 88.5                   | –2.6           |
| Medium noise (SNR=10dB, ~30% WER) | 86.3        | 82.9                   | –8.4           |
| Heavy noise (SNR=0dB, ~50% WER)  | 76.4        | 68.5                   | –18.3          |

Key metrics [2401.09354]:

- **Robustness:** ΔAcc(noise), Robustness Ratio $R = Acc_{noisy} / Acc_{clean}$, WER-ΔAcc correlation
- **Entity Extraction (Slot Filling):** F₁ declines from 93% (clean) to 78% (heavy noise)
- **Latency:** Mean inference time ~45 ms; 95th percentile ~80 ms; stable <50 ms across regimes
- **User Satisfaction:** ≥4.2/5 on Likert scale

Hope-speech task metrics [2512.22705]:

- **Binary classification (UrduBERT + XLM-R backbone):**  
  Accuracy: 95% | Precision: 95% | Recall: 95% | F₁ (macro): 95%
- **Multi-class classification:**  
  RoBERTa-Urdu + GHaLIB: F₁ (macro): 65.2%  
  UrduBERT + classical ensemble: F₁ (macro): ~62.3%

## 7. Limitations, Transferability, and Future Work

UrduBERT demonstrates high robustness and transferability in both ASR-NLU and social-text settings:

- **ASR-NLU**: Accuracy degrades gracefully with noise, only ~8% loss at real-world medium noise levels, outperforming mBERT by 3–8% absolute accuracy [2401.09354].
- **Hope-speech**: Excels in binary hope-speech classification (95% F₁), but multi-class performance is impacted by class imbalance and scarce annotated data [2512.22705].

Limitations include:

- **Data Scarcity:** Urdu corpora and labeled tasks remain limited for fully modeling underrepresented classes and pragmatic contexts.
- **Class Imbalance and Ambiguity:** Weighted losses partially mitigate, but rare hope-speech types (“Unrealistic/Realistic Hope”) remain challenging.
- **Morphological and Code-Mixed Complexity:** Rich inflectional morphology and English–Urdu code-mixing require robust subword modeling.
- **Domain Mismatch:** Pretraining on general text leaves gaps when adapting to specific pragmatic/discourse genres.
- **Comparative Trade-Offs:** RoBERTa-Urdu outperforms pure UrduBERT for multi-class hope speech, at the cost of monolingual model maintenance.

Future directions highlighted include larger and more diverse corpus development, extending to other regional languages, evaluation of adversarial domain adaptation methods, and unified end-to-end ASR+NLU pipelines [2401.09354].

---

UrduBERT establishes a strong foundation for Urdu NLP in both high-noise spoken command domains and low-resource, pragmatic text classification. Its technical design, empirical validation, and nuanced handling of Urdu and Roman-Urdu make it an essential tool for research groups addressing the challenges of under-resourced language technologies [2401.09354], [2512.22705].

Source: https://www.emergentmind.com/topics/urdubert