UrduLM Initiative: Advancing Urdu NLP

• UrduLM Initiative is a coordinated, multi-institutional effort advancing Urdu language modeling with curated corpora, specialized tokenizers, and state-of-the-art evaluation benchmarks.
• It employs robust preprocessing pipelines and parameter-efficient techniques like LoRA to overcome data scarcity and morphological challenges in Urdu.
• The initiative sets new standards by integrating UrduBench, multilingual translation, and downstream applications such as intent detection with high performance metrics.

The UrduLM Initiative is a coordinated, multi-institutional effort to advance foundational language modeling, benchmarking, and downstream application for Urdu—a morphologically rich, under-resourced language spoken by over 230 million people. The initiative comprises open-source corpora, dedicated model architectures (across monolingual, bilingual, and multilingual lines), standardized evaluation harnesses, and practical applications covering NLU, NLG, translation, intent detection, and safety alignment. This entry summarizes the methodologies, datasets, technical advances, and empirical findings that define the state-of-the-art in Urdu-centric language modeling as of early 2026.

1. Rationale and Challenges in Urdu Language Modeling

Urdu exhibits complex agglutinative morphology, diverse registers (formal/literary and colloquial), right-to-left Perso-Arabic script, and pervasive borrowing from Hindi, Punjabi, and Persian. Prior to the Initiative, mainstream multilingual LLMs (e.g., Llama, Qwen, Mistral, mBERT, XLM-R) consistently failed to address the unique demands of Urdu NLP, due to token scarcity in pretraining corpora, subword vocabulary misalignment, and insufficient safety/cultural alignments (Shafique et al., 10 Oct 2025, Hassan et al., 13 Jan 2026, Ali et al., 25 Jan 2026). The typical resulting failure modes included brittle translation, loss of cultural nuance, code-switching, and poor retention of Urdu knowledge during further pretraining (catastrophic forgetting).

Key challenges explicitly targeted by the Initiative include:

• Data scarcity: Insufficiency of large-scale, high-quality, domain-diverse Urdu corpora and benchmarks.
• Morphological and orthographic complexity: Nastaliq script, rich inflection, context-sensitive ligatures.
• Translation lossiness: Inadequate cultural and pragmatic fidelity through machine-translated data.
• Safety and ethical consistency: Content sensitivities and region-specific alignment absent from generic LLM filters.

2. Corpus Construction and Preprocessing Pipelines

Corpus initiatives within UrduLM range from curation of 33 GB monolingual corpora (Ali et al., 25 Jan 2026) to synthetic instruction-response data generation (Shafique et al., 10 Oct 2025). Sources span CC Datasets, OSCAR, Urdu Wikipedia, news and literature, OCR-digitized books, blog/web data, and proprietary translations (e.g., FineWeb-Urdu educational text). Robust preprocessing pipelines involve:

• Script normalization and digit conversion (UrduHack, U+0600–U+06FF regexes).
• Noise and PII filtering: Regexes for emails/numbers, language ID ≥ 0.9 (CLE API).
• Deduplication: MinHash-LSH and SimHash over clean strings.
• Tokenization: Custom BPE vocabularies (10K/20K/32K); a 32K Urdu-specific tokenizer achieves a 20–30% token reduction over GPT-style multilingual tokenizers (Ali et al., 25 Jan 2026).
• Sampling for diversity: Ensuring coverage of news, literature, legal, colloquial, and poetic domains; >90% Urdu purity rate in monolingual datasets (Hassan et al., 13 Jan 2026, Fiaz et al., 24 Feb 2025).

3. Model Architectures, Training, and Adaptation Strategies

Baseline and Monolingual Models

The UrduLM transformer backbone is instantiated at various scales:

Model	Params	Data Volume	Pretraining Strategy
UrduLM-100M	100M	33 GB (∼5–6B tok)	Monolingual GPT-3 style
UrduLLaMA 1.0	8B	128M tok	CLM from Llama-3.1-8B
Alif-1.0-8B	8B	200K UrduWiki art.	Continued Llama-3.1-8B
Qalb	8B	1.97B tok [1.84B Urdu]	Systematic CLM+SFT

Pretraining is performed via standard next-token CLM loss:

$$L_{CLM} = -\sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta)$$

Parameter-efficient adaptation is implemented via LoRA (Low-Rank Adaptation) inserted into self-attention and MLP weight matrices, with a typical rank of 64–128 and negligible additional overhead (Shafique et al., 10 Oct 2025, Hassan et al., 13 Jan 2026, Fiaz et al., 24 Feb 2025).

Synthetic Instruction and Bilingual Models

Instruction tuning uses the “Urdu-Instruct” corpus (51,686 synthetic instruction–response pairs distilled from GPT-4o), augmented with parallel translation and multilingual task data (Shafique et al., 10 Oct 2025). A modified self-instruct technique employs unique prompts for each of seven task categories (generation, ethics, QA, reasoning, translation, classification, sentiment):

• Instruction generation: Seeded GPT-4o, ROUGE-L diversity filtering, grammar correction, and factuality/safety inspection by a team of 20 native reviewers.
• Bilingual translation tasks: Prompts induce both Urdu→English→Urdu and English→Urdu→English flows.
• Ethical alignment: 9,000 regionally curated examples.
• Downstream evaluation set: ~150 manually verified examples per task (Shafique et al., 10 Oct 2025).

Resource-Efficient and Small-Scale Models

The 100M-parameter UrduLM model achieves 70–85% of the performance of models 10–30× larger on core tasks, demonstrating the effect of linguistic coverage, normalization, and tokenization efficiency (Ali et al., 25 Jan 2026).

4. Standardized Evaluation and Benchmarking

UrduBench and Evaluation Protocols

A major contribution is the UrduBench standardized reasoning suite (Shafique et al., 28 Jan 2026), translating MGSM (math word problems), MATH-500 (formal math), CommonSenseQA, and OpenBookQA into Urdu while preserving structural fidelity:

• Contextual Ensemble MT: 4 MT engines (IndicTrans2, NLLB-200, Qwen-3-30B, Gemini-2.5-Pro) produce candidates; GPT-5.1 fuses or selects candidates via weighted similarity aggregation.
• Heuristic Filters: Empty/incomplete/mixed-language outputs are auto-flagged; repeated retrials before human review.
• Human-in-the-Loop: Professional annotators optimize fidelity, fluency, and domain accuracy.

Prompting strategies incorporate both Direct and Chain-of-Thought (CoT) formats, with CoT prompts essential for multi-step arithmetic reasoning. All models are evaluated for both accuracy and language consistency (fraction of outputs in pure Urdu).

Model	MGSM (CoT)	MATH-500 (CoT)	CSQA (Direct)	OBQA (Direct)	LangConsist
Gemma-3-12B-it	59.4% avg.	Declines L1→L5	—	—	≥98%
Alif-1.0-8B-Instr	78.4	—	—	—	High

Task Metrics

Weighted average scores for Urdu benchmarks are defined as:

$$S_{\mathrm{avg}} = \frac{\sum_t w_t S_t}{\sum_t w_t}$$

where $S_t$ is the per-task score and $w_t$ the number of items.

UrduLM (100M) reports 66.6% accuracy in few-shot sentiment, BLEU >30 on grammar correction, with LLM-as-a-Judge averages (Gemini 2.0) of 7.2/10, within the range of multilingual LLMs 10–30× larger (Ali et al., 25 Jan 2026).

Qalb (8B) achieves SOTA on seven Urdu benchmarks, scoring 90.34 (vs. 87.1 for Alif-1.0-8B and 45.7 for Llama-3.1-8B-Instruct), with improvements spanning Generation (+43 pts), QA (+6.6), Reasoning (+5.09), and Translation (+5.11) (Hassan et al., 13 Jan 2026).

5. Specialized Architectures and Mechanistic Interventions

Lossy Translation Effect and Mechanistic Ablation

The MALT (Mechanistic Ablation of Lossy Translation) study (Bajwa, 27 Jan 2025) provides evidence that generic LLMs, when prompted in Urdu, internally reason in an English latent space and only translate to Urdu in the final layers. A single translation-specific direction $d_{\ell,\mathrm{norm}}$ is identified in layer activations, and ablating this direction results in output first decoded in English, then passed through a dedicated high-quality English→Urdu MT model (fine-tuned mBART):

$$z' = z - (z \cdot d_{\ell,\mathrm{norm}}) d_{\ell,\mathrm{norm}}$$

This mechanistic intervention produced BLEU gains (e.g., 46.7 vs. 18.2) and significantly better “cultural nuance” preservation, as judged by annotators:

Method	Human-Correct (%)	BLEU	Nuance (1–5)
Baseline	11.6	18.2	2.1
MALT	55.0	46.7	4.0

This pipeline minimizes translation artifacts, preserves context, and provides a pragmatic, on-the-fly alternative to retraining.

6. Downstream Applications and Open Benchmarks

Intent Detection and Prototype-Informed Pipelines

Contrastive re-training and prototype-attention methods have established SOTA in Urdu few-shot intent detection (Hassan et al., 8 May 2025). The LLMPIA pipeline—comprising self-supervised contrastive learning and advanced attention layers—produces F1-scores up to 98.25% in 4-way 5-shot ATIS tasks and up to 84.42% on Web Queries, outperforming existing baselines by >50 pp in zero-shot settings. Six PLMs and thirteen similarity measures (cosine, angular, KL, Dice, etc.) were benchmarked, with re-training delivering +5–25 pp over off-the-shelf PLMs.

Resource Sharing and Community Engagement

All major corpora, datasets, checkpoints, tokenizers, pretraining scripts, and evaluation harnesses are publicly released via GitHub and HuggingFace. Open participatory processes include community tasks (shared sentiment/Q&A/summarization), hackathons, and active benchmarking beyond MT (Urdu GLUE, poetry, code-switch tasks) (Shafique et al., 10 Oct 2025, Fiaz et al., 24 Feb 2025, Ali et al., 25 Jan 2026).

7. Limitations, Best Practices, and Roadmap

Despite significant advances, several limitations persist:

• Domain coverage: Narrowness in specific subdomains (legal, medical, regional dialects).
• Evaluation gaps: Benchmarks for generation and safety are less standardized relative to translation/reasoning.
• Scalability of annotation: Human-in-the-loop validation increases curation costs.

Best practices synthesized from across studies include:

• Curate diverse, high-purity, deduplicated corpora.
• Apply Urdu-specific tokenization to minimize sequence length and cost.
• Exploit LoRA adaptation for parameter-efficient task transfer.
• Use contextually ensembled and human-verified benchmarks for evaluation (Shafique et al., 28 Jan 2026).
• Quantify and address code-switching/consistency, not only accuracy.

The roadmap includes continued pretraining on larger, more diverse corpora; explicit modeling of regional, dialectal, and colloquial variation; extension of mechanistic and hybrid-generation pipelines; domain-specific SFT; and full reproducibility by open sourcing all assets, including failure modes and risk statements.

• "Alif: Advancing Urdu LLMs via Multilingual Synthetic Data Distillation" (Shafique et al., 10 Oct 2025)
• "UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop" (Shafique et al., 28 Jan 2026)
• "Qalb: Largest State-of-the-Art Urdu LLM for 230M Speakers with Systematic Continued Pre-training" (Hassan et al., 13 Jan 2026)
• "MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu" (Bajwa, 27 Jan 2025)
• "UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings" (Fiaz et al., 24 Feb 2025)
• "UrduLM: A Resource-Efficient Monolingual Urdu LLM" (Ali et al., 25 Jan 2026)
• "Enhanced Urdu Intent Detection with LLMs and Prototype-Informed Predictive Pipelines" (Hassan et al., 8 May 2025)