Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training

Abstract: Despite remarkable progress in LLMs, Urdu-a language spoken by over 230 million people-remains critically underrepresented in modern NLP systems. Existing multilingual models demonstrate poor performance on Urdu-specific tasks, struggling with the language's complex morphology, right-to-left Nastaliq script, and rich literary traditions. Even the base LLaMA-3.1 8B-Instruct model shows limited capability in generating fluent, contextually appropriate Urdu text. We introduce Qalb, an Urdu LLM developed through a two-stage approach: continued pre-training followed by supervised fine-tuning. Starting from LLaMA 3.1 8B, we perform continued pre-training on a dataset of 1.97 billion tokens. This corpus comprises 1.84 billion tokens of diverse Urdu text-spanning news archives, classical and contemporary literature, government documents, and social media-combined with 140 million tokens of English Wikipedia data to prevent catastrophic forgetting. We then fine-tune the resulting model on the Alif Urdu-instruct dataset. Through extensive evaluation on Urdu-specific benchmarks, Qalb demonstrates substantial improvements, achieving a weighted average score of 90.34 and outperforming the previous state-of-the-art Alif-1.0-Instruct model (87.1) by 3.24 points, while also surpassing the base LLaMA-3.1 8B-Instruct model by 44.64 points. Qalb achieves state-of-the-art performance with comprehensive evaluation across seven diverse tasks including Classification, Sentiment Analysis, and Reasoning. Our results demonstrate that continued pre-training on diverse, high-quality language data, combined with targeted instruction fine-tuning, effectively adapts foundation models to low-resource languages.

• The paper presents a novel 8B-parameter Urdu LLM, Qalb, that leverages systematic continued pre-training on an extensive Urdu corpus combined with targeted instruction fine-tuning.
• The paper demonstrates significant performance gains across tasks such as translation, reasoning, and question answering, outperforming previous state-of-the-art models.
• The paper shows that techniques like LoRA and quantization enable efficient training and scalable deployment, even in resource-constrained environments.

Qalb: State-of-the-Art Urdu LLM via Systematic Continued Pre-Training

Introduction

Qalb addresses the pronounced underrepresentation of Urdu—a language spoken by over 230 million individuals—in modern NLP systems. Despite the proliferation of high-performing LLMs in English and other major languages, Urdu NLP lags due to limited token coverage in foundational models. Qalb is a systematically developed 8B-parameter LLM for Urdu, building on Meta's LLaMA-3.1 8B by leveraging a two-phase approach: continued pre-training on a domain-diverse Urdu corpus and targeted instruction fine-tuning. The methodology demonstrates that extensive Urdu data exposure during pre-training is indispensable for downstream task competence, producing superior results to mere instruction fine-tuning on a small amount of Urdu data.

Figure 1: The full pipeline for Qalb, from large-scale data curation through continued pre-training to instruction fine-tuning and evaluation.

Dataset Construction and Pre-Training Strategy

The authors curated one of the largest Urdu datasets to date—1.97 billion tokens across 5.04 million documents—comprising 1.84 billion Urdu tokens sourced from domains such as news, literature, religion, government, and social media. To mitigate catastrophic forgetting of foundational English knowledge, 140 million tokens of English Wikipedia data were injected as a replay buffer, following best practices for language adaptation in multilingual LLMs.

Data cleaning employed an advanced multi-stage pipeline using crawl4ai for asynchronous scraping and Playwright for dynamic content rendering. Key steps included header/footer stripping, short document filtering, deduplication with hash-based detection, and junk metadata removal, yielding a 67.8% retention rate of high-quality semantic content.

Continued pre-training was enacted using Low-Rank Adaptation (LoRA) on approximately 14.72% of model parameters (∼1.18B), following resource-efficient protocols via the Unsloth library. AdamW-8bit optimizer and bfloat16 precision were leveraged on a single A100 80GB GPU, employing a cosine-decayed learning rate and batch sizes sufficient for gradient stability.

Training loss and validation perplexity monitored over 7,500 steps revealed rapid convergence and substantial Urdu text modeling ability early in the training (initial perplexity 2.35 at 2,500 steps, falling to ≈2.20 at completion).

Figure 2: Loss and perplexity curves during continued pre-training, indicating efficient Urdu language acquisition and model improvement over 7,500 steps.

Instruction Fine-Tuning and Prompt Engineering

Post pre-training, supervised instruction fine-tuning was performed on the Alif Urdu-instruct dataset, utilizing the Llama-3 chat template with explicit control token demarcation for System, User, and Assistant roles. The prompt structure directs focus solely on response generation, masking user input loss.

LoRA was maintained for fine-tuning, with a rank of 128, learning rate of 5e-5, two epochs, and batch size 64. The AdamW-8bit optimizer facilitated memory-efficient weight updates, and bfloat16 enabled greater batch sizes per device.

Evaluation and Benchmarking

Qalb was evaluated against seven diverse Urdu tasks: generation, translation, ethics, reasoning, classification, sentiment analysis, and question answering. Model comparisons centered on LLaMA-3.1 8B-Instruct (baseline), Alif-1.0-Instruct (previous SOTA), and other recent multilingual/Urdu models (Gemma, Aya, Mistral, Qwen).

Evaluation adopted the LLM-as-a-judge approach, using GPT-4o to score outputs for relevance, correctness, clarity, and formatting on a ten-point scale, as per Alif's protocol. This automatic judge achieved human evaluator agreement rates exceeding 85%, supporting its reliability for comparative benchmarking.

Qalb achieved a weighted average benchmark score of 90.34, outperforming Alif-1.0-Instruct (87.1) by 3.24 points, and producing a massive 44.64 point gain over LLaMA-3.1 8B-Instruct (45.7). Qalb led across six of seven tasks, with prominent improvements in QA (+6.6), translation (+5.11), and reasoning (+5.09). The model retained robust performance in classification and sentiment analysis, with marginally lower scores than Alif in generation, possibly due to prompt-specific or stylistic evaluation biases.

Notably, a 4-bit quantized Qalb version maintained 95% of its full-precision performance (86.02), facilitating deployment in memory-constrained environments.

Qualitative analysis highlighted Qalb's compact, instruction-adherent responses and explicit reasoning chains, contrasted with Alif's verbosity and propensity to hallucinate instructions in open-ended prompts. Minor issues, such as hallucinated signatures, were observed, generally traceable to residuals in forum-derived pre-training data.

Figure 3: Task-wise qualitative comparison of Qalb and Alif; Qalb demonstrates greater fidelity to instructions and concise output forms.

Discussion and Implications

Empirical results establish the primacy of extensive continued pre-training on target language data for low-resource LLM adaptation. Even with similar parameter counts and instruction tuning data, Qalb's deep Urdu exposure during pre-training led to SOTA performance, outstripping both general-purpose and previous customized Urdu models.

Trade-offs emerge in generation, where Alif marginally scores higher, but qualitative assessment exposes usability issues, such as instruction hallucination. Qalb's superior task coverage and performance across more domains assert its robustness and generality as an Urdu NLP foundation.

The integration of quantization (QLoRA) ensures scalability across deployment systems, permitting utilization in edge or mobile scenarios—a crucial consideration for accessibility in the 230M Urdu-speaking population.

Risks remain regarding model hallucination and cultural bias, particularly given the diversity of the pre-training corpus. Mitigation strategies will require active domain-specific finetuning and continuous evaluation, especially in sensitive applications.

Future Prospects

Qalb provides a replicable blueprint for low-resource language modeling, applicable to languages sharing scripts and characteristics with Urdu (e.g., Pashto, Sindhi, Punjabi). Immediate research avenues include domain-specific adaptation (medical, legal, educational), regional dialect modeling, and improved multilingual code-switching support. Public release of datasets and trained models will catalyze further innovation and democratization of NLP for underrepresented languages.

Conclusion

Qalb constitutes a significant advancement in Urdu LLMs, substantiating the necessity of systematic, large-scale continued pre-training for low-resource language adaptation. Its outperforming benchmark scores, extensive domain coverage, and quantization readiness form a versatile platform for Urdu NLP. The methodological clarity and reproducibility of the Qalb approach will inform future foundational model development for other linguistically marginalized communities.