BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition

Abstract: In this article, we present a gold-standard benchmark dataset for Biomedical Urdu Named Entity Recognition (BioUNER), developed by crawling health-related articles from online Urdu news portals, medical prescriptions, and hospital health blogs and websites. After preprocessing, three native annotators with familiarity in the medical domain participated in the annotation process using the Doccano text annotation tool and annotated 153K tokens. Following annotation, the proposed BioiUNER dataset was evaluated both intrinsically and extrinsically. An inter-annotator agreement score of 0.78 was achieved, thereby validating the dataset as gold-standard quality. To demonstrate the utility and benchmarking capability of the dataset, we evaluated several machine learning and deep learning models, including Support Vector Machines (SVM), Long Short-Term Memory networks (LSTM), Multilingual BERT (mBERT), and XLM-RoBERTa. The gold-standard BioUNER dataset serves as a reliable benchmark and a valuable addition to Urdu language processing resources.

• The paper introduces BioUNER, the first gold-standard dataset for clinical Urdu NER that supports domain-specific biomedical text mining.
• It employs a rigorous annotation protocol with a BIES labeling scheme and achieves strong inter-annotator agreement (κ=0.78) alongside high LSTM performance (F1=0.95).
• The study highlights that, in low-resource scenarios for morphologically rich languages like Urdu, recurrent models can outperform transformer-based approaches.

BioUNER: Establishing a Benchmark for Clinical Urdu Named Entity Recognition

Context and Motivation

The research introduces BioUNER, the first publicly available gold-standard dataset for clinical named entity recognition (NER) in Urdu (2604.02904). Urdu, despite its status as a major South Asian language and its use in clinical contexts, has lacked targeted biomedical NER resources, limiting progress in specialized information extraction and downstream clinical NLP tasks. Existing datasets for Urdu focus solely on the general domain, thus failing to provide the granularity or domain-specific annotation needed to parse the linguistic and semantic complexity present in biomedical narratives.

Dataset Development and Annotation Protocol

BioUNER was curated by aggregating clinical text from Urdu health blogs, medical articles, hospital records, and patient guides sourced from institutions such as Continental Hospitals, Aga Khan University Hospital, and several online health portals. Strict preprocessing pipelines eliminated HTML artifacts, standardized script variants, and segmented sentences for annotation suitability. Three native speakers with domain expertise carried out manual annotation using the Doccano tool under a set of expert guidelines, targeting six biomedical entity classes: Disease, Drug, Chemical, Protein, Gene, and CellLine.

Inter-annotator agreement achieved a substantial $\kappa = 0.78$, indicating robust consistency and reliability in the annotated output. A BIES (Begin-Inside-End-Single) sequence labeling scheme was used, enhancing precision in boundary marking for multi-token and nested entities.

Figure 1: Example of clinical Urdu text annotated in Doccano, demonstrating the delineation of entity boundaries and entity class assignments under the BIES scheme.

BioUNER comprises 153K tokens and over 15,000 entity mentions, with the Disease class being the most prevalent, reflecting a corpus tailored to clinically relevant concept extraction and aligning with usage in diagnostic and therapeutic contexts.

Experimental Evaluation and Benchmarks

Multiple modeling paradigms were evaluated for intrinsic benchmarking of the dataset:

• Support Vector Machine (SVM): Utilized token-level TF-IDF features from character n-grams (n=2-5), capturing morphological variance typical in Urdu. Despite providing a non-contextual baseline (F1=0.65), its lack of sequential modeling capability restricted entity boundary and ambiguity resolution.
• Conditional Random Fields (CRF): Modeled transition features explicit to sequence labeling, yielding strong consistency in boundary predictions and outperforming SVM (F1=0.93).
• Long Short-Term Memory (LSTM): Delivered the highest performance (F1=0.95), benefiting from context-sensitive embeddings and long-range dependency modeling.
• Transformer-based Models (mBERT and XLM-RoBERTa): Both transformers leveraged multilingual pretraining to address the scarcity of Urdu biomedical corpora. XLM-RoBERTa slightly outperformed mBERT (F1=0.73 vs. F1=0.72), with improved recall for multi-span entities. Notably, transformer models did not surpass the LSTM under current data regime.

Summary of results:

Model	Precision	Recall	F1-score
SVM	0.65	0.66	0.65
CRF	0.93	0.93	0.93
LSTM	0.95	0.95	0.95
mBERT	0.72	0.71	0.72
XLM-RoBERTa	0.72	0.73	0.73

The high numerical performance of the LSTM baseline contrary to transformer-based models is notable, especially given the prevailing trend for transformers to dominate multilingual biomedical NLP settings. This suggests that, under data-limited scenarios and with morphologically rich languages such as Urdu, recurrent architectures may retain unique advantages.

Practical and Theoretical Implications

BioUNER fills a crucial gap in Urdu language technological resources, enabling systematic benchmarking of NER systems in low-resource, domain-specific settings. The substantial agreement in annotation and diversity in entity types ensures that BioUNER can be effectively used for model pretraining, benchmarking, and domain adaptation experiments.

Practically, this resource can drive improved clinical text mining and information retrieval systems for Urdu-speaking populations, important for both electronic health record analytics and accessible health information systems. The public release of the dataset and pretrained models on Hugging Face will catalyze community adoption and comparative benchmarking.

Theoretically, the observed performance gap between LSTM and transformer models highlights open questions regarding inductive biases and sample efficiency for morphologically complex, low-resource languages. This may stimulate further investigation into hybrid architectures or systematic pretraining approaches for biomedical NER in underrepresented languages.

Future Directions

Potential research trajectories include:

• Expansion of BioUNER to broader subdomains (e.g., radiology, pathology), increasing coverage and sample diversity.
• Exploration of more advanced transformer architectures with domain-adaptive pretraining or annotation-efficient fine-tuning.
• Joint entity and relation extraction, capitalizing on nested and cross-sentence entity phenomena prevalent in clinical narratives.
• Cross-lingual transfer learning leveraging high-resource biomedical corpora in typologically similar languages to improve generalization in Urdu NER.

Conclusion

BioUNER constitutes a foundational resource for Urdu clinical NER, providing a robust, gold-standard dataset that enables comprehensive benchmarking and facilitating the development of high-precision clinical information extraction systems for a previously underserved language. Its release lays the groundwork for both methodological advancements in multilingual biomedical NLP and practical deployments enhancing clinical text analytics for Urdu-speaking clinical and patient communities.