MedArabiQ: Arabic Medical QA Benchmark

• MedArabiQ is a benchmark comprising seven Arabic medical tasks that blend exam-style questions with natural patient–doctor dialogues.
• It evaluates both closed-set classification and open generation, using metrics like accuracy and BERTScore for comprehensive performance analysis.
• The benchmark also serves as a platform for synthetic data augmentation, cross-lingual evaluation, and bias mitigation in Arabic clinical NLP.

MedArabiQ is an Arabic medical question answering benchmark introduced to evaluate LLMs on multiple Arabic medical tasks spanning multiple choice questions, fill-in-the-blank items, and patient–doctor question answering across multiple specialties (Daoud et al., 6 May 2025). In the AraHealthQA 2025 shared task, the name also designates the general Arabic health QA track covering broader medical domains such as internal medicine, pediatrics, and clinical decision making, alongside a complementary MentalQA track (Alhuzali et al., 27 Aug 2025). Subsequent work uses MedArabiQ not only as a benchmark reference but also as a target setting for Arabic medical text generation, synthetic data augmentation, and cross-lingual evaluation, which suggests a transition from a static benchmark into a broader research locus for Arabic clinical NLP (Allam et al., 12 Sep 2025).

1. Origin and conceptual scope

MedArabiQ was introduced as a benchmark dataset consisting of seven Arabic medical tasks and was motivated by the lack of high-quality domain-specific datasets and benchmarks for the Arabic medical domain (Daoud et al., 6 May 2025). Its stated purpose was to evaluate both knowledge retrieval and patient-style dialogue in Arabic, rather than restricting evaluation to a single task format or specialty. The benchmark therefore combines structured clinical assessment items with more naturalistic patient–doctor exchanges.

The benchmark’s initial design joins two complementary source regimes. One is exam-oriented, derived from paper-based past-exam questions and lecture-note–derived fill-in-the-blank items from regional medical schools. The other is conversational, based on patient–doctor question–answer pairs from AraMed, itself sourced from the Altibbi telemedicine forum (Daoud et al., 6 May 2025). This dual construction is central to MedArabiQ’s identity: it tests both formal medical knowledge and response generation in patient-facing contexts.

In later work, the same name acquires a second institutional meaning. AraHealthQA 2025 defines MedArabiQ as the track for general Arabic medical QA, with explicit attention to realistic clinical settings, multilingual pressures, and culturally nuanced healthcare contexts (Alhuzali et al., 27 Aug 2025). A plausible implication is that the literature now uses “MedArabiQ” at two levels: first, as a concrete benchmark release, and second, as a task family for Arabic medical QA evaluation.

2. Original benchmark architecture

The original MedArabiQ benchmark was built from 100 paper-based past-exam questions and their answer keys, 200 lecture-note–derived fill-in-the-blank items from regional medical schools, and 100 patient–doctor question–answer pairs selected from AraMed across fourteen specialties: cardiology, obstetrics & gynecology, surgery, pediatrics, neurology, oncology, endocrinology, dentistry, otolaryngology, public health, dermatology, primary care, pulmonology, and psychology (Daoud et al., 6 May 2025). The patient–doctor subset was further expanded into three parallel versions: the raw patient–doctor QA set in Modern Standard Arabic, a grammatical error–corrected version produced via an Arabic-tailored CAMeL-Tools→mBART pipeline, and an LLM-paraphrased version produced using GPT-4o.

A separate multiple-choice subset was modified by injecting seven categories of cognitive bias—confirmation, recency, frequency, cultural, false-consensus, status-quo, and self-diagnosis—to test bias susceptibility and bias mitigation (Daoud et al., 6 May 2025). This makes MedArabiQ unusual among Arabic medical benchmarks in that it does not only measure answer selection and text generation; it also measures robustness to bias-laden prompt formulations.

The benchmark comprises seven tasks of 100 instances each, for a total of 700 samples.

Task	Format	Instances
Standard Multiple-Choice Questions	Closed-set classification	100
MCQ + Bias	Closed-set classification	100
Fill-in-the-Blank with Choices	Closed-set classification	100
Fill-in-the-Blank without Choices	Open generation	100
Patient–Doctor QA	Open generation	100
Patient–Doctor QA + GEC	Open generation	100
Patient–Doctor QA + LLM Paraphrase	Open generation	100

This architecture operationalizes several capabilities in parallel: factual recall, option discrimination, robustness to adversarial or biased framing, tolerance to grammatical noise, and semantic fidelity under paraphrase. The construction also encodes an explicit distinction between closed-set answer selection and open-ended medical response generation.

3. MedArabiQ in AraHealthQA 2025

In AraHealthQA 2025, the MedArabiQ track was reconfigured into two subtasks operating entirely in Arabic and designed to reflect realistic clinical settings (Alhuzali et al., 27 Aug 2025). Questions were drawn from two primary sources: past regional Arabic medical-school and board-exam question banks covering specialties such as internal medicine, cardiology, pediatrics, obstetrics & gynecology, and clinical decision making, and AraMed patient–clinician dialogues from which realistic patient–doctor inquiries were sampled. To construct fill-in-the-blank variants, 50 of the multiple-choice items were systematically masked in the stem or in one of the answer options.

The development data consists of 700 question–answer pairs, with 300 for subtask 1 and 400 for subtask 2, while the blind test set comprises 200 items, 100 per subtask, with no overlap between development and test (Alhuzali et al., 27 Aug 2025). Subtask 1 is classification-oriented and consists of 50% pure multiple-choice items and 50% fill-in-the-blank items with candidate options. Subtask 2 is generative and consists of 50% fill-in-the-blank items without options and 50% patient–doctor free-text queries.

The annotation protocol is unusually explicit. All questions and reference answers were reviewed and curated by a panel of three board-certified Arabic-speaking physicians—one internist, one pediatrician, and one family medicine specialist—under the supervision of the MedArabiQ consortium. Annotators followed written guidelines specifying clinical correctness, style consistency, and dialectal neutrality. Quality control used a double-blind process in which each item was independently annotated by two physicians and conflicts were adjudicated by the third expert. Inter-annotator agreement on classification of distractor plausibility and answer correctness for multiple-choice items achieved Cohen’s $\kappa = 0.82$, and semantic similarity agreement for open-ended answers, measured in a small pilot with BERTScore, averaged $0.88$ (Alhuzali et al., 27 Aug 2025).

The development set’s specialty breakdown is internal medicine 30%, pediatrics 20%, obstetrics & gynecology 15%, cardiology 10%, and others—neurology, general surgery, and family medicine—25%. Question types are factoid/fill-in 45%, reasoning/explanation 25%, yes/no 10%, list-type 5%, and short-answer dialogue 15% (Alhuzali et al., 27 Aug 2025). All prompts and options are in Arabic, either Modern Standard Arabic or widely understood clinical colloquial terms. Distractors were intentionally drawn from regional exam questions reflecting local practice variations, such as choice of first-line antibiotic in certain countries, and patient–doctor dialogues preserve polite forms of address and culturally appropriate phrasing such as “حضرتك”.

4. Task formalization and evaluation methodology

MedArabiQ uses different evaluation schemes for closed-set and open-ended tasks. In the original benchmark, tasks 1–3 and the bias-augmented MCQ task are cast as closed-set classification problems, while fill-in-the-blank without choices and patient–doctor QA variants are treated as open generation (Daoud et al., 6 May 2025). In the shared-task formulation, the same distinction becomes Subtask 1 versus Subtask 2 (Alhuzali et al., 27 Aug 2025).

For classification, the shared task defines accuracy as

$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$

where $N$ is the number of questions, $\hat{y}_i$ is the predicted choice, and $y_i$ is the gold-standard choice (Alhuzali et al., 27 Aug 2025). Subtask 1 uses a UTF-8–encoded CSV input with “question text” and a “semicoloned list of candidate options,” and requires a plain-text output containing only the Arabic letter corresponding to the selected answer.

For generation, MedArabiQ relies on BERTScore rather than exact match, reflecting the fact that semantically correct Arabic medical answers may differ lexically from references. The shared task computes precision $P$ and recall $R$ over token embeddings from a pre-trained BERT model and then uses

$$\text{BERTScore} = \frac{2 \times P \times R}{P + R}\,.$$

Subtask 2 uses JSONL input with “id,” “question,” and optionally “blank_position,” and requires JSONL output with “id” and “generated_answer” in Modern Standard Arabic (Alhuzali et al., 27 Aug 2025).

The original benchmark evaluated five instruction-tuned LLMs in zero-shot mode: LLaMA 3.1 70B, Qwen 2.5 72B, GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5 Pro. Prompts were written in English for MCQs and fill-in tasks and in Arabic for patient–doctor QA, with no fine-tuning on MedArabiQ. For BERTScore, XLM-RoBERTa-Large was used as the evaluator (Daoud et al., 6 May 2025).

5. Baseline behavior and shared-task performance

The original MedArabiQ benchmark established a heterogeneous performance profile across tasks rather than a single dominant model (Daoud et al., 6 May 2025). On structured closed-set tasks, proprietary models performed best, while open-source models approached proprietary performance more closely on generative patient–doctor QA when semantic evaluation was used. Bias injection caused average accuracy drops of 10–20 points, and one-shot and few-shot mitigations recovered up to 70–85% of lost accuracy, whereas simple bias-education prompts often underperformed. No single model excelled uniformly across all fourteen specialties or all seven bias categories.

Benchmark task	Best reported model	Best score
MCQ	Gemini 1.5 Pro	57.5%
Fill-in-the-Blank with Choices	Gemini 1.5 Pro / Claude 3.5-Sonnet	72.7%
Fill-in-the-Blank without Choices	Gemini 1.5 Pro	88.8
Patient–Doctor QA	Qwen 2.5	85.2
QA + GEC	LLaMA 3.1 70B	85.5
QA + LLM Paraphrase	LLaMA 3.1 70B / Qwen 2.5	84.9

In the AraHealthQA 2025 MedArabiQ track, baseline systems for Subtask 1 were Gemini 1.5 Pro and DeepSeek v3, both zero-shot with Arabic prompts, and both scored accuracy $0.47$ on the blind test. The top systems substantially exceeded this level: NYUAD achieved $0.88$0, !MSA $0.88$1, and MedLingua $0.88$2 on Subtask 1. For Subtask 2, the Gemini 1.5 Pro zero-shot baseline obtained BERTScore $0.88$3, while MedGapGab reached $0.88$4, !MSA $0.88$5, NYUAD $0.88$6, Egyhealth $0.88$7, and MedLingua $0.88$8 (Alhuzali et al., 27 Aug 2025).

The shared-task organizers identified several salient trends. Structured classification saw substantial gains over baselines, with prompt engineering and ensemble voting driving more than 60% relative error reduction. Open-ended generation exhibited high absolute BERTScore, above $0.88$9, but very narrow score dispersion from $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$0 to $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$1, indicating saturation of surface similarity metrics. Teams that combined retrieval-augmented prompts, including MedGapGab’s TF-IDF-based example retrieval, with model-agnostic pipelines outperformed pure generative approaches, especially on fill-in-the-blank items (Alhuzali et al., 27 Aug 2025).

A detailed system report from !MSA provides a concrete illustration of this design space (Tarek et al., 14 Sep 2025). For Subtask 1, !MSA used Gemini 2.5 Flash with three prompt configurations—Arabic Few-Shot, English Translation + Answer, and Refinement + Answer—and combined them with a uniform-weight ensemble

$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$2

with $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$3 and tie-break priority RFA $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$4 AFS $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$5 ETA. This yielded 76.0% accuracy on the 100-item test set, compared with 74.0 for RFA alone, 71.0 for AFS, and 69.0 for ETA. For Subtask 2, the same group used a unified Arabic prompt with four few-shot demonstrations and achieved BERTScore $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$6. Their ablations showed that removing markdown stripping and Arabic normalization reduced BERTScore by approximately 2–3 points, and that biased questions were more frequently misclassified by single-prompt systems than by the ensemble.

6. Fine-tuning, generation pipelines, and synthetic augmentation

Parallel research has treated MedArabiQ as a target scenario for Arabic medical text generation rather than only as a benchmark (Allam et al., 12 Sep 2025). One line of work collected over 40,000 posts and replies from public Arabic-language medical discussions scraped from Facebook groups and similar platforms using Selenium and BeautifulSoup, then reduced this to about 20,000 high-quality question–answer pairs after cleaning and deduplication. The preprocessing pipeline stripped advertisements, emojis, incomplete posts, and non-medical chatter; unified spellings; removed diacritics; and normalized Egyptian, Levantine, and Gulf Arabic variants toward Modern Standard Arabic where possible. Fine-tuning with Hugging Face Transformers, Trainer API, and LoRA was applied to Mistral-7B-Instruct-v0.2, LLaMA-2-7B, and GPT-2 Medium under cross-entropy loss with AdamW, cosine warm-up scheduling, FP16, batch size 8, and 3 epochs. Mistral-7B-Instruct-v0.2 obtained average BERTScore precision 68.50, recall 69.08, and F1 68.50, ahead of LLaMA-2-7B and GPT-2 Medium (Allam et al., 12 Sep 2025).

A subsequent synthetic augmentation study explicitly framed its contribution as extending an Arabic medical QA system such as MedArabiQ (Allam et al., 12 Sep 2025). Starting from 20,000 real patient–doctor pairs, it generated 80,000 synthetic pairs using ChatGPT-4o and Gemini 2.5 Pro, with approximately 20 prompt templates covering complaint categories such as respiratory, gastrointestinal, chronic conditions, and mental health. Synthetic data were filtered with an Arabic BERT encoder by discarding items whose cosine similarity to the nearest five real examples was below 0.6 or above 0.95, checked with language-ID rules, and manually reviewed in a sample of 500 pairs scored for fluency, coherence, and medical plausibility by three native Arabic speakers with basic medical training.

The final merged corpus contained 100,000 records. Five models—AraGPT2-Base, Meta-LLaMA-2-7B, Mistral-7B-Instruct-v0.2, BLOOM-560M, and GPT-2 Medium—were fine-tuned with Trainer and PEFT/LoRA. Synthetic augmentation improved BERTScore F1 by +6–13 percentage points over 20,000 real-only fine-tuning, and Mistral-7B-Instruct achieved the top overall F1 of 81.36% on the 100K corpus (Allam et al., 12 Sep 2025). An ablation comparing synthetic sources found that ChatGPT-4o data consistently outperformed Gemini-generated data: for example, Meta-LLaMA-2-7B scored 68.65% versus 65.19%, AraGPT2-Base 59.67% versus 56.74%, and Mistral-7B 69.30% versus 67.22%, with 25–30% fewer hallucinations on average and paired-$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$7 significance at $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$8 across all five models.

These studies do not redefine the benchmark itself, but they expand the operational context in which MedArabiQ is used. They also foreground engineering components that the benchmark papers identify as consequential: dialect normalization, retrieval augmentation, post-processing, and human validation.

7. Cross-lingual constraints, reliability, and future directions

Later empirical work showed that Arabic medical QA remains constrained by language-specific modeling issues even when large models are strong in English (Abouzahir et al., 5 Feb 2026). A cross-lingual study on an Arabic medical QA dataset labeled “MedAraBench” / “MedArabiQ” used 24,883 items, with 19,894 train and 4,989 test, and compared Arabic MCQs against automatic English translations under matched prompting and decoding conditions. English accuracy exceeded Arabic by 3–20 percentage points for nearly all models: DeepSeek-V3.2 achieved near-parity at 62.39% Arabic versus 62.85% English, but Med42-70B showed a gap of 19.62 percentage points, and LLaMA 3.3 70B a gap of 15.51.

The same study attributes part of this gap to tokenization fragmentation. Arabic inputs were split into approximately 2.4 subwords per word under most tokenizers, with lower characters-per-token and high single-character token rates. A reported average for model-native tokenizers over DeepSeek, LLaMA, Meditron, and Med42 was 2.42 tokens per word, 2.29 characters per token, and 32% single-character tokens, while an Arabic-focused tokenizer, CAMeLBERT-MSA, reduced fragmentation to 1.76 tokens per word (Abouzahir et al., 5 Feb 2026). The study also found negative Pearson correlations between self-reported confidence and correctness, $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)\,,$$9 for Arabic and $N$0 for English, and observed frequent reasoning–label misalignment under explanation prompting.

These findings reinforce limitations already documented in MedArabiQ-focused work. The AraHealthQA 2025 report notes that limited dataset scale and narrow dialect coverage hinder generalization, especially for colloquial patient queries; that automatic metrics such as BERTScore do not capture answer safety, cultural appropriateness, or clinical trustworthiness; and that distractor bias reflecting regional misconceptions can mislead overly literal LLMs without domain grounding (Alhuzali et al., 27 Aug 2025). The original benchmark likewise concludes that culturally informed bias mitigation remains incomplete and that semantic metrics are necessary but insufficient for comprehensive medical evaluation (Daoud et al., 6 May 2025).

The principal future directions are correspondingly concrete. Recommended extensions include multi-dialectal data covering Gulf, Levantine, and Maghrebi Arabic; human-in-the-loop evaluation by Arabic-speaking clinicians; richer annotations such as rationales for correct options and evidence citations; additional subtasks such as medical summary generation and differential-diagnosis reasoning; multi-turn context windows; retrieval or fact-checking modules linked to trusted medical databases; and language-aware design choices such as Arabic-focused tokenizers and reduced instruction mismatch through fully Arabic prompting (Alhuzali et al., 27 Aug 2025). Taken together, these proposals define MedArabiQ’s next phase less as a single dataset release than as a progressively richer evaluation framework for safe, equitable, and culturally grounded Arabic medical QA.