AraHealthQA 2025: Arabic Health QA Benchmark

• AraHealthQA 2025 is a comprehensive benchmark that fills the gap in high-quality Arabic medical QA by combining classification and generative tasks across mental and general medicine.
• The two tracks—MentalQA for mental health inquiries and MedArabiQ for broader clinical topics—enable evaluation of culturally nuanced and linguistically diverse healthcare queries.
• The benchmark uses robust evaluation metrics and baseline models to assess performance in complex tasks, emphasizing the need for clinically and culturally sensitive QA systems.

AraHealthQA 2025 is the “Comprehensive Arabic Health Question Answering Shared Task,” held in conjunction with ArabicNLP 2025 and co-located with EMNLP 2025. It was introduced to address the paucity of high-quality Arabic medical QA resources through two complementary tracks: MentalQA, centered on Arabic mental health question answering, and MedArabiQ, covering broader medical domains such as internal medicine, pediatrics, and clinical decision making. The task was structured around realistic, multilingual, and culturally nuanced healthcare contexts, and the organizers described it as the first comprehensive Arabic health QA benchmark spanning mental health and general medicine (Alhuzali et al., 27 Aug 2025).

1. Scope, rationale, and benchmark position

AraHealthQA 2025 was designed around a specific resource gap: Arabic medical QA lacked shared benchmarks that jointly covered classification and generation, specialist medical content and patient-facing consultation, as well as culturally sensitive health domains. Its two-track design therefore separates mental health from broader clinical medicine while still maintaining a unified benchmarking framework (Alhuzali et al., 27 Aug 2025).

The benchmark rationale also sits within a larger trajectory of Arabic medical resource building. MedAraBench, a large-scale Arabic medical multiple-choice benchmark, was explicitly presented as informing the planning of AraHealthQA 2025. That report recommended using MedAraBench as a standardized MCQ baseline and extending evaluation toward open-ended questions with rationale grading, image-based items, multi-turn patient-clinician dialogues, dialectal Arabic, mixed-language corpora, and leaderboard infrastructure (Abu-Daoud et al., 2 Feb 2026). This suggests that AraHealthQA 2025 occupies an intermediate position between classical medical exam QA and clinically situated Arabic health assistance.

2. Track architecture and domain coverage

AraHealthQA 2025 is organized into two tracks with distinct domain emphases and output regimes (Alhuzali et al., 27 Aug 2025).

Track	Domain focus	Subtasks
MentalQA	Anxiety, depression, cognitive disorders, stigma, therapeutic practice	Question Type Classification; Answer Strategy Classification; Generative Q&A
MedArabiQ	Internal medicine, pediatrics, cardiology, clinical decision-making, patient education	Multiple-Choice Question Answering; Open-ended / Fill-in-the-blank Generation

MentalQA includes both “factual” questions, such as “What causes insomnia?”, and “supportive” questions, such as “How can I cope with my anxiety?”. Its subtasks reflect this duality. Subtask 1 is an 8-way single-label question type classification problem. Subtask 2 is a 3-way multi-label answer strategy classification problem. Subtask 3 requires generation of a concise, empathetic, clinically sound answer (Alhuzali et al., 27 Aug 2025).

MedArabiQ addresses broader medical QA through two formats. Subtask 1 is structured classification over multiple-choice questions. Subtask 2 is free-text generation for open-ended and fill-in-the-blank settings. The question formats span structured MCQs, fill-in-the-blank items with and without candidate lists, and open-ended patient–doctor Q&A (Alhuzali et al., 27 Aug 2025). A Track 2 system report further characterized MedArabiQ as covering twelve domains, with examples including Physiology, Neurosurgery, and Pharmacology, and highlighted a mix of Modern Standard Arabic and dialectal Arabic in clinically oriented prompts (Tarek et al., 14 Sep 2025).

3. Dataset construction and annotation protocol

The MentalQA track is built from the MentalQA corpus of Alhuzali et al. 2024 and 2025, consisting of real patient inquiries in Modern Standard Arabic paired with expert-written doctor responses. Its annotation protocol includes question-type labels drawn from eight categories—Diagnosis, Treatment, Anatomy/Physiology, Epidemiology, Healthy Lifestyle, Provider Choice, and Stigma/Other—and answer-strategy labels drawn from the multi-label set {Information Provision, Direct Guidance, Emotional Support}. Gold labels were agreed upon by two clinical annotators with $\kappa > 0.75$, followed by adjudication by a third annotator (Alhuzali et al., 27 Aug 2025).

The MentalQA splits are identical across all three subtasks: Train = 300 QA pairs, Dev = 50 QA pairs, Test = 150 QA pairs, for a Total = 500 pairs. This uniform split simplifies cross-subtask comparison while keeping the dataset modest in size (Alhuzali et al., 27 Aug 2025).

MedArabiQ draws on two sources: digitized multiple-choice questions from past regional Arabic medical exams, and patient–doctor dialogues and textbook-style answers from the AraMed corpus. Reference answers or answer keys were provided by domain experts or textbooks. Its development material totals 700 questions, divided into 300 questions for Subtask 1 and 400 for Subtask 2. The test set totals 200 questions, split evenly across the two subtasks. For Subtask 1, the 100 test questions comprise 50 MCQs and 50 fill-in-the-blank items with choices; for Subtask 2, the 100 test questions comprise 50 fill-in-the-blank items without choices and 50 patient–doctor items (Alhuzali et al., 27 Aug 2025).

A broader contextual comparison is instructive. MedAraBench comprises 24,883 manually digitized Arabic MCQs across 19 medical specialties, derived from scanned paper-based exams and filtered from an initial pool of 34,333 items. It preserves real-world variation in Arabic terminology and was benchmarked as a large-scale MCQ resource for Arabic medicine (Abu-Daoud et al., 2 Feb 2026). A plausible implication is that AraHealthQA 2025 narrows the scale of MCQ evaluation in exchange for task heterogeneity, especially generation and patient-facing consultation.

4. Evaluation framework and official baselines

AraHealthQA 2025 evaluates classification and generation with standardized automatic metrics. For class $i$, precision and recall are defined as

$$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$

and the classwise $F_1$ score is

$$F_{1,i} = 2 \times \frac{\mathrm{Precision}_i \times \mathrm{Recall}_i}{\mathrm{Precision}_i+\mathrm{Recall}_i}.$$

Weighted $F_1$ is

$$F_{1}^{\mathrm{(weighted)}} = \sum_i w_i\,F_{1,i}, \qquad w_i = \frac{N_i}{\sum_j N_j},$$

the Jaccard Score for multi-label overlap is

$J = \frac{|\hat Y\cap Y|}{|\hat Y\cup Y|},$

and classification accuracy is

$$\mathrm{Accuracy} = \frac{\#\{\textrm{correct}\}}{\#\textrm{instances}}.$$

Generation was scored with BERTScore, described in the task paper as a pre-trained transformer-based token alignment $F_1$ metric, with Zhang et al. 2020 cited for details (Alhuzali et al., 27 Aug 2025).

The split protocol was fixed by track. Track 1 uses 300/50/150 train/dev/test splits for all three subtasks. Track 2 uses 300/–/100 for Subtask 1 and 400/–/100 for Subtask 2. No formal $i$0-value testing was reported in the description paper; leaderboard rankings relied on held-out test sets and automatic metrics (Alhuzali et al., 27 Aug 2025).

The official baselines reflect the asymmetry between tracks. In Track 1 Subtasks 1 and 2, the baseline is a Most-Frequent label predictor. For Subtask 1, this yields Weighted-F$i$1 = 0.24 and Jaccard = 0.40; for Subtask 2, Weighted-F$i$2 = 0.44 and Jaccard = 0.56. For Track 1 Subtask 3, no official generation baseline beyond simple retrieval was provided. In Track 2, Gemini 1.5 Pro and DeepSeek v3 both achieved Accuracy = 0.47 on Subtask 1, while Gemini 1.5 Pro achieved BERTScore = 0.844 on Subtask 2 (Alhuzali et al., 27 Aug 2025).

5. Participation profile and modeling strategies

MentalQA attracted 46 registered teams. Final submissions numbered 9 teams for Subtask 1, 7 teams for Subtask 2, and 6 teams for Subtask 3, with 10 description papers accepted. The model families reported for this track include Arabic-tuned PLMs such as AraBERT and Saka-14B, multilingual LLMs such as Llama and GPT-3/4, retrieval-augmented generation, and few-shot or zero-shot prompting (Alhuzali et al., 27 Aug 2025).

MedArabiQ involved 26 registered participants, 7 submissions for Subtask 1, 11 submissions for Subtask 2, and 5 system description papers. The reported system families include proprietary LLMs such as Gemini 2.5, DeepSeek v3, and GPT, together with ensemble voting, prompt engineering, and retrieval-based context augmentation (Alhuzali et al., 27 Aug 2025).

A concrete example of Track 2 methodology is provided by the !MSA system report. That system used Gemini 2.5 Flash with fixed decoding hyperparameters $i$3, top-$i$4, and top-$i$5. For Subtask 1 it combined three prompting schemes—Arabic Few-Shot (AFS), English Translation + Answer (ETA), and Refinement + Answer (RFA)—and then applied majority voting. For Subtask 2 it used a single Arabic prompt with four few-shot examples to handle fill-in-the-blank, consultation, grammatical error correction, and paraphrase settings. The same report also details minimal preprocessing and post-processing, including punctuation normalization, option-label normalization, symbol mapping to Arabic letters, markdown removal, and concise-output enforcement (Tarek et al., 14 Sep 2025).

6. Empirical outcomes and error structure

The MentalQA results show substantial variance in classification but tight clustering in generation. In Subtask 1, the best system, mucAI, reached $i$6 and $i$7, while lower-performing systems were approximately in the $i$8–$i$9 range. In Subtask 2, Sindbad/MarsadLab reached $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$0 and $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$1, substantially above the baseline value of $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$2. By contrast, generative performance in Subtask 3 clustered narrowly, with top BERTScore $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$3 for RetAug and bottom performance around $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$4. The task paper interprets this as indicating the difficulty of open-ended mental-health advice and the limits of automatic metrics (Alhuzali et al., 27 Aug 2025).

The reported MentalQA errors are diagnostically revealing. Low-frequency question types and emotional-support labels were under-predicted, producing precision/recall imbalances. Generative outputs also exhibited hallucinations or culturally inappropriate phrasing. These failure modes are particularly significant because MentalQA requires not only semantic adequacy but also empathetic and culturally situated responses (Alhuzali et al., 27 Aug 2025).

Track 2 exhibited a different profile. In Subtask 1, all three teams achieved approximately $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$5–$$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$6 accuracy, clearly exceeding the $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$7 baseline. In Subtask 2, BERTScores ranged from $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$8 to $$\mathrm{Precision}_i = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_i = \frac{TP_i}{TP_i + FN_i},$$9 against the baseline $F_1$0, which the organizers described as a narrow but meaningful gain. Error categories included confusion among similarly phrased distractors in MCQs and incomplete or under-specific answers in open-ended patient–doctor prompts (Alhuzali et al., 27 Aug 2025).

The !MSA report provides a system-level explanation for part of this behavior. Its ensemble decision rule for Subtask 1 was

$F_1$1

with tie priority RFA $F_1$2 AFS $F_1$3 ETA. On the 100-item test set, ETA achieved 69.0%, AFS 71.0%, RFA 74.0%, and the ensemble 76.0%, securing second place. On Subtask 2, unified prompting achieved 86.953% BERTScore, also securing second place. The same report states that removing RFA from the ensemble reduced accuracy by 3 percentage points, removing AFS by 2 percentage points, and removing ETA by 1 percentage point; it also states that post-processing improved Subtask 2 BERTScore by approximately 2–3 percentage points (Tarek et al., 14 Sep 2025). This suggests that, within MedArabiQ, lightweight prompt diversification and output normalization were operationally important.

7. Limitations, ethics, and future development

AraHealthQA 2025 explicitly identifies resource scarcity as a continuing bottleneck. The organizers argue that Arabic-specific mental health and medical corpora remain limited in size and that expansion in dialect and specialty coverage is critical. They further report that hybrid approaches combining lightweight fine-tuning with retrieval-augmented prompts outperformed pure zero-shot or few-shot methods, and that prompt engineering techniques such as chain of thought, instruction tuning, and GrIPS consistently improved results, although irrelevant examples could introduce hallucinations (Alhuzali et al., 27 Aug 2025).

The benchmark also draws a sharp distinction between automatic metric performance and clinically meaningful robustness. Weighted $F_1$4, Jaccard, and BERTScore were described as capturing surface alignment but not clinical safety or cultural sensitivity. Future editions were therefore recommended to incorporate human-in-the-loop evaluation with clinicians and native speakers, as well as human expert panels for final answer validation and reliability scoring. Proposed extensions include larger datasets, greater dialectal diversity, additional specialties such as oncology and neurology, multilingual Spanish/English/Arabic evaluation, and richer patient-education scenarios (Alhuzali et al., 27 Aug 2025).

Ethical constraints are treated as central rather than peripheral. The organizers state that the submitted systems remain research prototypes and that real-world deployment must guard against unsafe or stigmatizing advice through bias audits and clinician supervision (Alhuzali et al., 27 Aug 2025). A common misconception in this area is that strong leaderboard scores suffice for deployment readiness. The task description rejects that view indirectly: no formal significance testing was reported, automatic metrics were acknowledged as incomplete, and open-ended mental-health generation remained vulnerable to hallucinations and culturally inappropriate phrasing (Alhuzali et al., 27 Aug 2025).

Within the wider Arabic medical QA landscape, AraHealthQA 2025 can therefore be understood as both a benchmark and a design template. MedAraBench demonstrated the value of large-scale standardized MCQ evaluation across 19 specialties and five difficulty levels, while also showing that clinical reasoning remains challenging and that dialectal or code-mixed clinical narratives were still missing (Abu-Daoud et al., 2 Feb 2026). AraHealthQA 2025 extends that agenda by combining exam-style medical QA with patient-facing, supportive, and generative tasks, making evaluation of Arabic health question answering more representative of real clinical communication.