MentalQA: Arabic Mental Health QA Corpus
- MentalQA is an annotated Arabic corpus of mental-health QA exchanges capturing real-life patient–doctor interactions and diverse support styles.
- The resource comprises 2,621 unique QA pairs from Altibbi.com, with detailed annotations for question types and answer strategies by native Arabic experts.
- Benchmark studies using Arabic PLMs like MARBERT show strong F1 and Jaccard scores, highlighting its utility for developing culturally-aware mental-health support systems.
MentalQA is an annotated Arabic corpus of mental-health question–answer interactions and, subsequently, a benchmark track for Arabic mental-health question answering. It was introduced to address a specific resource gap: Arabic mental-health NLP had been dominated by one-way texts such as tweets, especially for depression detection, whereas practical support systems require data that reflects how patients ask for help and how professionals respond. The resource therefore centers on conversational patient–doctor exchanges and was framed around three tasks: question type classification, answer strategy classification, and, later, generative Arabic mental-health QA (Alhuzali et al., 2024, Alhuzali et al., 27 Aug 2025).
1. Origins, scope, and source material
MentalQA was constructed from Altibbi.com, an Arabic medical platform offering articles, glossaries, and question–answer services. The collection process drew on QA posts from 2020–2021 together with the platform’s most popular posts, initially yielding 53,402 unique QA pairs. Filtering to the Mental Health section produced 2,621 unique question–answer pairs, each consisting of one patient question and one doctor answer. At the corpus level, the paper reports 84 distinct doctors, an average question length of 30 words, and an average answer length of 31 words (Alhuzali et al., 2024).
The resource was explicitly positioned against prior Arabic work focused on social-media classification rather than interactive support. Its domain scope is broader than datasets centered on a single disorder such as depression, suicidality, self-harm, or anxiety. Instead, MentalQA covers multiple mental-health information needs and multiple professional response styles, which makes it applicable to intent recognition, answer retrieval, response-style modeling, and Arabic conversational support systems for early support, triage, and patient education (Alhuzali et al., 2024).
A second stage in the resource’s history occurred when AraHealthQA 2025 defined a dedicated MentalQA track as part of a shared task on Arabic health question answering. In that setting, MentalQA was presented as the benchmark component focused specifically on Arabic mental-health QA and described as targeting responses that are not only accurate, but also supportive, culturally aware, and contextually appropriate (Alhuzali et al., 27 Aug 2025).
2. Corpus construction and annotation procedure
Although the filtered corpus contains 2,621 mental-health QA pairs, the annotation study covered 500 QA posts, producing 500 question annotations + 500 answer annotations = 1,000 annotations in the authors’ accounting. The annotators were three native Arabic speakers with experience in biomedical text and natural language processing. The paper does not state that they were clinicians or psychiatrists (Alhuzali et al., 2024).
The annotation workflow was staged. It began with small batches of 20 questions, later increasing gradually to 100 questions as familiarity improved. The first 20 questions formed a trial batch annotated in parallel by all annotators and then reviewed for quality. For the first 200 questions, all three annotators worked collaboratively on the same items; before each new batch, they held group meetings to discuss disagreements and document agreed resolutions. For the remaining 300 questions, each annotator was assigned 100 questions independently. Final labels were produced by majority voting (Alhuzali et al., 2024).
The paper also records several procedural constraints and ambiguities. It confirms that 200 posts were annotated by all three annotators, which implies that the reliability analysis was performed on the overlapping subset rather than the full 500-item annotation pool. It does not provide a train/dev/test split, benchmark models, tokenization details, normalization rules, or detailed preprocessing beyond filtering to the mental-health category and anonymization. For ethics and privacy, the authors state that personally identifiable information was removed, the dataset was anonymized, and data security measures were applied (Alhuzali et al., 2024).
3. Annotation schema and task formulation
MentalQA’s schema combines and adapts a health-question taxonomy and answer-strategy labels from prior work. The original paper describes an internal inconsistency in the question taxonomy: the prose mentions an Other category and Table 1 reports 7 question categories, while the abstract and main narrative foreground 6 core question types. The later benchmarking paper operationalizes question classification as a seven-label multi-label task with Diagnosis, Treatment, Anatomy, Epidemiology, Healthy Lifestyle, Health Provider Choice, and Other (Alhuzali et al., 2024, Alhuzali et al., 2024).
| Layer | Labels | Notes |
|---|---|---|
| Question types | Diagnosis; Treatment; Anatomy and physiology; Epidemiology; Healthy lifestyle; Provider choice; Other | Original analysis emphasizes six core types; later benchmark uses seven labels |
| Answer strategies | Information; Direct Guidance; Emotional Support | Multi-label; strategy labels, not quality labels |
The six core question types in the original dataset paper are defined as follows. Diagnosis covers interpretation of symptoms, tests, disease manifestations, or diagnostic criteria. Treatment covers treatment options, medication use, side effects, contraindications, and how to use a drug. Anatomy and physiology covers basic medical knowledge such as tissues, organs, metabolism, or bodily and functional explanations relevant to mental-health concerns. Epidemiology covers disease course, prognosis, sequelae, etiology, causation, and risk factors. Healthy lifestyle covers diet, exercise, mood regulation, and related lifestyle factors. Provider choice covers recommendations about hospitals, departments, doctors, or the process of visiting and selecting a provider (Alhuzali et al., 2024).
The answer taxonomy has three labels. Information includes factual information, explanations, resources, and even requests for more information from the patient; prior Restatement and Interpretation labels were merged into this category. Direct Guidance captures suggestions, instructions, and action-oriented advice. Emotional Support broadens a prior Approval and Reassurance label and includes reassurance and supportive care. Self-disclosure was excluded because all answers came from doctors rather than peers. The paper stresses that these labels characterize response strategy only; they do not evaluate clinical correctness, completeness, or whether the answer satisfied user needs (Alhuzali et al., 2024).
4. Reliability, quantitative properties, and descriptive findings
Inter-annotator agreement was measured with Fleiss’ Kappa. The question-type value is consistent at . For answer strategies, the paper contains an internal inconsistency: the abstract reports , whereas the results section and Table 2 report . Table 2 also reports 500 annotated questions, 500 annotated answers, 200 posts where all three annotators annotate, 192 question posts where at least two annotators agreed on at least one category, 100 where all three agreed on at least one question category, 200 answer posts where at least two annotators agreed on at least one category, and 152 where all three agreed on at least one answer category. At the subset level, 82\% of questions and 69\% of answers had a single label, while 18\% of questions and 31\% of answers had multiple labels (Alhuzali et al., 2024).
Among annotated posts where at least two annotators agreed, the most common question types were Treatment: 296 (57\%) and Diagnosis: 286 (55\%). Epidemiology occurred 116 (22\%), Healthy lifestyle 125 (24\%), Provider choices 20 (4\%), and Anatomy and physiology 32 times. For answers, Information occurred 388 (75\%), Direct guidance 290 (56\%), and Emotional support 61 (11\%). Because MentalQA is multi-label, these percentages sum to more than 100 (Alhuzali et al., 2024).
The paper adds several descriptive analyses. It reports a strong positive relationship between Diagnosis and Treatment questions and the answer strategies Information and Direct Guidance, while Emotional Support is especially associated with Diagnosis questions. For gender patterns, it reports that male patients most frequently asked treatment and diagnosis questions, while female patients most frequently asked treatment and diagnosis questions. The age analysis shows that treatment questions dominate across all groups, with reported values of for users under 20, for ages 20–30, for ages 30–40, and 0 above 40, where 1 denote diagnosis, treatment, epidemiology, and healthy lifestyle respectively (Alhuzali et al., 2024).
Exploratory sentiment analysis was performed with CAMelBERT. Patient questions were predominantly negative (415 negative, 75 neutral, 10 positive), whereas doctor answers were more often neutral (325 neutral, 139 negative, 36 positive). The paper also reports answer timing and length by question-type combinations, including Diagnosis: 2.61 days, 22.33 words, Treatment: 11.73 days, 31.37 words, and Diagnosis + Treatment: 11.06 days, 47.30 words, and concludes that diagnosis questions receive faster but shorter responses, treatment questions take longer and require more detail, and mixed-type questions often require longer replies (Alhuzali et al., 2024).
5. Benchmarking studies and the AraHealthQA 2025 shared task
The first dedicated benchmark study on MentalQA treated both question classification and answer classification as multi-label classification tasks over a 60/20/20 split, yielding 300 training, 100 validation, and 100 test examples for each task. It compared traditional sparse features with SVM, Arabic PLMs used as frozen feature extractors with SVM, fine-tuned Arabic PLMs, and prompted GPT models. The strongest reported question-classification result in Table 3 is MARBERT with 0.85 micro F1 / 0.85 weighted F1 / 0.80 Jaccard. For answer classification, the best table result is again MARBERT, with 0.95 micro F1 / 0.95 weighted F1 / 0.94 Jaccard. The same paper notes that classical sparse baselines are unexpectedly competitive on this small dataset, that fine-tuning is consistently beneficial, that more training data improves performance, and that few-shot prompting substantially improves over zero-shot, especially for answer classification. It also contains an internal inconsistency: the abstract states a best answer-classification Jaccard of 0.86, whereas Table 3 reports 0.94 (Alhuzali et al., 2024).
In AraHealthQA 2025, MentalQA became a shared-task benchmark with three subtasks over the same 500 question-answer pairs, split into 300 training, 50 development, and 150 test examples for all subtasks. Subtask 1 is Question Type Classification and Subtask 2 is Answer Strategy Classification, both evaluated by Weighted F1 and Jaccard score. Subtask 3 is generative Question Answering, evaluated by BERTScore. Evaluation ran on Codabench with a blind test set, up to five runs per subtask, and only the highest-scoring submission per team counted. Track 1 registered 46 unique teams, with 9 submissions from 9 teams for Subtask 1, 7 submissions from 7 teams for Subtask 2, 6 submissions from 6 teams for Subtask 3, and 10 system description papers accepted (Alhuzali et al., 27 Aug 2025).
Official shared-task results show that question-type classification is the hardest of the three tasks: mucAI ranked first on Subtask 1 with 0.61 Weighted-F1 and 0.53 Jaccard. Answer-strategy classification was stronger overall, with Sindbad and MarsadLab tied at 0.79 Weighted-F1, while Sindbad and Binary_Bunch shared the highest 0.71 Jaccard. For generative QA, RetAug ranked first with 0.679 BERTScore, ahead of MindLLM at 0.670 and Sindbad at 0.668. The shared-task paper emphasizes that retrieval augmentation, prompt engineering, instruction tuning, and hybrid pipelines were effective, but also that automatic semantic-overlap metrics do not fully capture cultural sensitivity, empathy, safety, or contextual appropriateness in mental-health QA (Alhuzali et al., 27 Aug 2025).
6. Limitations, safety considerations, and place in the broader landscape
MentalQA’s limitations are explicit in the original dataset paper. The data come from a single consumer health platform, so the source population is not representative of everyone with mental-health concerns. Only 500 of 2,621 filtered QA pairs were annotated. The labels characterize communicative strategy rather than answer quality. The paper also leaves several reporting inconsistencies unresolved, including the 6-versus-7 question-category issue, the 0.96-versus-0.98 answer-strategy agreement issue, and the fact that only 200 posts are clearly described as triple-annotated even though 500 posts are described as annotated (Alhuzali et al., 2024).
The shared-task paper extends the safety discussion. It states that the datasets were curated from publicly available or anonymized sources, warns that models may still generate misleading, unsafe, or culturally inappropriate responses, and argues that systems should be treated as research artifacts rather than clinical advice. It recommends future human-in-the-loop evaluation with clinicians and native speakers, along with stronger bias evaluation and safety guardrails (Alhuzali et al., 27 Aug 2025). A later auditing study on mental-health QA reinforces that caution by arguing that LLMs in mental-health settings can amplify stigma, suppress nuance, and differentially frame symptoms and treatment across demographic intersections; it treats multi-hop QA as a way to expose such “amplified” and “silenced” perspectives in mental-health discourse (Haider et al., 22 Jun 2025).
Within the broader landscape of mental-health QA resources, MentalQA occupies a distinct niche. Unlike MHQA, which is a multiple-choice benchmark built from PubMed abstracts across anxiety, depression, trauma, and obsessive/compulsive issues, MentalQA is grounded in real patient–doctor Arabic exchanges (Racha et al., 21 Feb 2025). Unlike MENTAT, which is a clinician-authored dataset of psychiatry decision-making spanning treatment, diagnosis, documentation, monitoring, and triage, MentalQA focuses on support-oriented Arabic QA and communicative intent at the question and answer levels (Lamparth et al., 22 Feb 2025). Its significance therefore lies less in exam-style knowledge testing than in providing an Arabic, conversational, professionally answered substrate for research on classification, retrieval, generation, and safety-sensitive benchmarking in mental-health communication (Alhuzali et al., 2024, Alhuzali et al., 27 Aug 2025).
