PediatricsMQA: Pediatric Medical QA Benchmark
- PediatricsMQA is a multimodal pediatric QA benchmark designed to expose age bias by evaluating both text and image-based questions.
- The benchmark comprises 5,484 multiple-choice questions covering 131 text topics and 67 imaging modalities, with explicit developmental-stage stratification.
- Reported findings reveal poor model performance on PediatricsMQA, underscoring the need for age-aware and specialized pediatric evaluation.
PediatricsMQA is a multi-modal pediatric medical question-answering benchmark introduced to evaluate LLMs on text questions and vision-LLMs on image-grounded pediatric questions. It is designed around a specific claim: generic medical QA benchmarks underrepresent children, and current medical LLMs and VLMs exhibit age bias in pediatric settings. The benchmark therefore emphasizes pediatric specialization, multimodality, and developmental-stage stratification rather than treating child health as a sparse subset of general medicine. In its reported form, PediatricsMQA contains 3,417 text-based multiple-choice questions across 131 pediatric topics and 2,067 vision-based multiple-choice questions derived from 634 pediatric images spanning 67 imaging modalities and 256 anatomical regions (Bahaj et al., 22 Aug 2025).
1. Rationale and conceptual scope
PediatricsMQA is framed as a response to two linked problems. The first is model-level age bias: the paper cites prior work indicating that medical text QA benchmarks are more relevant to older people than younger people, and that medical VLMs also perform better on older age groups than younger age groups. The second is broader pediatric underrepresentation in medicine, including lower pediatric funding and fewer pediatric trials despite substantial disease burden. Within that framing, PediatricsMQA is presented as a pediatric-focused benchmark intended to make age-related failures more visible during evaluation rather than after deployment (Bahaj et al., 22 Aug 2025).
The benchmark’s scope is deliberately narrower than “medicine” and broader than a single pediatric subspecialty. On the text side, it targets pediatric medical reasoning over 131 categories. On the vision side, it targets pediatric visual reasoning across a heterogeneous image base that includes 67 imaging modalities and 256 anatomical regions. This suggests an attempt to stress both breadth and heterogeneity rather than optimize for one canonical pediatric task such as board-style text QA or radiology-only VQA.
A notable design decision is that PediatricsMQA contains only multiple-choice questions. The paper explicitly excludes binary QA and open-ended QA, arguing that MCQs are more challenging than binary questions and more exact than open-ended responses for evaluating knowledge. This keeps evaluation mechanically simple, but it also means the benchmark primarily measures answer selection rather than long-form pediatric explanation or dynamic clinical dialogue (Bahaj et al., 22 Aug 2025).
2. Dataset structure and developmental stratification
PediatricsMQA contains 5,484 multiple-choice QA pairs in total. The benchmark is divided into a text pediatric QA component and a vision pediatric QA component.
| Component | Count | Scope |
|---|---|---|
| Text QA | 3,417 | 131 pediatric topics |
| Vision QA | 2,067 | 634 images, 67 modalities, 256 regions |
| Total | 5,484 | MCQ-only benchmark |
A defining feature is explicit developmental-stage annotation. The paper partitions the benchmark into seven child development stages: Prenate, Neonate, Infant, Toddler, Preschool, School-Age, and Adolescent. The exact ranges are given as Prenate: months to 0 weeks, Neonate: 0 weeks to 4 weeks, Infant: 1 month to 1 year, Toddler: 1–3 years, Preschool: 3–5 years, School-Age: 6–12 years, and Adolescent: 13–18 years. These labels function both as dataset metadata and as subgroup axes for evaluation (Bahaj et al., 22 Aug 2025).
The benchmark summary in the paper does not provide a full numeric count table for every developmental stage, topic, modality, or anatomical region, but it does provide stage-wise and subgroup analyses. For text QA, the benchmark is described as covering 131 pediatric categories. For vision QA, the benchmark is organized around 67 imaging modalities and 256 anatomical regions. The appendix further identifies recurring difficult and easier subdomains. Frequently difficult text categories include Lipid Disorders, Pharmacology, and Neuroradiology, while recurrently easier ones include Developmental Psychology, Porphyria, and Paediatric orthopaedic disorders. On the vision side, more structured modalities such as optical, physical exam, surgical specimen, and IHC staining are often easier, whereas natural image, cytopathology, VCE, and Sanger sequencing are often harder (Bahaj et al., 22 Aug 2025).
3. Construction pipeline
The benchmark is built with a hybrid manual-automatic pipeline. Its text component extends PediatricsQA, which originally had 831 TQA pairs. The paper states that PediatricsMQA expanded this to 3401 TQA pairs during construction, while the benchmark summary and abstract consistently report 3,417 TQA pairs; the text does not explain this discrepancy. The text QA corpus was assembled from PediatricsQA, pediatric books, question banks, medical exams, and pediatric medical books including Nelson textbook of pediatrics. The abstract also describes the source mixture as including peer-reviewed pediatric literature, validated question banks, existing benchmarks, and existing QA resources (Bahaj et al., 22 Aug 2025).
Text questions were either manually created from pediatric medical books or manually extracted from books already containing QA pairs. To reduce copyright concerns, the authors used Gemini-2.0-Flash to paraphrase questions and options while preserving meaning and medical terminology. A subsequent manual filtering round removed questions with obvious answers and questions that required contextual information not present in the MCQ itself. The paper also notes that some questions about women’s pregnancy were retained because of their importance for prenate children’s health (Bahaj et al., 22 Aug 2025).
The vision component was assembled from three routes. One route used pediatric subsets from HAM10000, with handwritten templates to generate diagnosis and localization MCQs from lesion metadata. A second route used pediatric subsets from FairVLMed, where Gemini-2.0-Flash generated five MCQs per case from the image and accompanying medical notes, followed by manual filtering to retain only visually answerable items whose answers were supported by the notes. A third route extracted images, captions, and passages from Case Reports in Pediatrics, then used Gemini-2.0-Flash to generate five relevant MCQs per image, again followed by manual curation because many generated questions were out of context or not visually answerable without reading the passage (Bahaj et al., 22 Aug 2025).
For VQA, human labelers also added two metadata fields—question source and answer source—indicating whether a question depended on caption or passage context and whether the answer was supported by that context or by the model’s own knowledge. Age and gender were additionally extracted from article text. Adult images were filtered out except when a child was included indirectly, such as prenatal imaging of a pregnant woman (Bahaj et al., 22 Aug 2025).
4. Task formulation and evaluation protocol
PediatricsMQA is evaluated as a direct-answer MCQ benchmark. For text QA, models receive a pediatric medical question with 4 to 6 answer options and are instructed to output only the option index or letter. For vision QA, models receive a pediatric medical image, a question, and multiple options, and are instructed to output only the option letter. The appendix prompts are consistent with zero-shot direct-answer evaluation rather than tool-augmented or chain-of-thought-heavy inference (Bahaj et al., 22 Aug 2025).
The paper reports accuracy as the central metric: overall benchmark accuracy, developmental-stage accuracies, and subgroup accuracies by topic, modality, and anatomical region. It does not introduce a formal mathematical scoring equation, nor does it report calibration metrics, subgroup-gap equations, or statistical significance tests. The benchmark is therefore primarily empirical and descriptive rather than formally metric-driven (Bahaj et al., 22 Aug 2025).
The evaluated text models are MedAlpaca, Llama-Medx, Gemini-1.5-Flash, Gemini-2.0-Flash, Llama-3.1 (8B), Llama-4-scout (17B), and Llama-4-Maverick (17B). The evaluated vision-LLMs are LLaVa-Med-7B, HuatuoGPT-Vision-7B, Gemini-1.5-Flash, and Gemini-2.0-Flash. Smaller open models were run locally on an RTX 3090 GPU, whereas Gemini and some Llama-family models were evaluated through API services (Bahaj et al., 22 Aug 2025).
5. Reported findings
A central empirical result is that PediatricsMQA is harder than the comparison benchmarks used in the paper. On the text side, PediatricsMQA is the worst-performing dataset for every evaluated model when compared against PediatricsQA, PubMedQA, MedQA, and MedMCQA. The best reported text accuracy on PediatricsMQA is 72.81 for Llama-4-Maverick, followed by 70.61 for Gemini-2.0-Flash and 66.14 for Llama-4-scout. Older or smaller medical models are markedly lower, including 50.43 for Llama-Medx and 31.05 for MedAlpaca (Bahaj et al., 22 Aug 2025).
On the vision side, PediatricsMQA is likewise the worst-performing benchmark for all tested VLMs relative to VQA-RAD, SLAKE, and PathVQA. The strongest reported PediatricsMQA VQA scores are 56.70 for HuatuoGPT-Vision-7B and 55.0 for Gemini-2.0-Flash, with LLaVa-Med-7B at 34.95 and Gemini-1.5-Flash at 30.01. This indicates that the benchmark is not merely pediatric in topic but also difficult in multimodal reasoning terms (Bahaj et al., 22 Aug 2025).
The paper interprets the developmental-stage results as evidence of age bias. The detailed tables, however, show modality-dependent variation rather than a single monotonic pattern. In text QA, stage-wise differences are substantial but model-specific: for example, Llama-4-Maverick ranges from 66.17 on Toddler to 77.98 on Adolescent, and Gemini-2.0-Flash ranges from 65.13 on Neonate to 74.67 on Adolescent. In vision QA, the pattern is more directional: Neonate and Infant are often stronger, while Adolescent and Preschool are often weaker. Gemini-2.0-Flash, for instance, scores 68.18 on Neonate and 47.24 on Adolescent, while HuatuoGPT-Vision scores 73.77 on Neonate and 54.04 on Adolescent (Bahaj et al., 22 Aug 2025).
Appendix analyses add further heterogeneity. In TQA, recurrently difficult categories include Lipid Disorders, Pharmacology, and Neuroradiology. In VQA, performance varies sharply by modality and anatomy: blood cells, buccal mucosa cells, coronary artery, and vertebral body are often easier, while gums, genital regions, parietal region, occipital region, axilla/armpit, and bronchus are often harder. The benchmark therefore exposes not only age sensitivity but also fine-grained instability across pediatric subdomains (Bahaj et al., 22 Aug 2025).
6. Position within pediatric QA research
PediatricsMQA occupies a distinct position within the emerging pediatric benchmark landscape. PediaBench is a Chinese pediatric benchmark with 5,749 questions spanning 12 disease groups and combining objective and subjective QA, including essay/short-answer and case analysis, whereas PediatricsMQA is multimodal and MCQ-only (Zhang et al., 2024). PEDIASBench is broader still: it evaluates foundational knowledge, dynamic diagnosis and treatment capability, and pediatric medical safety and ethics across 19 pediatric subspecialties and 211 representative diseases, making it a pediatrician-competence framework rather than a pure QA benchmark (Zhu et al., 17 Nov 2025). AfriMed-QA includes a pediatric subset of 747 questions overall and 585 expert questions, but it is multispecialty and LMIC-oriented rather than pediatric-exclusive (Olatunji et al., 2024).
Within that comparison space, PediatricsMQA’s distinguishing features are its multimodal design, its explicit developmental-stage stratification, and its use as a stress test for age-aware pediatric evaluation. The paper also outlines several future directions: a pediatrics leaderboard, expansion with more questions, images, topics, modalities, and variability, extension to video and audio, and the construction of more sophisticated multi-step reasoning tasks. At the same time, it identifies limits and risks: coverage remains incomplete, the benchmark is restricted to text and images, the current task format is MCQ-only, and broader-impact concerns include privacy, over-reliance on model outputs, and benchmark overfitting (Bahaj et al., 22 Aug 2025).
In that sense, PediatricsMQA is best understood not as a complete representation of pediatric clinical intelligence, but as a benchmark that makes a particular failure mode legible: strong performance on generic medical QA does not imply strong performance on child health, and pediatric evaluation requires age-aware, domain-specific, and multimodal testing (Bahaj et al., 22 Aug 2025).