MediQAl: French Medical QA Benchmark

• MediQAl is a French medical QA benchmark based on exam questions that test factual recall and clinical reasoning across 41 subjects.
• It comprises 32,603 questions formatted as MCQU, MCQM, and OEQ, with explicit labeling for Understanding and Reasoning tasks.
• Evaluations on 14 language models reveal that reasoning questions score about 5.12 points lower than understanding, offering diagnostic insights.

Searching arXiv for the specific MediQAl paper and closely related named resources to ground the article. MediQAl is a French medical question answering benchmark designed to evaluate both factual medical recall and clinical reasoning in LLMs. It contains 32,603 questions sourced from French medical examinations across 41 medical subjects, and it spans three task formats: Multiple-Choice Question with Unique answer (MCQU), Multiple-Choice Question with Multiple answer (MCQM), and Open-Ended Question with Short-Answer (OEQ). A defining feature is that every question is labeled as either Understanding or Reasoning, enabling explicit separation between medical knowledge retrieval and reasoning-heavy inference over clinical scenarios (Bazoge, 28 Jul 2025).

1. Origin, scope, and motivation

MediQAl was introduced in response to several limitations in prior medical QA benchmarking. Existing resources are described as heavily English-centric, often dominated by single-answer multiple-choice questions, limited in task diversity and medical subject coverage, and frequently weak at testing real-world reasoning or open-ended clinical answering (Bazoge, 28 Jul 2025). Within French medical NLP, the benchmark addresses a particularly clear resource gap: FrenchMedMCQA has only 3,105 questions and is limited to pharmacy, MedFrenchmark has only 114 open questions, and the French subset of MedExpQA is translated from Spanish rather than derived from native French examination material (Bazoge, 28 Jul 2025).

The dataset is built from French medical licensing examinations and related training materials used to prepare for the national French medical exam (ECN). The questions were manually created and verified by academic and hospital faculty members, which gives the benchmark the character of an exam-grounded evaluation resource rather than a synthetic or loosely curated corpus (Bazoge, 28 Jul 2025). It spans subjects including cardiology, pediatrics, genetics, ophthalmology, biochemistry, neurology, hematology, dermatology, microbiology, pharmacology, psychiatry, and emergency medicine (Bazoge, 28 Jul 2025).

This design suggests that MediQAl is intended not merely as a leaderboard dataset, but as a structured probe of whether models can handle the breadth of French medical knowledge and the depth of clinically framed reasoning.

2. Corpus composition and distributional structure

MediQAl contains 32,603 questions distributed across three task types:

Task	Total questions	Understanding / Reasoning
MCQU	17,017	11,336 / 5,681
MCQM	10,617	7,742 / 2,875
OEQ	4,969	1,842 / 3,125

The dataset also distinguishes between isolated questions and in-context questions with clinical scenarios. For MCQU, the split is 10,087 isolated and 6,930 in-context; for MCQM, 6,543 isolated and 4,074 in-context; for OEQ, 1,015 isolated and 3,954 in-context (Bazoge, 28 Jul 2025). The high proportion of in-context items in OEQ is especially notable, because it shifts evaluation away from pure recall and toward concise free-text reasoning over case material.

Average lengths further reinforce the task differences. MCQU has an average question length of 19.82 words and average clinical scenario length of 99.97 words; MCQM has 13.99 and 107.24; OEQ has 19.40 for the question, 132.19 for the clinical scenario, and 34.68 for the answer (Bazoge, 28 Jul 2025). OEQ is therefore the most textually demanding component in both input and output space.

3. Formal task definitions and cognitive labels

The benchmark defines three task formats formally. For MCQU, the instance structure is

$X = \{C, Q, (O_1, ..., O_5), A\}$

where $C$ is an optional clinical scenario, $Q$ is the question, $(O_1, ..., O_5)$ are five candidate answers, and $A$ is a single correct option $O_i$ (Bazoge, 28 Jul 2025).

For MCQM, the same structure is used,

$X = \{C, Q, (O_1, ..., O_5), A\},$

but now $A$ is a subset of the options, with $|\mathcal{A}| \geq 2$ (Bazoge, 28 Jul 2025). This makes MCQM materially harder than single-answer selection, because partial correctness becomes central to evaluation.

For OEQ, the structure is

$X = \{C, Q, A\},$

where $C$0 is a short free-text answer with answer length less than 200 tokens (Bazoge, 28 Jul 2025).

A central innovation is the Understanding / Reasoning labeling. Understanding denotes questions with little to no multi-step reasoning, mainly testing medical knowledge and recall; Reasoning denotes complicated, reasoning-heavy questions requiring multi-step clinical inference (Bazoge, 28 Jul 2025). This explicit cognitive annotation makes MediQAl more than a format-diverse benchmark: it becomes a tool for decomposing model competence into recall-oriented and reasoning-oriented behavior.

4. Data acquisition, filtering, and annotation quality

The raw material was collected from publicly available websites, forums where professors and students share exam questions, and training materials for the French national medical exam, especially ECN-style questions (Bazoge, 28 Jul 2025). The MCQU and MCQM subsets were automatically extracted from the qcmlab website in March 2024. The OEQ subset came from a mixture of HTML and PDF files: HTML files were parsed automatically with regular expressions, while PDF files were manually curated when automatic parsing was not feasible (Bazoge, 28 Jul 2025).

The dataset construction process includes several filtering stages. For MCQU and MCQM, questions missing correct answers or options were removed. A three-model filtering process then excluded questions deemed too easy using Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3; if any of these models answered a question correctly, the item was removed from the test set (Bazoge, 28 Jul 2025). For OEQ, questions containing images or tables were removed, duplicates were filtered using TF-IDF cosine similarity with threshold 0.70, and answers longer than 200 tokens were excluded (Bazoge, 28 Jul 2025).

The Understanding/Reasoning labels were assigned using gpt-4o-2024-08-06 and then manually checked on 858 questions. The authors found 72 explicitly mislabeled questions, corresponding to an 8.4% error rate (Bazoge, 28 Jul 2025). This does not eliminate annotation noise, but it does quantify it, which is unusually valuable for interpreting reasoning-label analyses.

The final splits are MCQU: Train 10,113; Validation 2,561; Test 4,343, MCQM: Train 5,767; Validation 1,466; Test 3,384, and OEQ: Test only 4,969 (Bazoge, 28 Jul 2025).

5. Evaluation protocol and empirical findings

MediQAl was evaluated in a zero-shot setting on 14 LLMs, with greedy decoding when available (Bazoge, 28 Jul 2025). The evaluated systems include vanilla LLMs such as GPT-4o-2024-08-06, DeepSeek-V3, Qwen2.5-72B-Instruct, and Llama-3.3-70B-Instruct, as well as reasoning-oriented models such as o3-2025-04-16, DeepSeek-R1, DeepSeek-R1-Distill-Llama-70B, HuatuoGPT-o1-8B, and FineMedLM-o1 8B (Bazoge, 28 Jul 2025).

The evaluation metrics are task-specific. MCQU uses Accuracy. MCQM uses two metrics:

$C$1

and

$C$2

For OEQ, the benchmark reports ROUGE-1, BLEU-4, BERTScore, and LLM-as-Judge using Gemini-2.0-Flash, with judge scores assigned from 1 to 10 and then scaled to 0–100 (Bazoge, 28 Jul 2025).

The strongest reported results come from o3. On MCQU, o3 reaches 73.15 average accuracy, compared with 67.03 for DeepSeek-R1, 63.32 for DeepSeek-V3, and 60.95 for GPT-4o (Bazoge, 28 Jul 2025). On MCQM, o3 reaches 55.05 EMR and 79.7 Hamming, ahead of DeepSeek-R1 at 48.88 / 77.54, DeepSeek-V3 at 46.13 / 76.45, and GPT-4o at 43.65 / 74.04 (Bazoge, 28 Jul 2025). On OEQ, o3 scores 82.16 under LLM-as-Judge, followed by DeepSeek-R1 at 74.29, GPT-4o at 68.83, and DeepSeek-V3 at 52.19 (Bazoge, 28 Jul 2025).

One of the benchmark’s central findings is the persistent gap between factual recall and reasoning. Averaged across all model-task combinations, Reasoning questions are 5.12 points lower than Understanding questions. By task, the gap is 7.54 for OEQ, 3.90 for MCQU, and 3.93 for MCQM (Bazoge, 28 Jul 2025). Reasoning-augmented models reduce this gap but do not eliminate it: on MCQU, the gap is 2.15 for reasoning-augmented models versus 5.49 for vanilla models; on MCQM, 2.02 versus 5.55; on OEQ, 7.29 versus 7.79 (Bazoge, 28 Jul 2025).

The paper also identifies a metric-related caution. On OEQ, DeepSeek-R1 can outperform o3 on ROUGE and BLEU, but o3 is better under LLM-as-Judge, which the authors consider more aligned with clinical quality (Bazoge, 28 Jul 2025). A common misconception is therefore that lexical overlap necessarily tracks medically preferable answers; MediQAl’s results argue against that simplification.

6. Position within medical QA research and naming ambiguity

MediQAl belongs to a broader family of medical QA benchmarks, but its combination of native French exam content, three task formats, 41 medical subjects, and explicit reasoning labels gives it a distinct role (Bazoge, 28 Jul 2025). It differs sharply from MeDiSumQA, which is a patient-oriented benchmark over MIMIC-IV discharge summaries and contains 416 QA pairs after physician curation, emphasizing safe and patient-friendly answers rather than exam-style medical reasoning (Dada et al., 5 Feb 2025). It also differs from TriMediQ, which targets interactive medical QA by converting dialogue turns into triplet-structured knowledge graphs and reports up to 10.4% improvement in accuracy on iMedQA over five baselines (Meng et al., 3 Oct 2025).

The name also risks confusion with similarly titled initiatives. “Medico 2025: Visual Question Answering for Gastrointestinal Imaging” frames MediQAl / Medico 2025 as a benchmark and shared challenge for visual question answering in gastrointestinal endoscopy, built around Kvasir-VQA-x1 and emphasizing correct answer + explainable reasoning over endoscopic images rather than text-only medical QA (Gautam et al., 14 Aug 2025). More broadly, the MEDIQA 2019 shared task concerned medical natural language inference, recognizing question entailment, and QA re-ranking, with systems such as DoubleTransfer and Dr.Quad emphasizing transfer learning, domain adaptation, and contextualized representations rather than multilingual exam benchmarking (Xu et al., 2019); (Kumar et al., 2019).

Within that landscape, MediQAl’s main significance is methodological rather than merely linguistic. It provides a benchmark where multilinguality, task heterogeneity, and reasoning analysis are jointly operationalized. This suggests that its value lies not only in expanding French medical NLP resources, but also in making model failures more diagnosable: a system can now be examined by answer format, by clinical context, and by the distinction between Understanding and Reasoning rather than by aggregate accuracy alone.