---
title: 'MediQAl: French Medical QA Benchmark'
url: https://www.emergentmind.com/topics/mediqal
type: topic
---

# MediQAl: French Medical QA Benchmark

Searching arXiv for the specific MediQAl paper and closely related named resources to ground the article.
MediQAl is a French medical question answering benchmark designed to evaluate both factual medical recall and clinical reasoning in language models. It contains **32,603 questions** sourced from **French medical examinations** across **41 medical subjects**, and it spans three task formats: **Multiple-Choice Question with Unique answer (MCQU)**, **Multiple-Choice Question with Multiple answer (MCQM)**, and **Open-Ended Question with Short-Answer (OEQ)**. A defining feature is that every question is labeled as either **Understanding** or **Reasoning**, enabling explicit separation between medical knowledge retrieval and reasoning-heavy inference over clinical scenarios [2507.20917].

## 1. Origin, scope, and motivation

MediQAl was introduced in response to several limitations in prior medical QA benchmarking. Existing resources are described as **heavily English-centric**, often dominated by **single-answer multiple-choice questions**, limited in **task diversity** and **medical subject coverage**, and frequently weak at testing **real-world reasoning** or **open-ended clinical answering** [2507.20917]. Within French medical NLP, the benchmark addresses a particularly clear resource gap: **FrenchMedMCQA** has only **3,105 questions** and is limited to **pharmacy**, **MedFrenchmark** has only **114 open questions**, and the French subset of **MedExpQA** is translated from Spanish rather than derived from native French examination material [2507.20917].

The dataset is built from **French medical licensing examinations** and related training materials used to prepare for the national French medical exam (**ECN**). The questions were **manually created and verified by academic and hospital faculty members**, which gives the benchmark the character of an exam-grounded evaluation resource rather than a synthetic or loosely curated corpus [2507.20917]. It spans subjects including **cardiology, pediatrics, genetics, ophthalmology, biochemistry, neurology, hematology, dermatology, microbiology, pharmacology, psychiatry,** and **emergency medicine** [2507.20917].

This design suggests that MediQAl is intended not merely as a leaderboard dataset, but as a structured probe of whether models can handle the breadth of French medical knowledge and the depth of clinically framed reasoning.

## 2. Corpus composition and distributional structure

MediQAl contains **32,603 questions** distributed across three task types:

| Task | Total questions | Understanding / Reasoning |
|---|---:|---:|
| MCQU | 17,017 | 11,336 / 5,681 |
| MCQM | 10,617 | 7,742 / 2,875 |
| OEQ | 4,969 | 1,842 / 3,125 |

The dataset also distinguishes between **isolated** questions and **in-context** questions with clinical scenarios. For **MCQU**, the split is **10,087 isolated** and **6,930 in-context**; for **MCQM**, **6,543 isolated** and **4,074 in-context**; for **OEQ**, **1,015 isolated** and **3,954 in-context** [2507.20917]. The high proportion of in-context items in OEQ is especially notable, because it shifts evaluation away from pure recall and toward concise free-text reasoning over case material.

Average lengths further reinforce the task differences. **MCQU** has an average question length of **19.82** words and average clinical scenario length of **99.97** words; **MCQM** has **13.99** and **107.24**; **OEQ** has **19.40** for the question, **132.19** for the clinical scenario, and **34.68** for the answer [2507.20917]. OEQ is therefore the most textually demanding component in both input and output space.

## 3. Formal task definitions and cognitive labels

The benchmark defines three task formats formally. For **MCQU**, the instance structure is

$$
X = \{C, Q, (O_1, ..., O_5), A\}
$$

where \(C\) is an optional clinical scenario, \(Q\) is the question, \((O_1, ..., O_5)\) are five candidate answers, and \(A\) is a **single correct option** \(O_i\) [2507.20917].

For **MCQM**, the same structure is used,

$$
X = \{C, Q, (O_1, ..., O_5), A\},
$$

but now \(A\) is a **subset** of the options, with \(|\mathcal{A}| \geq 2\) [2507.20917]. This makes MCQM materially harder than single-answer selection, because partial correctness becomes central to evaluation.

For **OEQ**, the structure is

$$
X = \{C, Q, A\},
$$

where \(A\) is a short free-text answer with answer length **less than 200 tokens** [2507.20917].

A central innovation is the **Understanding / Reasoning** labeling. **Understanding** denotes questions with little to no multi-step reasoning, mainly testing medical knowledge and recall; **Reasoning** denotes **complicated, reasoning-heavy questions** requiring multi-step clinical inference [2507.20917]. This explicit cognitive annotation makes MediQAl more than a format-diverse benchmark: it becomes a tool for decomposing model competence into recall-oriented and reasoning-oriented behavior.

## 4. Data acquisition, filtering, and annotation quality

The raw material was collected from **publicly available websites**, **forums where professors and students share exam questions**, and **training materials** for the French national medical exam, especially **ECN-style questions** [2507.20917]. The **MCQU** and **MCQM** subsets were automatically extracted from the **qcmlab** website in **March 2024**. The **OEQ** subset came from a mixture of **HTML** and **PDF** files: HTML files were parsed automatically with regular expressions, while PDF files were manually curated when automatic parsing was not feasible [2507.20917].

The dataset construction process includes several filtering stages. For **MCQU** and **MCQM**, questions missing correct answers or options were removed. A **three-model filtering process** then excluded questions deemed too easy using **Llama-3.1-8B-Instruct**, **Qwen2.5-7B-Instruct**, and **Mistral-7B-Instruct-v0.3**; if any of these models answered a question correctly, the item was removed from the test set [2507.20917]. For **OEQ**, questions containing images or tables were removed, duplicates were filtered using **TF-IDF cosine similarity** with threshold **0.70**, and answers longer than **200 tokens** were excluded [2507.20917].

The Understanding/Reasoning labels were assigned using **gpt-4o-2024-08-06** and then manually checked on **858 questions**. The authors found **72 explicitly mislabeled questions**, corresponding to an **8.4% error rate** [2507.20917]. This does not eliminate annotation noise, but it does quantify it, which is unusually valuable for interpreting reasoning-label analyses.

The final splits are **MCQU: Train 10,113; Validation 2,561; Test 4,343**, **MCQM: Train 5,767; Validation 1,466; Test 3,384**, and **OEQ: Test only 4,969** [2507.20917].

## 5. Evaluation protocol and empirical findings

MediQAl was evaluated in a **zero-shot** setting on **14 large language models**, with **greedy decoding when available** [2507.20917]. The evaluated systems include vanilla LLMs such as **GPT-4o-2024-08-06**, **DeepSeek-V3**, **Qwen2.5-72B-Instruct**, and **Llama-3.3-70B-Instruct**, as well as reasoning-oriented models such as **o3-2025-04-16**, **DeepSeek-R1**, **DeepSeek-R1-Distill-Llama-70B**, **HuatuoGPT-o1-8B**, and **FineMedLM-o1 8B** [2507.20917].

The evaluation metrics are task-specific. **MCQU** uses **Accuracy**. **MCQM** uses two metrics:

$$
\text{Exact Match Ratio (EMR)} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i]
$$

and

$$
\text{Hamming Score} = \frac{1}{N} \sum_{i=1}^{N} \frac{|y_i \cap \hat{y}_i|}{|y_i \cup \hat{y}_i|}.
$$

For **OEQ**, the benchmark reports **ROUGE-1**, **BLEU-4**, **BERTScore**, and **LLM-as-Judge** using **Gemini-2.0-Flash**, with judge scores assigned from **1 to 10** and then scaled to **0–100** [2507.20917].

The strongest reported results come from **o3**. On **MCQU**, **o3** reaches **73.15** average accuracy, compared with **67.03** for **DeepSeek-R1**, **63.32** for **DeepSeek-V3**, and **60.95** for **GPT-4o** [2507.20917]. On **MCQM**, **o3** reaches **55.05 EMR** and **79.7 Hamming**, ahead of **DeepSeek-R1** at **48.88 / 77.54**, **DeepSeek-V3** at **46.13 / 76.45**, and **GPT-4o** at **43.65 / 74.04** [2507.20917]. On **OEQ**, **o3** scores **82.16** under **LLM-as-Judge**, followed by **DeepSeek-R1** at **74.29**, **GPT-4o** at **68.83**, and **DeepSeek-V3** at **52.19** [2507.20917].

One of the benchmark’s central findings is the persistent gap between factual recall and reasoning. Averaged across all model-task combinations, **Reasoning questions are 5.12 points lower** than **Understanding** questions. By task, the gap is **7.54** for **OEQ**, **3.90** for **MCQU**, and **3.93** for **MCQM** [2507.20917]. Reasoning-augmented models reduce this gap but do not eliminate it: on **MCQU**, the gap is **2.15** for reasoning-augmented models versus **5.49** for vanilla models; on **MCQM**, **2.02** versus **5.55**; on **OEQ**, **7.29** versus **7.79** [2507.20917].

The paper also identifies a metric-related caution. On OEQ, **DeepSeek-R1 can outperform o3 on ROUGE and BLEU**, but **o3 is better under LLM-as-Judge**, which the authors consider more aligned with clinical quality [2507.20917]. A common misconception is therefore that lexical overlap necessarily tracks medically preferable answers; MediQAl’s results argue against that simplification.

## 6. Position within medical QA research and naming ambiguity

MediQAl belongs to a broader family of medical QA benchmarks, but its combination of **native French exam content**, **three task formats**, **41 medical subjects**, and **explicit reasoning labels** gives it a distinct role [2507.20917]. It differs sharply from **MeDiSumQA**, which is a **patient-oriented** benchmark over **MIMIC-IV discharge summaries** and contains **416 QA pairs** after physician curation, emphasizing safe and patient-friendly answers rather than exam-style medical reasoning [2502.03298]. It also differs from **TriMediQ**, which targets **interactive** medical QA by converting dialogue turns into triplet-structured knowledge graphs and reports up to **10.4% improvement in accuracy** on **iMedQA** over five baselines [2510.03536].

The name also risks confusion with similarly titled initiatives. **“Medico 2025: Visual Question Answering for Gastrointestinal Imaging”** frames **MediQAl / Medico 2025** as a **benchmark and shared challenge for visual question answering in gastrointestinal endoscopy**, built around **Kvasir-VQA-x1** and emphasizing **correct answer + explainable reasoning** over endoscopic images rather than text-only medical QA [2508.10869]. More broadly, the **MEDIQA 2019** shared task concerned **medical natural language inference**, **recognizing question entailment**, and **QA re-ranking**, with systems such as **DoubleTransfer** and **Dr.Quad** emphasizing transfer learning, domain adaptation, and contextualized representations rather than multilingual exam benchmarking [1906.04382]; [1907.10136].

Within that landscape, MediQAl’s main significance is methodological rather than merely linguistic. It provides a benchmark where multilinguality, task heterogeneity, and reasoning analysis are jointly operationalized. This suggests that its value lies not only in expanding French medical NLP resources, but also in making model failures more diagnosable: a system can now be examined by answer format, by clinical context, and by the distinction between **Understanding** and **Reasoning** rather than by aggregate accuracy alone.

Source: https://www.emergentmind.com/topics/mediqal