MEDEC Clinical Error Benchmark
- MEDEC is a benchmark dataset for medical error detection and correction in clinical notes, incorporating both synthetic and authentic data.
- It comprises 3,848 texts from two domains with five distinct error types, including Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism.
- The evaluation framework spans error flagging, sentence extraction, and correction generation, revealing performance differences between LLMs and human experts.
MEDEC is a benchmark for medical error detection and correction in clinical notes. It was introduced as the first publicly available benchmark explicitly designed for this purpose and consists of 3,848 English clinical texts covering five error types—Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism—including 488 diagnosis support notes from three University of Washington hospital systems that were not previously seen by any LLM. The benchmark supports error flagging, erroneous-sentence identification, and correction generation, and it served as the basis for the MEDIQA-CORR shared task. Reported evaluations with recent LLMs and with two medical doctors indicate that the benchmark is challenging and that doctors still outperform LLMs overall on detection and correction (Abacha et al., 2024).
1. Scope and nomenclature
In current biomedical ML literature, the acronym “MEDEC” appears in more than one benchmark context. The term most commonly denotes the clinical-note benchmark for medical error detection and correction introduced in late 2024 (Abacha et al., 2024). However, “MEDEC” is also used for “Meta-Dataset Clinical,” a TCGA-derived benchmark of 174 gene-expression classification tasks (Samiei et al., 2019). A separate emergency-care benchmark, “MDS-ED,” has at times been informally conflated with MEDEC, but its official name is “Multimodal Decision Support in the Emergency Department,” and the paper explicitly states that it is not named MEDEC (Alcaraz et al., 2024).
Within clinical NLP, the medical-error MEDEC benchmark occupies a specific position: it evaluates whether a model can inspect a multi-sentence clinical narrative for correctness and internal consistency, identify the erroneous sentence when an error exists, and propose a medically sound correction. This distinguishes it from medical QA datasets that primarily assess answer retrieval or exam-style reasoning (Abacha et al., 2024).
2. Corpus composition and provenance
MEDEC contains 3,848 clinical texts in English and is organized around two complementary source domains. The larger component, MEDEC-MS, comprises 3,360 texts derived from MedQA narratives by injecting correct or incorrect answers into the scenario text. The second component, MEDEC-UW, contains 488 diagnosis support notes from three University of Washington hospital systems—Harborview Medical Center, UW Medical Center, and Seattle Cancer Care Alliance—spanning 2009–2021 (Abacha et al., 2024).
The split structure reflects these two domains. The benchmark reports 2,189 training texts, 734 validation texts, and 925 test texts. Class balance is also reported: the training split contains 970 texts without errors and 1,219 with errors; validation contains 335 without errors and 399 with errors; test contains 450 without errors and 475 with errors (Abacha et al., 2024). In the error-detection setting used by later prompt-optimization work, the MS subset is split into train 2,189, validation 574, and test 597, whereas the UW subset provides validation 160 and test 328, with no UW training split (Myles et al., 25 Feb 2026).
The two domains are deliberately heterogeneous. MEDEC-MS is synthetic and exam-like, while MEDEC-UW consists of real, de-identified clinical notes with authentic documentation style. This domain contrast is central to the benchmark’s role as a robustness test: later studies treat UW as an out-of-distribution setting and report performance drops for many models relative to MS (Myles et al., 25 Feb 2026).
3. Error ontology and task design
MEDEC covers five clinically salient error classes chosen after analysis of frequent board-exam topics (Abacha et al., 2024).
| Error type | Benchmark definition |
|---|---|
| Diagnosis | The stated diagnosis is inaccurate |
| Management | The next step in management is inaccurate |
| Treatment | The recommended treatment is inaccurate |
| Pharmacotherapy | The recommended drug therapy is inaccurate |
| Causal Organism | The indicated pathogen is inaccurate |
The benchmark defines a three-subtask pipeline. In Subtask A, Error Flag Prediction, the input is a multi-sentence clinical note and the output is a binary flag: 0 for no error and 1 for error present. In Subtask B, Error Sentence Extraction, the output is if no error exists and otherwise the Sentence ID containing the error. In Subtask C, Error Correction Generation, the output is “NA” if no error exists and otherwise the corrected sentence text (Abacha et al., 2024).
Input notes are line-numbered, with each line formatted as sentence_id | sentence_text. In the evaluation setting emphasized by subsequent work, each erroneous narrative contains exactly one medical error sentence, while other narratives are entirely correct (Myles et al., 25 Feb 2026). The official prompting protocol P#1 instructs the model to return CORRECT if no medical error is present; otherwise it must return the erroneous sentence ID followed by a corrected version of that sentence. P#2 retains the same structure but adds a few-shot example from training (Abacha et al., 2024).
The benchmark reports Accuracy for the detection and sentence-extraction subtasks, and Recall per error type on examples with error flag . For correction, it uses ROUGE-1, BLEURT-20, and BERTScore, with the aggregate metric defined as
Correction metrics are computed only when both system and reference corrections are non-NA (Abacha et al., 2024). The specific test-set counts used for per-type reporting are Diagnosis 174, Management 168, Treatment 58, Pharmacotherapy 57, and Causal Organism 18 (Abacha et al., 2024).
4. Construction, annotation, and privacy controls
MEDEC uses two distinct creation pipelines. Method #1 generates MEDEC-MS from MedQA multiple-choice clinical narratives. Four medical annotators injected either the correct or a wrong answer into each scenario to create a correct version and an incorrect version, and they manually verified faithfulness of the rewrites to the original scenario before selecting one correct and one incorrect version for the dataset (Abacha et al., 2024).
Method #2 constructs MEDEC-UW from real diagnosis support notes. The source pool consisted of 17,453 notes, from which 488 were selected; 244 were modified to include errors and the remainder were kept correct. QuickUMLS was used to surface candidate entities, after which annotators chose spans or created new spans and labeled them with one of the five error types. Erroneous alternatives were inserted using annotator-crafted replacements or SNOMED- and LLM-based suggestions that did not use the input text itself. Each injected error was required to contradict at least two other parts of the note, and annotators documented justifications (Abacha et al., 2024).
The human annotation pipeline was medically informed throughout. Eight medical annotators contributed across methods, and four medical students handled UW error injection and review. The UW subset was de-identified with Philter, then independently reviewed by two annotators, with a third adjudicating discrepancies. Two medical doctors later performed expert evaluation on the test set for detection and correction (Abacha et al., 2024).
Privacy and access controls differ by subset. MEDEC-MS is publicly available, whereas MEDEC-UW requires a Data Use Agreement. This partition simultaneously improves realism and reduces contamination risk, since the UW notes were not previously seen by any LLM (Abacha et al., 2024). The paper does not report inter-annotator agreement metrics such as Cohen’s .
5. Benchmark results and methodological implications
The benchmark paper evaluates a range of LLMs under the standardized prompt formats. For overall detection, Claude 3.5 Sonnet achieved the best reported accuracy, with Error Flag Accuracy $0.7016$ and Error Sentence Accuracy $0.6562$. o1-mini achieved the second-best error-flag accuracy at $0.6908$. For correction generation, o1-preview achieved the best AggregateScore at $0.6976$, while GPT-4 with P#2 was second at $0.6387$ (Abacha et al., 2024).
Human expert performance remained stronger overall. Doctor #1 achieved Error Flag Accuracy $0.7961$, Sentence Accuracy 0, and AggregateScore 1. Doctor #2 achieved Error Flag Accuracy 2, Sentence Accuracy 3, and AggregateScore 4, which was the best overall correction quality in the study (Abacha et al., 2024). This suggests that detection accuracy and correction generation quality are partially dissociable capabilities: Claude 3.5 Sonnet led in detection but had the lowest correction scores, whereas Doctor #2 did not have the highest flagging accuracy yet achieved the strongest correction metric.
Subset-specific results reinforce the role of distribution shift. Claude 3.5 Sonnet scored 5 flag accuracy on MS and 6 on UW, while o1-preview scored 7 on MS but only 8 on UW. The authors interpret o1-preview’s profile as high recall accompanied by lower overall detection accuracy, suggesting precision issues and over-prediction of errors (Abacha et al., 2024). Later work on prompt optimization similarly describes UW as more authentic clinical prose and treats the MS-to-UW transfer gap as a core benchmark challenge (Myles et al., 25 Feb 2026).
Per-error-type reporting shows substantial heterogeneity. For o1-preview, recall on notes with errors reached 9 for Diagnosis, 0 for Management, 1 for Treatment, 2 for Pharmacotherapy, and 3 for Causal Organism at the flagging stage; sentence-level recall was correspondingly lower for some categories, notably Management at 4 (Abacha et al., 2024). The benchmark paper also emphasizes limitations of current automatic correction metrics: clinically equivalent synonyms and explanatory corrections can be penalized by ROUGE-1, BLEURT, and BERTScore.
6. Access, reuse, and relation to newer benchmarks
MEDEC is hosted at https://github.com/abachaa/MEDEC, and the evaluation scripts used for MEDIQA-CORR are available at https://github.com/abachaa/MEDIQA-CORR-2024/tree/main/evaluation. The paper states that the MEDEC-MS subset is openly available, whereas MEDEC-UW is available under a Data Use Agreement; license details are not specified in the paper and are to be checked in the repository (Abacha et al., 2024).
The benchmark has already functioned as a substrate for methodological research. A later study on prompt optimization used the official MEDEC P#1 protocol for the error-detection task and reported that GEPA improved combined accuracy from 5 to 6 for GPT-5 and from 7 to 8 for Qwen3-32B, with evaluation on both MS-test and UW-test and strict separation between optimization and held-out testing (Myles et al., 25 Feb 2026). In that setting, conservative prompting—explicitly excluding stylistic, formatting, grammar, and acceptable-practice variations from the definition of “error”—was reported to improve specificity on MEDEC (Myles et al., 25 Feb 2026).
MEDEC also serves as a reference point for newer multilingual resources. MedErrBench positions itself relative to the monolingual MEDEC benchmark, expanding the taxonomy from five to ten error types and extending coverage to English, Arabic, and Chinese, with sentence-level localization, correction targets, difficulty labels, reasoning-type annotations, and “important clinical words” (Ma et al., 5 Feb 2026). That progression highlights both the importance and the limits of the original MEDEC design: it is English-only, contains a modest total of 3,848 notes, and covers five error types, but it established a practical and auditable benchmark for detect–localize–correct evaluation in clinical text (Abacha et al., 2024).
The continued need for disambiguation remains important. In genomics, “MEDEC” can refer to the TCGA Meta-Dataset Clinical benchmark rather than clinical-note error detection (Samiei et al., 2019). In emergency-care decision support, the relevant benchmark is MDS-ED, not MEDEC (Alcaraz et al., 2024). Within clinical-note validation, however, MEDEC has become the canonical reference dataset for benchmarking medical error detection and correction in English clinical narratives (Abacha et al., 2024).