MedRECT: Cross-Lingual Medical Error Benchmark
- MedRECT is a cross-lingual benchmark defining error detection, localization, and correction in clinical narratives to enhance safe medical reasoning.
- It decomposes error handling into three tasks evaluated with metrics like binary F1, exact-match accuracy, ROUGE-1, BERTScore, and BLEURT.
- The resource pairs MedRECT-ja and MedRECT-en to enable cross-lingual comparisons, offering insights into model performance and safety in medical text handling.
MedRECT is a cross-lingual benchmark for medical error handling in clinical free-text narratives, introduced in "MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts" (Iwase et al., 1 Nov 2025). It evaluates whether LLMs can determine whether a clinical text is erroneous, identify the sentence containing the error, and generate a medically accurate correction. The benchmark comprises MedRECT-ja and MedRECT-en, covering Japanese and English respectively, and was designed to address three stated gaps: the need to assess safety-critical reasoning beyond multiple-choice examinations, the absence of standardized Japanese resources for medical error correction, and the lack of scalable, reproducible pipelines for constructing such benchmarks across languages.
1. Conceptual scope and motivation
MedRECT was introduced in response to a safety problem in medical LLM deployment: strong performance on MCQA benchmarks such as licensing-style examinations does not establish that a model can safely detect and repair errors embedded in realistic clinical narratives (Iwase et al., 1 Nov 2025). The benchmark therefore targets error handling rather than answer selection. Its underlying premise is that medical LLMs may produce plausible outputs via flawed reasoning and may reproduce human cognitive biases, including anchoring and confirmation bias. In this setting, the relevant capability is not merely recall of medical facts, but structured recognition and correction of unsafe statements in context.
The benchmark is explicitly cross-lingual. English-only resources, including MEDEC, leave unresolved whether clinical reasoning transfers across languages and medical cultures. MedRECT addresses that question by pairing a newly constructed Japanese resource with an aligned English counterpart. This design makes it possible to measure bilingual performance, quantify transfer gaps, and test whether reasoning-oriented models remain robust when the language changes.
Within the paper’s framing, MedRECT advances medical LLM safety by decomposing error handling into granular subtasks and by exposing strengths and failure modes that would remain obscured in aggregate exam scores. A plausible implication is that the benchmark is intended less as a general language-evaluation suite than as a safety-oriented stress test for clinically situated reasoning.
2. Task formulation and evaluation protocol
MedRECT formulates medical error handling as three progressively evaluated subtasks (Iwase et al., 1 Nov 2025).
For error detection, the input is a clinical text segmented into sentences, written as , and the model predicts . This is a binary classification problem over the full narrative.
For error localization, evaluation is restricted to texts with a ground-truth error. The model receives the full text and must predict the sentence index containing the error. The benchmark uses single-sentence selection with exact sentence-index matching; only samples with one gold error sentence are included, and there is no multi-sentence IoU-style scoring.
For error correction, the model must output a corrected sentence for the erroneous sentence , with the requirement that the corrected sentence be medically accurate and aligned to the original error location.
The prompting protocol is deliberately constrained. The evaluation uses zero-shot instructions in both languages: if no error is present, the model must output CORRECT; if an error is present, it must output sentence_number: corrected_sentence. Explanations are disallowed. This setup reduces confounding from free-form rationales and forces a single structured output that jointly supports detection, localization, and correction.
The benchmark reports different metrics for each subtask. Error detection is scored with binary F1. Error localization is scored with exact-match accuracy on the sentence index. Error correction is evaluated using ROUGE-1 F-score, BERTScore F1, BLEURT, and their arithmetic average, with language-specific tokenization and configuration choices: MeCab for Japanese ROUGE tokenization, whitespace tokenization for English, microsoft/deberta-xlarge-mnli for BERTScore with Japanese and English configurations, and BLEURT-20 for BLEURT.
3. Dataset construction and corpus composition
MedRECT uses different construction pipelines for Japanese and English, while aligning the final resources as closely as possible (Iwase et al., 1 Nov 2025). The Japanese benchmark is synthesized from the Japanese Medical Licensing Examinations, whereas the English benchmark is adapted from MEDEC.
| Corpus | Texts | Error / No error |
|---|---|---|
| MedRECT-ja | 663 | 367 / 296 |
| MedRECT-en | 458 | 243 / 215 |
MedRECT-ja is built from JMLE 2024 and 2025. The authors begin with 287 clinical case questions, excluding image-based, calculation, and underlined-text questions. In the first step, two LLMs—DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507—convert MCQA items into clinical texts. Each answer choice is embedded into the scenario, so correct choices yield CORRECT samples and incorrect choices yield ERROR samples. Errors are labeled into clinical domains including history taking, physical findings, test interpretation, diagnosis, monitoring/management, medication selection, medication dosage, and procedures/intervention. This stage produces 2,792 synthesized candidates.
The second step applies quality filtering by model consensus. Eleven validation models attempt error detection and sentence extraction. For CORRECT samples, retention requires detection accuracy among the 11 models to lie between 1/11 and 7/11. For ERROR samples, retention requires sentence extraction accuracy between 1/11 and 7/11 and a detection–extraction accuracy gap no greater than 3/11. This reduces the pool from 2,792 to 1,057 samples. The third step removes duplicates generated from the same question-choice pairs while balancing synthesis-model representation, reducing 1,057 samples to 720. The fourth step uses Gemini 2.5 Pro as an LLM-as-a-Judge filter over five binary dimensions: ambiguous_error, extra_elements, multiple_errors, numerical_error, and synthesis_consistency_error. Any positive flag triggers exclusion. The resulting retention is 663 samples, or 92.1% of the post-deduplication set. The primary exclusion reasons are synthesis_consistency_error (27), multiple_errors (21), extra_elements (9), ambiguous_error (3), and numerical_error (1).
MedRECT-en is derived from the MEDEC MS Subset Test, which contains 597 samples originally constructed by medical experts from MedQA with manual error injection. The same LLM-as-a-Judge process is applied, but the flags extra_elements and synthesis_consistency_error are replaced with unrealistic_scenario and inconsistent_context to fit MEDEC’s format and provenance. This yields 458 retained samples, corresponding to 76.7%.
The two language subsets have comparable error/no-error balance, approximately 55:45. Their error-type distributions differ substantially. In MedRECT-ja, diagnosis, monitoring/management, and physical findings account for 21.0%, 21.5%, and 19.6% respectively, indicating a relatively broad spread across clinical reasoning categories. In MedRECT-en, diagnosis and medication selection dominate at 40.3% and 28.8%, with much smaller proportions for physical findings and history taking. This suggests that cross-lingual comparisons in MedRECT are not only language comparisons but also comparisons across somewhat different error-profile distributions.
4. Model evaluation and cross-lingual findings
The benchmark evaluates nine contemporary LLM families spanning proprietary, open-weight, and reasoning-oriented systems (Iwase et al., 1 Nov 2025). The reasoning group includes GPT-5, o3, Claude Sonnet 4, DeepSeek-R1-0528, gpt-oss-120b, gpt-oss-20b, and Qwen3-32B in think mode. Non-reasoning baselines include GPT-4.1, DeepSeek-V3-0324, and Qwen3-32B in no-think mode. Proprietary and DeepSeek models are accessed via OpenRouter, while open-weight models are run locally.
A central empirical result is that reasoning models substantially outperform standard architectures. The controlled comparison within the same architecture family is especially direct: on MedRECT-ja, Qwen3-32B in think mode reaches error-detection F1 of 0.723 versus 0.637 in no-think mode, a 13.5% relative improvement. On sentence extraction, the same comparison is 72.5% versus 48.0%, a 51.0% relative improvement. Because this comparison holds architecture constant, the reported gain is attributed to reasoning mode rather than model scale or training corpus differences.
Cross-lingual evaluation reveals persistent English-to-Japanese performance gaps. For error detection, many reasoning models show gaps of roughly 5 to 10 percentage points: o3 scores 0.852 on English versus 0.764 on Japanese, and GPT-5 scores 0.818 versus 0.758. Claude Sonnet 4 is an exception, with 0.784 on English and 0.795 on Japanese. Sentence-extraction gaps are generally larger and more model-dependent: GPT-5 records 96.3% on English versus 83.7% on Japanese, o3 records 87.7% versus 71.4%, and Qwen3-32B think records 83.5% versus 72.5%. The paper’s interpretation is that reasoning models exhibit smaller detection disparities and stronger cross-lingual robustness than non-reasoning baselines; GPT-4.1, for example, shows a larger detection gap of 0.789 on English versus 0.658 on Japanese.
The benchmark therefore functions not merely as a ranking device for models but as a diagnostic instrument. It separates detection from localization and correction, making it possible to observe where performance deteriorates across languages and where reasoning-oriented behavior appears to mitigate the deterioration.
5. Fine-tuning, human comparison, and failure analysis
The paper also studies targeted LoRA fine-tuning on Qwen3-32B, using rank 64, , and learning rate (Iwase et al., 1 Nov 2025). The bilingual training data consist of 5,538 synthesized Japanese samples from JMLE 2018–2023 and 2,439 English samples from MEDEC training and validation. Reasoning chains are generated by DeepSeek-R1-0528 with prompts engineered to prevent meta-reference leakage.
The fine-tuning results are asymmetric but consistently favorable for correction. On MedRECT-ja, the averaged error-correction score rises from 0.549 to 0.627, a gain of +0.078; error-detection F1 improves from 0.723 to 0.743; and sentence extraction improves from 72.5% to 81.5%. On MedRECT-en, the averaged error-correction score rises from 0.550 to 0.718, a gain of +0.168; sentence extraction improves from 83.5% to 90.9%; and detection F1 slightly declines from 0.740 to 0.728. The paper characterizes these outcomes as preserving reasoning capability, since the think variant remains superior to the no-think variant after fine-tuning.
The benchmark’s human comparison is reported on the original MEDEC benchmark of 597 samples. Fine-tuned Qwen3-32B + LoRA in think mode achieves sentence extraction accuracy of 90.6%, compared with 76.7% for physician #1 and 64.6% for physician #2. For the averaged error-correction score, the model reaches 0.714, compared with 0.491 and 0.678. For error-detection accuracy, however, the model reaches 62.0%, below physician #1 at 81.3% and physician #2 at 68.9%. The stated explanation is that the model is more sensitive and produces more false positives. Statistical significance is not reported, so the comparison is descriptive rather than inferential.
The qualitative analysis identifies recurring error categories that models detect or correct relatively well: diagnostic misattribution, procedural misjudgment, and medication selection. Reported examples include correction of a neonatal meningitis organism from Pseudomonas aeruginosa to GBS (Streptococcus agalactiae), correction of an inappropriate gastrostomy recommendation in end-of-life care to non-invasive nutritional management, and replacement of doxycycline with itraconazole in an English example. The most challenging category is medication dosage, for which sentence-extraction performance is lowest across models; the paper notes that some models show very low accuracy, including 27.3% for Qwen3-32B + LoRA on dosage. History taking and physical findings are also difficult because they require contextual interpretation and empathy; the paper reports a large gap between Qwen think and no-think for history taking, 68.1% versus 36.2%.
The main failure modes include false positives on CORRECT texts, over-correction that adds unnecessary findings or alters accurate statements, temporal or management inconsistencies, and misjudgment of contraindications or procedural appropriateness given prognosis. Qualitatively, the fine-tuned model is reported to produce more patient-centered and empathetic corrections in sensitive scenarios and to reduce over-correction by correctly outputting CORRECT when no error exists.
6. Reproducibility, limitations, and naming ambiguity
MedRECT is accompanied by explicit reproducibility resources: datasets, code, prompts, and evaluation scripts are available at https://github.com/pfnet-research/medrect, and the fine-tuned model is released as pfnet/Preferred-MedRECT-32B on Hugging Face (Iwase et al., 1 Nov 2025). The paper also documents the construction steps, retention statistics, evaluation prompt, and language-specific metric configurations. Because the benchmark is built from licensing examination materials and benchmark corpora rather than identifiable patient records, it contains no PHI.
The paper identifies several limitations. The Japanese resource is constrained by the availability of suitable JMLE clinical cases: 287 usable questions ultimately yield 663 retained samples. The injected or synthesized errors are systematic, but they may not represent the full diversity of real-world clinical mistakes. The construction pipeline depends on specific LLMs for synthesis and filtering, including DeepSeek-R1-0528, Qwen3-235B-A22B, Gemini 2.5 Pro, and 11 validation models; this may introduce biases, even though the authors attempt mitigation through multi-model consensus and mid-difficulty selection. The correction metrics—ROUGE, BERTScore, and BLEURT—measure textual overlap and semantic similarity rather than full clinical adequacy. The benchmark is text-only and does not cover images or tables. The paper therefore states that MedRECT is an evaluation resource, not a clinical decision tool, and that any deployment should include human oversight, calibration against false positives, robust monitoring, and attention to local guidelines and language or cultural context.
The term “MedRECT” is not used uniformly across the supplied literature. In "MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT Report Generation" (Kyung et al., 29 Jun 2025), the authors explicitly state that the correct model name is MedRegion-CT, and that “MedRECT” should be treated only as an informal shorthand if encountered elsewhere. In "Learnable Retrieval Enhanced Visual-Text Alignment and Fusion for Radiology Report Generation" (Zhou et al., 10 Jul 2025), “MedRECT” is used more generically to denote retrieval-enhanced cross-modal transformer paradigms for radiology report generation, with REVTAF described as belonging conceptually to that design space rather than bearing the name itself. This suggests a terminological ambiguity in adjacent medical-AI literature. In the strict bibliographic sense, however, MedRECT refers most precisely to the bilingual benchmark for medical error detection, localization, and correction introduced in (Iwase et al., 1 Nov 2025).