---
title: 'CUREMED-BENCH: Multilingual Medical Reasoning'
url: https://www.emergentmind.com/topics/curemed-bench
type: topic
---

# CUREMED-BENCH: Multilingual Medical Reasoning

Searching arXiv for the benchmark and related papers cited in the provided data.
CUREMED-BENCH is a multilingual medical reasoning benchmark introduced in the context of the CURE-Med framework for evaluating open-ended clinical reasoning under language variation [2601.13262]. It consists of 15,774 open-ended question-answer instances spanning 13 languages, including underrepresented languages such as Amharic, Yoruba, Swahili, and Hausa, and is explicitly designed to measure two distinct properties: logical correctness, defined as whether the final answer is clinically correct, and language consistency, defined as whether the final answer is in the language of the query [2601.13262]. Each item is grounded in MedlinePlus content and is constructed to have a single verifiable answer, with any clinically equivalent paraphrase accepted as correct. In the surrounding literature, the name is easily conflated with two adjacent resources: “CURE,” a clinical passage-ranking retrieval benchmark, and “CURE-Bench,” a benchmark for therapeutic decision-making with tool use; these are related by theme but are distinct datasets with different task formulations [2412.06954].

## 1. Terminology and benchmark identity

The exact term **CUREMED-BENCH** appears in the paper “CURE-Med: Curriculum-Informed Reinforcement Learning for Multilingual Medical Reasoning” and denotes a benchmark for **multilingual medical reasoning** rather than retrieval or tool-augmented therapeutic selection [2601.13262]. Its core unit is an open-ended reasoning query with a single verifiable answer, and its evaluation protocol separates clinical answer validity from language fidelity.

A recurrent source of confusion is the proximity of three similarly named resources:

| Name | Primary task | Distinctive scope |
|---|---|---|
| CURE | Passage ranking | Point-of-care clinical retrieval, with English and cross-lingual French/Spanish to English tracks |
| CUREMED-BENCH | Open-ended reasoning | Multilingual medical reasoning with single verifiable answers in 13 languages |
| CURE-Bench | Therapeutic decision-making | Multiple-choice and tool-augmented reasoning over a large external tool ecosystem |

“CURE,” introduced as “A Dataset for Clinical Understanding & Retrieval Evaluation,” is an ad-hoc retrieval test dataset for passage ranking with 2,000 queries across 10 medical domains and monolingual as well as cross-lingual evaluation conditions [2412.06954]. “CURE-Bench,” as discussed in CureAgent, targets therapeutic decision-making and clinically grounded reasoning that requires dynamic evidence use via a large external tool ecosystem and is evaluated primarily through multiple-choice accuracy [2512.05576]. CUREMED-BENCH is neither of these. Its emphasis is instead on open-ended multilingual reasoning, explicit reasoning supervision, and the joint evaluation of answer correctness and output language.

This distinction is methodologically important. A plausible implication is that papers referencing “CUREMed-Bench” are usually engaging with multilingual reasoning and language-fidelity evaluation, whereas work on “CURE” is concerned with retrieval generalization, and work on “CURE-Bench” is concerned with tool use, evidence aggregation, and therapeutic decision-making.

## 2. Dataset scope, representation, and linguistic coverage

CUREMED-BENCH contains **15,774 open-ended QA instances** across **13 languages**: Amharic, Bengali, French, Hausa, Hindi, Japanese, Korean, Spanish, Swahili, Thai, Turkish, Vietnamese, and Yoruba [2601.13262]. The benchmark prominently includes underrepresented languages such as Amharic, Yoruba, Swahili, and Hausa. Its content is sourced from **MedlinePlus** and spans symptoms, causes, risk factors, diagnostics, treatments, and prevention across common clinical topics and specialties, including infectious disease, cardiology-related symptoms, respiratory complaints, gastrointestinal conditions, and preventive care.

The conceptual schema of each instance is given as
$$
D = \{(x, r, y^*)\},
$$
where $x$ is the open-ended prompt in the target language, $r$ is a human-validated reasoning chain, and $y^*$ is the unique clinically correct answer [2601.13262]. The phrase **single verifiable answer** means that there is exactly one unambiguous, clinically correct conclusion supported by MedlinePlus; during verification, any clinically equivalent paraphrase of $y^*$ is accepted as correct.

The benchmark therefore combines three representational layers. First, it contains the target-language question $x$. Second, it includes a reasoning trace $r$, which is intended for supervision and analysis rather than direct scoring. Third, it provides the answer $y^*$, which is the object of correctness verification. For training and evaluation with CURE-Med, model outputs are structured with explicit reasoning and answer segments such as `<thinking> ... </thinking>` and `<answer> ... </answer>`, but the benchmark scores the **final answer** for correctness and language [2601.13262].

An example entry described in the paper is the French item:
- question: “Une femme de 34 ans présente des douleurs dans l’hypochondre droit avec nausées et vomissements. Avec des antécédents de calculs biliaires, quel est le diagnostic le plus probable ?”
- answer: “Cholécystite aiguë”

This formulation places CUREMED-BENCH in a different regime from multiple-choice medical QA benchmarks. The benchmark is not designed to test option selection; it is designed to test whether a model can produce the clinically correct answer in the correct language under open-ended generation constraints.

## 3. Construction pipeline and quality control

The source material for CUREMED-BENCH is **MedlinePlus**, described as a U.S. federal, clinically validated resource [2601.13262]. The construction pipeline proceeds in three stages.

First, **GPT-4o** retrieves MedlinePlus content and drafts **4-option, single-answer MCQs** in each target language. Each item includes exactly one correct option. Second, a **difficulty filtering** stage removes trivial items that all three compact LLMs—**Qwen2.5-3B, Qwen2.5-7B, and LLaMA-3.1-8B**—answer correctly. In the same stage, GPT-4o flags and discards items that are under-specified, ambiguous, or cross-lingually inconsistent. Third, the remaining MCQs are converted to **open-ended prompts** $x$, and an explicit reasoning chain $r$ plus a free-form ground truth $y^*$ are generated, yielding the dataset form $D=\{(x,r,y^*)\}$ [2601.13262].

Human verification is central to the benchmark’s quality-control regime. Reviewers are **native speakers and medical experts**, including physicians, advanced medical students, and nursing PhD candidates. They assess **clinical correctness**, **linguistic fidelity**, and **cultural appropriateness**, and they correct translation artifacts or inappropriate content. The work was conducted under an **IRB-approved protocol**. The reported quality outcome is an **average human rating of 4.89/5 across languages**, with per-language medical correctness and language quality typically in the **4.6–5.0** range [2601.13262].

The paper does not explicitly report a deduplication module, but it states that multi-stage filtering removed trivial or redundant items and cross-lingual inconsistencies. It also notes a specific treatment of code-switching: the provided reasoning traces $r$ may exhibit **controlled code-switching** in intermediate steps for training stability, while the **final answers are kept in the target language** for fidelity and evaluation [2601.13262]. This design directly anticipates one of the benchmark’s central concerns, namely language drift in multilingual medical reasoning.

A plausible implication is that the benchmark encodes not only task difficulty but also a particular theory of failure: multilingual medical reasoning degrades through both **clinical reasoning errors** and **language-control failures**, and these must be measured separately.

## 4. Evaluation formalism

CUREMED-BENCH is defined by two primary metrics: **Language consistency (LC)** and **Logical correctness (LA)** [2601.13262]. LC measures whether the model’s final answer is in the same language as the input query. The per-example reward is

$$
R_{\mathrm{lang}}(y \mid \ell) =
\begin{cases}
1 & \text{if the language of } y \text{ matches } \ell \\
0 & \text{otherwise}
\end{cases}
$$

and aggregate language consistency over $N$ items is

$$
LC = \frac{1}{N} \sum_i 1\{\mathrm{lang}(y_i)=\ell_i\}.
$$

LA measures whether the final answer is clinically correct relative to $y^*$. The paper uses an **LLM-as-a-judge verifier**—**GPT-4o/4.1**—to compare $y$ to $y^*$. At training time, the correctness reward is

$$
R_{\mathrm{acc}}(y \mid x, y^*) = v_{\mathrm{acc}}(x,y,y^*) \in [0,1],
$$

while the evaluation-time metric is the binary aggregate

$$
LA = \frac{1}{N} \sum_i 1\{\mathrm{verifier}(x_i, y_i, y^*_i)=\mathrm{True}\}.
$$

The framework also defines **format compliance** for training:
$$
R_{\mathrm{fmt}}(y) =
\begin{cases}
1 & \text{if required structure is followed} \\
0 & \text{otherwise}
\end{cases}
$$

and a composite reward
$$
R(y \mid x, y^*, \ell) = \lambda_{\mathrm{acc}} R_{\mathrm{acc}} + \lambda_{\mathrm{lang}} R_{\mathrm{lang}} + \lambda_{\mathrm{fmt}} R_{\mathrm{fmt}},
$$
with weights **0.65**, **0.30**, and **0.05**, respectively [2601.13262].

Two technical properties distinguish this evaluation setup. First, it explicitly decouples **clinical validity** from **language fidelity**, so a model can be correct but linguistically inconsistent, or language-faithful but clinically wrong. Second, evaluation is performed on the **final answer only**, even when the model emits a structured reasoning trace. This makes the benchmark relatively robust to stylistic variation in explanations while retaining strictness in answer validation.

The paper’s practical guidance recommends using a held-out LLM-as-a-judge with **temperature 0** and a concise rubric focused on **clinical equivalence**, and computing LC with either a robust language-ID detector or a language-judge prompt. It also recommends mitigating judge bias by using a verifier different from the model under test [2601.13262].

## 5. Benchmark results and scaling behavior

The paper reports results for **28 models across sizes and domains**, including general-purpose models such as Qwen2.5 Instruct, LLaMA-3.x, Gemma, Mistral, Apollo2, and Ministral, and medical-specific models such as MedAlpaca, Meditron, UltraMedical, HuatuoGPT, OpenBioLLM, BioMistral, and MMed-LLaMA [2601.13262]. Aggregate performance is reported as the mean over 13 languages.

A compact summary of the headline numbers is:

| Model | LC | LA |
|---|---:|---:|
| Cure-Med-Qwen2.5-7B | 85.21% | 54.35% |
| Cure-Med-Qwen2.5-32B | 94.96% | 70.04% |
| HuatuoGPT-o1-8B | 67.30% | 46.86% |

At **7B**, Cure-Med-Qwen2.5-7B achieves **85.21% LC** and **54.35% LA**. At **32B**, Cure-Med-Qwen2.5-32B reaches **94.96% LC** and **70.04% LA**. For context, among the reported baselines, **HuatuoGPT-o1-8B** reaches **46.86% LA** and **67.30% LC**, while baselines at **$\ge 70$B** reach up to approximately **60.80% LA** and **79.66% LC**. The paper also reports a clear scaling trend: as model size increases from **1.5B to 32B**, LC improves from approximately **57.6% to approximately 95%**, and LA from approximately **28.3% to approximately 70.0%** [2601.13262].

The language-wise results emphasize robustness gains in lower-resource settings. At **7B**, **Amharic** improves from **0.95% to 17.14%** in LA and from **0.00% to 64.76%** in LC, while **Yoruba** improves from **0.00% to 40.86%** in LA and from **0.00% to 77.42%** in LC. High-resource languages also improve while maintaining language fidelity; for **French**, LC improves from **71.43% to 96.43%** [2601.13262].

The ablation results isolate the contributions of the training recipe built around the benchmark. **Naïve multilingual SFT** yields small or unstable gains and can reduce logic accuracy at small scales, whereas **CURE-Med SFT without RL** shows large, consistent gains. At **1.5B**, LC rises from **3.84% to 53.67%** and LA from **6.20% to 22.97%** under the CURE-Med SFT configuration. Subsequent **curriculum-aware GRPO** further improves both metrics, reaching **57.60% LC** and **28.32% LA** at **1.5B** after RL, and **94.96% LC** and **70.04% LA** at **32B** after RL [2601.13262].

These results support a specific empirical claim: the dominant multilingual failure mode is not only lower medical reasoning accuracy in non-English settings, but also language drift. CUREMED-BENCH is designed so that both effects become numerically visible.

## 6. Relation to the CURE-Med training framework

CUREMED-BENCH functions not only as an evaluation suite but also as the training substrate for the **CURE-Med** framework, which combines **code-switching-aware supervised fine-tuning** with **Group Relative Policy Optimization** [2601.13262]. In the SFT stage, the training objective is

$$
L_{\mathrm{SFT}} = -\log p_\theta(r, y^* \mid x).
$$

Here, $r$ may include controlled code-switching to stabilize multi-step reasoning, whereas $y^*$ is always in the target language $\ell$. In the RL stage, the composite reward is the previously defined
$$
R = \lambda_{\mathrm{acc}} R_{\mathrm{acc}} + \lambda_{\mathrm{lang}} R_{\mathrm{lang}} + \lambda_{\mathrm{fmt}} R_{\mathrm{fmt}}.
$$

The curriculum is organized by language tiers based on baseline difficulty:
- **High**: French, Japanese, Spanish, Vietnamese
- **Medium**: Korean, Thai, Turkish, Bengali
- **Low**: Amharic, Yoruba, Hausa, Hindi, Swahili

Retention-mixing across phases is defined as
$$
D_i = \alpha D_{i-1} + (1-\alpha) D_{L_i},
$$
with **$\alpha = 0.85$** [2601.13262].

This makes the benchmark unusually coupled to a methodological program. It is not merely a held-out test set; it supplies target-language prompts, reasoning traces, and verifiable answers in a form suitable for both supervised and reinforcement fine-tuning. The paper’s practical guidance therefore recommends starting with SFT on $(x,r,y^*)$, then adding curriculum-aware GRPO with the reward
$$
R = 0.65R_{\mathrm{acc}} + 0.30R_{\mathrm{lang}} + 0.05R_{\mathrm{fmt}},
$$
progressing from high- to low-resource languages while retaining approximately **0.85** of earlier data at each curriculum stage [2601.13262].

The benchmark also exhibits **OOD generalization** when used for training. The paper reports transfer gains on **MMedBench**, **MedExpQA**, and **MedQA**, especially at smaller scales; examples include **MMedBench 1.5B: 6.00 → 24.00 in French**, **MedExpQA 1.5B: English 1.40 → 44.80**, and **MedQA Chinese 1.5B: 21.00 → 59.50** [2601.13262]. This suggests that the benchmark’s supervision is not limited to narrow memorization of MedlinePlus formulations.

## 7. Positioning, limitations, and common misconceptions

Relative to prior medical QA datasets, CUREMED-BENCH is characterized in the paper by three elements: **open-ended reasoning with a single verifiable answer**, **broad multilingual coverage across 13 languages**, and **reasoning supervision through $r$** [2601.13262]. The paper contrasts it with multilingual or medical benchmarks that often have fewer languages, focus on multiple-choice questions, or lack explicit reasoning traces and language-fidelity evaluation.

Several misconceptions are clarified by the surrounding literature. First, CUREMED-BENCH is **not** the retrieval benchmark called **CURE**, which evaluates passage ranking over English passages with English, French, and Spanish queries [2412.06954]. Second, it is **not** the therapeutic decision benchmark called **CURE-Bench**, which is evaluated with accuracy on multiple-choice tasks and tool-augmented reasoning over a large external tool ecosystem [2512.05576]. Third, despite its clinical grounding, it is **not a clinical decision-support system**; the paper explicitly states that it is intended for **research benchmarking only**, contains **no patient data**, and should not be used alone in real care settings [2601.13262].

The paper also identifies several limitations. Coverage is constrained by the breadth of **MedlinePlus** topics in different languages, and difficulty may vary by language. The benchmark is restricted to **single-visit, text-only questions**, with no longitudinal care, multimodal inputs, or complex multi-answer cases. Parts of the construction and verification pipeline rely on **proprietary APIs**, specifically **GPT-4o/4.1**, which may affect cost and reproducibility. Although reasoning traces can contain controlled code-switching for training stability, evaluation focuses on the final answer only, which narrows the benchmark’s direct assessment of internal reasoning quality [2601.13262].

These constraints define the benchmark’s proper interpretation. CUREMED-BENCH is best understood as a clinically grounded testbed for **multilingual answer validity and language fidelity** under open-ended reasoning, rather than as a complete simulation of clinical practice. Its significance lies in making multilingual medical reasoning failures measurable in a form that exposes both logic errors and language drift.

Source: https://www.emergentmind.com/topics/curemed-bench