CUREMED-BENCH: Multilingual Medical Reasoning

• CUREMED-BENCH is a benchmark for open-ended, multilingual medical reasoning that requires a single verifiable answer per instance.
• It comprises 15,774 QA instances across 13 languages, including underrepresented ones, all sourced from MedlinePlus content.
• The evaluation framework decouples clinical correctness and language consistency using metrics like LC and LA for rigorous model assessment.

Searching arXiv for the benchmark and related papers cited in the provided data. CUREMED-BENCH is a multilingual medical reasoning benchmark introduced in the context of the CURE-Med framework for evaluating open-ended clinical reasoning under language variation (Onyame et al., 19 Jan 2026). It consists of 15,774 open-ended question-answer instances spanning 13 languages, including underrepresented languages such as Amharic, Yoruba, Swahili, and Hausa, and is explicitly designed to measure two distinct properties: logical correctness, defined as whether the final answer is clinically correct, and language consistency, defined as whether the final answer is in the language of the query (Onyame et al., 19 Jan 2026). Each item is grounded in MedlinePlus content and is constructed to have a single verifiable answer, with any clinically equivalent paraphrase accepted as correct. In the surrounding literature, the name is easily conflated with two adjacent resources: “CURE,” a clinical passage-ranking retrieval benchmark, and “CURE-Bench,” a benchmark for therapeutic decision-making with tool use; these are related by theme but are distinct datasets with different task formulations (Sheikh et al., 2024).

1. Terminology and benchmark identity

The exact term CUREMED-BENCH appears in the paper “CURE-Med: Curriculum-Informed Reinforcement Learning for Multilingual Medical Reasoning” and denotes a benchmark for multilingual medical reasoning rather than retrieval or tool-augmented therapeutic selection (Onyame et al., 19 Jan 2026). Its core unit is an open-ended reasoning query with a single verifiable answer, and its evaluation protocol separates clinical answer validity from language fidelity.

A recurrent source of confusion is the proximity of three similarly named resources:

Name	Primary task	Distinctive scope
CURE	Passage ranking	Point-of-care clinical retrieval, with English and cross-lingual French/Spanish to English tracks
CUREMED-BENCH	Open-ended reasoning	Multilingual medical reasoning with single verifiable answers in 13 languages
CURE-Bench	Therapeutic decision-making	Multiple-choice and tool-augmented reasoning over a large external tool ecosystem

“CURE,” introduced as “A Dataset for Clinical Understanding & Retrieval Evaluation,” is an ad-hoc retrieval test dataset for passage ranking with 2,000 queries across 10 medical domains and monolingual as well as cross-lingual evaluation conditions (Sheikh et al., 2024). “CURE-Bench,” as discussed in CureAgent, targets therapeutic decision-making and clinically grounded reasoning that requires dynamic evidence use via a large external tool ecosystem and is evaluated primarily through multiple-choice accuracy (Xie et al., 5 Dec 2025). CUREMED-BENCH is neither of these. Its emphasis is instead on open-ended multilingual reasoning, explicit reasoning supervision, and the joint evaluation of answer correctness and output language.

This distinction is methodologically important. A plausible implication is that papers referencing “CUREMed-Bench” are usually engaging with multilingual reasoning and language-fidelity evaluation, whereas work on “CURE” is concerned with retrieval generalization, and work on “CURE-Bench” is concerned with tool use, evidence aggregation, and therapeutic decision-making.

2. Dataset scope, representation, and linguistic coverage

CUREMED-BENCH contains 15,774 open-ended QA instances across 13 languages: Amharic, Bengali, French, Hausa, Hindi, Japanese, Korean, Spanish, Swahili, Thai, Turkish, Vietnamese, and Yoruba (Onyame et al., 19 Jan 2026). The benchmark prominently includes underrepresented languages such as Amharic, Yoruba, Swahili, and Hausa. Its content is sourced from MedlinePlus and spans symptoms, causes, risk factors, diagnostics, treatments, and prevention across common clinical topics and specialties, including infectious disease, cardiology-related symptoms, respiratory complaints, gastrointestinal conditions, and preventive care.

The conceptual schema of each instance is given as

$D = \{(x, r, y^*)\},$

where $x$ is the open-ended prompt in the target language, $r$ is a human-validated reasoning chain, and $y^*$ is the unique clinically correct answer (Onyame et al., 19 Jan 2026). The phrase single verifiable answer means that there is exactly one unambiguous, clinically correct conclusion supported by MedlinePlus; during verification, any clinically equivalent paraphrase of $y^*$ is accepted as correct.

The benchmark therefore combines three representational layers. First, it contains the target-language question $x$. Second, it includes a reasoning trace $r$, which is intended for supervision and analysis rather than direct scoring. Third, it provides the answer $y^*$, which is the object of correctness verification. For training and evaluation with CURE-Med, model outputs are structured with explicit reasoning and answer segments such as <thinking> ... </thinking> and <answer> ... </answer>, but the benchmark scores the final answer for correctness and language (Onyame et al., 19 Jan 2026).

An example entry described in the paper is the French item:

• question: “Une femme de 34 ans présente des douleurs dans l’hypochondre droit avec nausées et vomissements. Avec des antécédents de calculs biliaires, quel est le diagnostic le plus probable ?”
• answer: “Cholécystite aiguë”

This formulation places CUREMED-BENCH in a different regime from multiple-choice medical QA benchmarks. The benchmark is not designed to test option selection; it is designed to test whether a model can produce the clinically correct answer in the correct language under open-ended generation constraints.

3. Construction pipeline and quality control

The source material for CUREMED-BENCH is MedlinePlus, described as a U.S. federal, clinically validated resource (Onyame et al., 19 Jan 2026). The construction pipeline proceeds in three stages.

First, GPT-4o retrieves MedlinePlus content and drafts 4-option, single-answer MCQs in each target language. Each item includes exactly one correct option. Second, a difficulty filtering stage removes trivial items that all three compact LLMs—Qwen2.5-3B, Qwen2.5-7B, and LLaMA-3.1-8B—answer correctly. In the same stage, GPT-4o flags and discards items that are under-specified, ambiguous, or cross-lingually inconsistent. Third, the remaining MCQs are converted to open-ended prompts $x$, and an explicit reasoning chain $r$ plus a free-form ground truth $x$0 are generated, yielding the dataset form $x$1 (Onyame et al., 19 Jan 2026).

Human verification is central to the benchmark’s quality-control regime. Reviewers are native speakers and medical experts, including physicians, advanced medical students, and nursing PhD candidates. They assess clinical correctness, linguistic fidelity, and cultural appropriateness, and they correct translation artifacts or inappropriate content. The work was conducted under an IRB-approved protocol. The reported quality outcome is an average human rating of 4.89/5 across languages, with per-language medical correctness and language quality typically in the 4.6–5.0 range (Onyame et al., 19 Jan 2026).

The paper does not explicitly report a deduplication module, but it states that multi-stage filtering removed trivial or redundant items and cross-lingual inconsistencies. It also notes a specific treatment of code-switching: the provided reasoning traces $x$2 may exhibit controlled code-switching in intermediate steps for training stability, while the final answers are kept in the target language for fidelity and evaluation (Onyame et al., 19 Jan 2026). This design directly anticipates one of the benchmark’s central concerns, namely language drift in multilingual medical reasoning.

A plausible implication is that the benchmark encodes not only task difficulty but also a particular theory of failure: multilingual medical reasoning degrades through both clinical reasoning errors and language-control failures, and these must be measured separately.

4. Evaluation formalism

CUREMED-BENCH is defined by two primary metrics: Language consistency (LC) and Logical correctness (LA) (Onyame et al., 19 Jan 2026). LC measures whether the model’s final answer is in the same language as the input query. The per-example reward is

$x$3

and aggregate language consistency over $x$4 items is

$x$5

LA measures whether the final answer is clinically correct relative to $x$6. The paper uses an LLM-as-a-judge verifier—GPT-4o/4.1—to compare $x$7 to $x$8. At training time, the correctness reward is

$x$9

while the evaluation-time metric is the binary aggregate

$r$0

The framework also defines format compliance for training:

$r$1

and a composite reward

$r$2

with weights 0.65, 0.30, and 0.05, respectively (Onyame et al., 19 Jan 2026).

Two technical properties distinguish this evaluation setup. First, it explicitly decouples clinical validity from language fidelity, so a model can be correct but linguistically inconsistent, or language-faithful but clinically wrong. Second, evaluation is performed on the final answer only, even when the model emits a structured reasoning trace. This makes the benchmark relatively robust to stylistic variation in explanations while retaining strictness in answer validation.

The paper’s practical guidance recommends using a held-out LLM-as-a-judge with temperature 0 and a concise rubric focused on clinical equivalence, and computing LC with either a robust language-ID detector or a language-judge prompt. It also recommends mitigating judge bias by using a verifier different from the model under test (Onyame et al., 19 Jan 2026).

5. Benchmark results and scaling behavior

The paper reports results for 28 models across sizes and domains, including general-purpose models such as Qwen2.5 Instruct, LLaMA-3.x, Gemma, Mistral, Apollo2, and Ministral, and medical-specific models such as MedAlpaca, Meditron, UltraMedical, HuatuoGPT, OpenBioLLM, BioMistral, and MMed-LLaMA (Onyame et al., 19 Jan 2026). Aggregate performance is reported as the mean over 13 languages.

A compact summary of the headline numbers is:

Model	LC	LA
Cure-Med-Qwen2.5-7B	85.21%	54.35%
Cure-Med-Qwen2.5-32B	94.96%	70.04%
HuatuoGPT-o1-8B	67.30%	46.86%

At 7B, Cure-Med-Qwen2.5-7B achieves 85.21% LC and 54.35% LA. At 32B, Cure-Med-Qwen2.5-32B reaches 94.96% LC and 70.04% LA. For context, among the reported baselines, HuatuoGPT-o1-8B reaches 46.86% LA and 67.30% LC, while baselines at $r$3B reach up to approximately 60.80% LA and 79.66% LC. The paper also reports a clear scaling trend: as model size increases from 1.5B to 32B, LC improves from approximately 57.6% to approximately 95%, and LA from approximately 28.3% to approximately 70.0% (Onyame et al., 19 Jan 2026).

The language-wise results emphasize robustness gains in lower-resource settings. At 7B, Amharic improves from 0.95% to 17.14% in LA and from 0.00% to 64.76% in LC, while Yoruba improves from 0.00% to 40.86% in LA and from 0.00% to 77.42% in LC. High-resource languages also improve while maintaining language fidelity; for French, LC improves from 71.43% to 96.43% (Onyame et al., 19 Jan 2026).

The ablation results isolate the contributions of the training recipe built around the benchmark. Naïve multilingual SFT yields small or unstable gains and can reduce logic accuracy at small scales, whereas CURE-Med SFT without RL shows large, consistent gains. At 1.5B, LC rises from 3.84% to 53.67% and LA from 6.20% to 22.97% under the CURE-Med SFT configuration. Subsequent curriculum-aware GRPO further improves both metrics, reaching 57.60% LC and 28.32% LA at 1.5B after RL, and 94.96% LC and 70.04% LA at 32B after RL (Onyame et al., 19 Jan 2026).

These results support a specific empirical claim: the dominant multilingual failure mode is not only lower medical reasoning accuracy in non-English settings, but also language drift. CUREMED-BENCH is designed so that both effects become numerically visible.

6. Relation to the CURE-Med training framework

CUREMED-BENCH functions not only as an evaluation suite but also as the training substrate for the CURE-Med framework, which combines code-switching-aware supervised fine-tuning with Group Relative Policy Optimization (Onyame et al., 19 Jan 2026). In the SFT stage, the training objective is

$r$4

Here, $r$5 may include controlled code-switching to stabilize multi-step reasoning, whereas $r$6 is always in the target language $r$7. In the RL stage, the composite reward is the previously defined

$r$8

The curriculum is organized by language tiers based on baseline difficulty:

• High: French, Japanese, Spanish, Vietnamese
• Medium: Korean, Thai, Turkish, Bengali
• Low: Amharic, Yoruba, Hausa, Hindi, Swahili

Retention-mixing across phases is defined as

$r$9

with $y^*$0 (Onyame et al., 19 Jan 2026).

This makes the benchmark unusually coupled to a methodological program. It is not merely a held-out test set; it supplies target-language prompts, reasoning traces, and verifiable answers in a form suitable for both supervised and reinforcement fine-tuning. The paper’s practical guidance therefore recommends starting with SFT on $y^*$1, then adding curriculum-aware GRPO with the reward

$y^*$2

progressing from high- to low-resource languages while retaining approximately 0.85 of earlier data at each curriculum stage (Onyame et al., 19 Jan 2026).

The benchmark also exhibits OOD generalization when used for training. The paper reports transfer gains on MMedBench, MedExpQA, and MedQA, especially at smaller scales; examples include MMedBench 1.5B: 6.00 → 24.00 in French, MedExpQA 1.5B: English 1.40 → 44.80, and MedQA Chinese 1.5B: 21.00 → 59.50 (Onyame et al., 19 Jan 2026). This suggests that the benchmark’s supervision is not limited to narrow memorization of MedlinePlus formulations.

7. Positioning, limitations, and common misconceptions

Relative to prior medical QA datasets, CUREMED-BENCH is characterized in the paper by three elements: open-ended reasoning with a single verifiable answer, broad multilingual coverage across 13 languages, and reasoning supervision through $y^*$3 (Onyame et al., 19 Jan 2026). The paper contrasts it with multilingual or medical benchmarks that often have fewer languages, focus on multiple-choice questions, or lack explicit reasoning traces and language-fidelity evaluation.

Several misconceptions are clarified by the surrounding literature. First, CUREMED-BENCH is not the retrieval benchmark called CURE, which evaluates passage ranking over English passages with English, French, and Spanish queries (Sheikh et al., 2024). Second, it is not the therapeutic decision benchmark called CURE-Bench, which is evaluated with accuracy on multiple-choice tasks and tool-augmented reasoning over a large external tool ecosystem (Xie et al., 5 Dec 2025). Third, despite its clinical grounding, it is not a clinical decision-support system; the paper explicitly states that it is intended for research benchmarking only, contains no patient data, and should not be used alone in real care settings (Onyame et al., 19 Jan 2026).

The paper also identifies several limitations. Coverage is constrained by the breadth of MedlinePlus topics in different languages, and difficulty may vary by language. The benchmark is restricted to single-visit, text-only questions, with no longitudinal care, multimodal inputs, or complex multi-answer cases. Parts of the construction and verification pipeline rely on proprietary APIs, specifically GPT-4o/4.1, which may affect cost and reproducibility. Although reasoning traces can contain controlled code-switching for training stability, evaluation focuses on the final answer only, which narrows the benchmark’s direct assessment of internal reasoning quality (Onyame et al., 19 Jan 2026).

These constraints define the benchmark’s proper interpretation. CUREMED-BENCH is best understood as a clinically grounded testbed for multilingual answer validity and language fidelity under open-ended reasoning, rather than as a complete simulation of clinical practice. Its significance lies in making multilingual medical reasoning failures measurable in a form that exposes both logic errors and language drift.