RoMedQA: Romanian Medical QA Benchmarks

• RoMedQA is a pair of Romanian medical QA resources, one with exam-style multiple-choice questions and the other with clinical oncology case summaries.
• The exam-style dataset comprises 4,127 curated items with five answer choices per question, enabling controlled evaluation of language model performance.
• The oncology benchmark provides 102,646 QA pairs from 1,011 cancer patient records, annotated by experts to support robust clinical reasoning assessments.

RoMedQA denotes Romanian-language medical question answering resources introduced in two distinct research contexts. In "RoQLlama: A Lightweight Romanian Adapted LLM," the name refers to a dataset of single-choice medical questions in Romanian, derived from advanced biology material used in Romanian medical school entrance examinations (Dima et al., 2024). In "RoMedQA: The First Benchmark for Romanian Medical Question Answering," the same name designates a later, substantially different benchmark comprising question-answer pairs over oncology case summaries from 1,011 cancer patients (Rogoz et al., 22 Aug 2025). Taken together, these resources mark two complementary directions in Romanian medical QA: exam-style multiple-choice benchmarking and clinically grounded free-form QA over longitudinal patient documentation.

1. Terminological scope and benchmark lineage

The term RoMedQA is associated with two non-identical datasets. The earlier resource, introduced in the RoQLlama study, contains 4,127 entries of single-choice multiple-choice questions in Romanian, each with five answer choices and only one correct answer per question. Its stated purpose is to address the underrepresentation of Romanian-language NLP resources in medicine and to support evaluation and adaptation of Romanian LLMs on specialized QA tasks (Dima et al., 2024).

The later resource, introduced as a standalone benchmark, is presented as the first Romanian benchmark for medical question answering and comprises 102,646 QA pairs linked to 1,011 real-world cancer patients. Rather than exam questions, it uses medical case summaries (“epicrises”) and targets clinically oriented answer generation spanning binary, extractive, and reasoning questions (Rogoz et al., 22 Aug 2025).

A common misconception is to treat these two resources as successive versions of a single unchanged dataset. The published descriptions indicate instead that they differ in source material, supervision format, task structure, scale, and intended evaluation regime. This suggests that RoMedQA is best understood as a benchmark name applied to two separate Romanian medical QA efforts rather than as a single monolithic corpus.

2. Exam-style RoMedQA: design, structure, and curation

In the RoQLlama paper, RoMedQA is a medical—specifically advanced biology dataset reflecting entrance exam material for Romanian medical schools. The dataset contains 4,127 entries, each organized as one question with five answer choices (numbered 1–5) and one correct answer. The answer classes are described as well balanced, which supports stable evaluation under a five-way classification regime (Dima et al., 2024).

Its collection pipeline integrates heterogeneous document sources from Romanian medical university admissions, including HTML web pages, PDFs, and scanned documents. Extraction was performed through web scraping (HTML), PDF parsing and OCR for scanned images, followed by manual curation for low-quality scans or noisy OCR outputs. Every question-answer pair was explicitly checked, and the dataset was sanitized to remove noise so that no further data pre-processing is necessary before use (Dima et al., 2024).

The vocabulary profile reported through TF-IDF highlights terms such as “celulă,” “mușchi,” and “nerv,” indicating the dataset’s biological emphasis. The paper gives the standard scoring formula

$$\text{TF-IDF}(w) = \text{TF}(w) \times \log\left( \frac{N}{\text{DF}(w)} \right),$$

with $\text{TF}(w)$ denoting term frequency, $N$ the total number of documents, and $\text{DF}(w)$ the number of documents containing $w$ (Dima et al., 2024).

The resource is described as the first Romanian medical QA dataset and as a benchmark intended for medical question answering, general-purpose Romanian MCQ evaluation, and few-shot and zero-shot prompting evaluation of LLMs. A plausible implication is that its design emphasizes controlled answer spaces and benchmarking regularity more than open-ended clinical discourse.

3. Benchmarking role of the exam-style dataset

Within the RoQLlama study, RoMedQA is used as a primary evaluation task under zero-shot prompting. The reported test split size is 831 questions. The evaluation prompt constrains models to answer by returning only the digit for the correct option, and the reported metrics are Accuracy, Macro F1-score, and NFI (Not Followed Instruction) (Dima et al., 2024).

The zero-shot results reported in the paper are as follows:

Model	Accuracy	F1	NFI
Llama2-7b	22.77	3.60	0.00
Llama2-7b-chat	11.69	1.79	0.00
RoQLlama-7b	21.57	3.67	0.00

The study notes that all models performed well below random-guess accuracy (expected ~20% for 5 choices), and interprets this as evidence of the complexity and specialized nature of the questions together with the models’ lack of targeted medical training (Dima et al., 2024). Because NFI is reported as 0.0%, the difficulty does not appear to stem from instruction-following failures; instead, it is concentrated in answer selection itself.

The paper also states that RoMedQA was not used to fine-tune RoQLlama-7b and was reserved strictly for benchmarking/assessment. This separation is methodologically important because it prevents conflation between adaptation gains and in-domain exposure. The low scores further suggest that specialized Romanian medical MCQ competence is not recovered by generic Romanian adaptation alone.

4. Oncology RoMedQA: source data and annotation protocol

The later benchmark, introduced in 2025, shifts from admission-style MCQs to patient-centric clinical QA. Its source material consists of medical case summaries (“epicrises”) of 1,011 real-world cancer patients from Colțea Clinical Hospital, Bucharest, Romania, including 796 breast cancer patients and 215 lung cancer patients (Rogoz et al., 22 Aug 2025).

The benchmark contains 102,646 QA pairs, partitioned into 76,416 for breast cancer and 26,230 for lung cancer. The split is performed at the patient level with 70% training, 15% validation, and 15% test, so that each subset contains unique patients and associated QA pairs. This design explicitly targets leakage prevention across patient records (Rogoz et al., 22 Aug 2025).

The annotation process is highly manual and clinically specialized. It involved seven physicians—including residents and board-certified oncologists or radiotherapists—who collectively spent about 2,100 work hours. The workflow is described as double-layered annotation: residents annotate and senior oncologists review for accuracy. On 350 shared QA pairs, the reported inter-annotator agreement is Cohen’s $\kappa = 0.9562$, characterized as near-perfect agreement (Rogoz et al., 22 Aug 2025).

Experts first defined reference questions for each cancer site—48 for breast and 61 for lung cancer—and each question was then paraphrased 8–10 times by different physicians for linguistic diversity, with two reformulations randomly sampled per patient-question pair. The annotation guide enforced domain conventions including TNM staging, CTCAE grading, standard abbreviations, and structured fields such as ECOG status, Ki-67 index, tumor size, and metastases. The dataset is also described as anonymized, with use approved by a local Ethics Committee in line with GDPR and the Helsinki Declaration (Rogoz et al., 22 Aug 2025).

5. QA typology and textual characteristics of the oncology benchmark

Each oncology instance is represented as a tuple of patient ID, epicrisis, question, answer. The benchmark defines three question categories: Binary (Yes/No), Extractive, and Reasoning. The binary category is skewed, with about two-thirds of answers being “Nu” and about one-third “Da”. Extractive questions require answers explicitly present in the epicrisis, whereas reasoning questions require multi-step inference and synthesis across patient records (Rogoz et al., 22 Aug 2025).

The textual statistics reported in the paper underscore the long-context character of the task. Epicrises range from 207 to 34,247 tokens with mean ≈ 7,171 tokens. Questions are short, ranging from 7 to 30 tokens with mean ≈ 15, while answers range from 1 to 442 tokens with mean ≈ 4 (Rogoz et al., 22 Aug 2025). This distribution creates a strongly asymmetric QA setting in which short outputs must be derived from lengthy clinical context.

The paper positions the dataset as one order of magnitude larger than previous Romanian QA datasets, and stresses that it is original, clinically grounded content—not translated or synthetic (Rogoz et al., 22 Aug 2025). This suggests a benchmark orientation toward genuine clinical abstraction and longitudinal reasoning rather than examination recall. In practical terms, the benchmark tests whether a model can identify sparse, medically relevant evidence within long Romanian case summaries and map it to a concise clinical answer.

6. Modeling results and benchmark implications

The oncology benchmark evaluates RoLLaMA2-7B-Instruct, RoMistral-7B-Instruct, Phi-4-mini-instruct, and LLaMA3-OpenBioLLM-8B, each in zero-shot prompting and supervised fine-tuning settings using LoRA. Prompting follows a Question + Epicrisis + Answer (Q+E+A) structure, and context lengths ranging from 1,024 to 16,384 tokens are explored depending on the model (Rogoz et al., 22 Aug 2025).

Evaluation uses F1 Score, Exact Match (EM), BLEU, and METEOR. The paper reports that fine-tuned models vastly outperform zero-shot models and heuristic baselines, and that zero-shot models, including models adapted to Romanian or medicine, do not generalize and perform at or below the simple “majority” baseline (Rogoz et al., 22 Aug 2025).

An excerpt of the reported test scores is shown below:

Model (best setting)	F1	EM	BLEU	METEOR
Phi-4-mini-instruct, fine-tuned, 3,072 tokens	0.667	0.701	0.652	0.401
RoMistral-7B-Instruct, fine-tuned, 4,096 tokens	0.596	0.604	0.555	0.359
LLaMA3-OpenBioLLM-8B, fine-tuned, 2,048 tokens	0.596	0.622	0.579	0.359
RoLLaMA2-7B-Instruct, fine-tuned, 2,048 tokens	0.594	0.615	0.578	0.355
Majority answer baseline	0.215	0.184	0.188	0.118

The reported best model is fine-tuned Phi-4-mini-instruct at 3,072 token context, with F1 = 0.667. The study also finds that models using only the first chunk of the epicrisis outperform those using all chunks or longer contexts, and concludes that using excessively long contexts does not improve performance despite epicrisis length (Rogoz et al., 22 Aug 2025). Binary questions are described as the easiest, while extractive and reasoning questions remain more difficult. This suggests that context selection, not merely context capacity, is a central bottleneck for Romanian clinical QA.

7. Significance for Romanian medical NLP

Across both incarnations, RoMedQA addresses a structural gap in Romanian-language medical NLP. The exam-style dataset contributes a high-quality, noise-inspected resource for medical question answering, few-shot and zero-shot prompting evaluation, and Romanian MCQ evaluation over specialized biology content (Dima et al., 2024). The oncology benchmark extends this trajectory toward clinically grounded, long-context QA and provides evidence that both domain-specific and language-specific fine-tuning are essential for reliable Romanian clinical QA (Rogoz et al., 22 Aug 2025).

The two datasets also instantiate different evaluation philosophies. The MCQ version emphasizes answer-format discipline, balanced answer positions, and specialized exam difficulty. The oncology version emphasizes patient-level splitting, clinician-authored paraphrases, expert review, and multi-metric evaluation over free-form answers. This suggests that the shared name masks a broader research agenda: building Romanian medical QA benchmarks that cover both structured examination settings and realistic clinical documentation.

A further implication is methodological. In the MCQ setting, Romanian adaptation alone was insufficient for strong performance, while in the oncology setting, fine-tuning on the benchmark itself produced large gains over zero-shot inference. Read together, these results support a consistent conclusion: Romanian medical QA remains highly sensitive to the interaction between task format, domain supervision, and language adaptation.