---
title: 'RoMedQA: Romanian Medical QA Benchmarks'
url: https://www.emergentmind.com/topics/romedqa
type: topic
---

# RoMedQA: Romanian Medical QA Benchmarks

RoMedQA denotes Romanian-language medical question answering resources introduced in two distinct research contexts. In "RoQLlama: A Lightweight Romanian Adapted Language Model," the name refers to a dataset of single-choice medical questions in Romanian, derived from advanced biology material used in Romanian medical school entrance examinations [2410.04269]. In "RoMedQA: The First Benchmark for Romanian Medical Question Answering," the same name designates a later, substantially different benchmark comprising question-answer pairs over oncology case summaries from 1,011 cancer patients [2508.16390]. Taken together, these resources mark two complementary directions in Romanian medical QA: exam-style multiple-choice benchmarking and clinically grounded free-form QA over longitudinal patient documentation.

## 1. Terminological scope and benchmark lineage

The term **RoMedQA** is associated with two non-identical datasets. The earlier resource, introduced in the RoQLlama study, contains **4,127 entries** of **single-choice multiple-choice questions** in Romanian, each with **five answer choices** and **only one correct answer per question**. Its stated purpose is to address the underrepresentation of Romanian-language NLP resources in medicine and to support evaluation and adaptation of Romanian LLMs on specialized QA tasks [2410.04269].

The later resource, introduced as a standalone benchmark, is presented as the **first Romanian benchmark for medical question answering** and comprises **102,646 QA pairs** linked to **1,011 real-world cancer patients**. Rather than exam questions, it uses medical case summaries (“epicrises”) and targets clinically oriented answer generation spanning binary, extractive, and reasoning questions [2508.16390].

A common misconception is to treat these two resources as successive versions of a single unchanged dataset. The published descriptions indicate instead that they differ in source material, supervision format, task structure, scale, and intended evaluation regime. This suggests that **RoMedQA** is best understood as a benchmark name applied to two separate Romanian medical QA efforts rather than as a single monolithic corpus.

## 2. Exam-style RoMedQA: design, structure, and curation

In the RoQLlama paper, RoMedQA is a **medical—specifically advanced biology** dataset reflecting **entrance exam material for Romanian medical schools**. The dataset contains **4,127 entries**, each organized as one question with **five answer choices (numbered 1–5)** and **one correct answer**. The answer classes are described as **well balanced**, which supports stable evaluation under a five-way classification regime [2410.04269].

Its collection pipeline integrates heterogeneous document sources from **Romanian medical university admissions**, including **HTML web pages, PDFs, and scanned documents**. Extraction was performed through **web scraping (HTML)**, **PDF parsing and OCR for scanned images**, followed by **manual curation** for low-quality scans or noisy OCR outputs. Every question-answer pair was explicitly checked, and the dataset was sanitized to remove noise so that **no further data pre-processing is necessary before use** [2410.04269].

The vocabulary profile reported through TF-IDF highlights terms such as **“celulă,” “mușchi,”** and **“nerv,”** indicating the dataset’s biological emphasis. The paper gives the standard scoring formula
$$
\text{TF-IDF}(w) = \text{TF}(w) \times \log\left( \frac{N}{\text{DF}(w)} \right),
$$
with $\text{TF}(w)$ denoting term frequency, $N$ the total number of documents, and $\text{DF}(w)$ the number of documents containing $w$ [2410.04269].

The resource is described as the **first Romanian medical QA dataset** and as a benchmark intended for **medical question answering**, **general-purpose Romanian MCQ evaluation**, and **few-shot and zero-shot prompting evaluation of LLMs**. A plausible implication is that its design emphasizes controlled answer spaces and benchmarking regularity more than open-ended clinical discourse.

## 3. Benchmarking role of the exam-style dataset

Within the RoQLlama study, RoMedQA is used as a **primary evaluation task** under **zero-shot prompting**. The reported **test split size** is **831 questions**. The evaluation prompt constrains models to answer by returning **only the digit for the correct option**, and the reported metrics are **Accuracy**, **Macro F1-score**, and **NFI (Not Followed Instruction)** [2410.04269].

The zero-shot results reported in the paper are as follows:

| Model | Accuracy | F1 | NFI |
|---|---:|---:|---:|
| Llama2-7b | 22.77 | 3.60 | 0.00 |
| Llama2-7b-chat | 11.69 | 1.79 | 0.00 |
| RoQLlama-7b | 21.57 | 3.67 | 0.00 |

The study notes that all models performed **well below random-guess accuracy (expected ~20% for 5 choices)**, and interprets this as evidence of the **complexity and specialized nature** of the questions together with the models’ **lack of targeted medical training** [2410.04269]. Because NFI is reported as **0.0%**, the difficulty does not appear to stem from instruction-following failures; instead, it is concentrated in answer selection itself.

The paper also states that RoMedQA **was not used to fine-tune RoQLlama-7b** and was reserved strictly for **benchmarking/assessment**. This separation is methodologically important because it prevents conflation between adaptation gains and in-domain exposure. The low scores further suggest that specialized Romanian medical MCQ competence is not recovered by generic Romanian adaptation alone.

## 4. Oncology RoMedQA: source data and annotation protocol

The later benchmark, introduced in 2025, shifts from admission-style MCQs to patient-centric clinical QA. Its source material consists of medical case summaries (“epicrises”) of **1,011 real-world cancer patients** from **Colțea Clinical Hospital, Bucharest, Romania**, including **796 breast cancer patients** and **215 lung cancer patients** [2508.16390].

The benchmark contains **102,646 QA pairs**, partitioned into **76,416** for breast cancer and **26,230** for lung cancer. The split is performed at the **patient level** with **70% training**, **15% validation**, and **15% test**, so that each subset contains unique patients and associated QA pairs. This design explicitly targets leakage prevention across patient records [2508.16390].

The annotation process is highly manual and clinically specialized. It involved **seven physicians**—including residents and board-certified oncologists or radiotherapists—who collectively spent **about 2,100 work hours**. The workflow is described as **double-layered annotation**: residents annotate and senior oncologists review for accuracy. On **350 shared QA pairs**, the reported inter-annotator agreement is **Cohen’s $\kappa = 0.9562$**, characterized as near-perfect agreement [2508.16390].

Experts first defined **reference questions** for each cancer site—**48 for breast** and **61 for lung cancer**—and each question was then **paraphrased 8–10 times by different physicians for linguistic diversity**, with **two reformulations randomly sampled per patient-question pair**. The annotation guide enforced domain conventions including **TNM staging**, **CTCAE grading**, **standard abbreviations**, and structured fields such as **ECOG status**, **Ki-67 index**, **tumor size**, and **metastases**. The dataset is also described as **anonymized**, with use approved by a local Ethics Committee in line with **GDPR** and the **Helsinki Declaration** [2508.16390].

## 5. QA typology and textual characteristics of the oncology benchmark

Each oncology instance is represented as a tuple of **patient ID, epicrisis, question, answer**. The benchmark defines three question categories: **Binary (Yes/No)**, **Extractive**, and **Reasoning**. The binary category is skewed, with **about two-thirds** of answers being **“Nu”** and about one-third **“Da”**. Extractive questions require answers explicitly present in the epicrisis, whereas reasoning questions require **multi-step inference and synthesis across patient records** [2508.16390].

The textual statistics reported in the paper underscore the long-context character of the task. **Epicrises** range from **207 to 34,247 tokens** with **mean ≈ 7,171 tokens**. **Questions** are short, ranging from **7 to 30 tokens** with **mean ≈ 15**, while **answers** range from **1 to 442 tokens** with **mean ≈ 4** [2508.16390]. This distribution creates a strongly asymmetric QA setting in which short outputs must be derived from lengthy clinical context.

The paper positions the dataset as **one order of magnitude larger than previous Romanian QA datasets**, and stresses that it is **original, clinically grounded content—not translated or synthetic** [2508.16390]. This suggests a benchmark orientation toward genuine clinical abstraction and longitudinal reasoning rather than examination recall. In practical terms, the benchmark tests whether a model can identify sparse, medically relevant evidence within long Romanian case summaries and map it to a concise clinical answer.

## 6. Modeling results and benchmark implications

The oncology benchmark evaluates **RoLLaMA2-7B-Instruct**, **RoMistral-7B-Instruct**, **Phi-4-mini-instruct**, and **LLaMA3-OpenBioLLM-8B**, each in **zero-shot prompting** and **supervised fine-tuning** settings using **LoRA**. Prompting follows a **Question + Epicrisis + Answer (Q+E+A)** structure, and context lengths ranging from **1,024** to **16,384** tokens are explored depending on the model [2508.16390].

Evaluation uses **F1 Score**, **Exact Match (EM)**, **BLEU**, and **METEOR**. The paper reports that **fine-tuned models vastly outperform zero-shot models and heuristic baselines**, and that zero-shot models, including models adapted to Romanian or medicine, **do not generalize** and perform **at or below the simple “majority” baseline** [2508.16390].

An excerpt of the reported test scores is shown below:

| Model (best setting) | F1 | EM | BLEU | METEOR |
|---|---:|---:|---:|---:|
| Phi-4-mini-instruct, fine-tuned, 3,072 tokens | 0.667 | 0.701 | 0.652 | 0.401 |
| RoMistral-7B-Instruct, fine-tuned, 4,096 tokens | 0.596 | 0.604 | 0.555 | 0.359 |
| LLaMA3-OpenBioLLM-8B, fine-tuned, 2,048 tokens | 0.596 | 0.622 | 0.579 | 0.359 |
| RoLLaMA2-7B-Instruct, fine-tuned, 2,048 tokens | 0.594 | 0.615 | 0.578 | 0.355 |
| Majority answer baseline | 0.215 | 0.184 | 0.188 | 0.118 |

The reported **best model** is **fine-tuned Phi-4-mini-instruct at 3,072 token context**, with **F1 = 0.667**. The study also finds that **models using only the first chunk of the epicrisis outperform those using all chunks or longer contexts**, and concludes that **using excessively long contexts does not improve performance despite epicrisis length** [2508.16390]. Binary questions are described as the easiest, while extractive and reasoning questions remain more difficult. This suggests that context selection, not merely context capacity, is a central bottleneck for Romanian clinical QA.

## 7. Significance for Romanian medical NLP

Across both incarnations, RoMedQA addresses a structural gap in Romanian-language medical NLP. The exam-style dataset contributes a **high-quality, noise-inspected resource** for **medical question answering**, **few-shot and zero-shot prompting evaluation**, and Romanian **MCQ evaluation** over specialized biology content [2410.04269]. The oncology benchmark extends this trajectory toward clinically grounded, long-context QA and provides evidence that **both domain-specific and language-specific fine-tuning are essential** for reliable Romanian clinical QA [2508.16390].

The two datasets also instantiate different evaluation philosophies. The MCQ version emphasizes answer-format discipline, balanced answer positions, and specialized exam difficulty. The oncology version emphasizes patient-level splitting, clinician-authored paraphrases, expert review, and multi-metric evaluation over free-form answers. This suggests that the shared name masks a broader research agenda: building Romanian medical QA benchmarks that cover both structured examination settings and realistic clinical documentation.

A further implication is methodological. In the MCQ setting, Romanian adaptation alone was insufficient for strong performance, while in the oncology setting, fine-tuning on the benchmark itself produced large gains over zero-shot inference. Read together, these results support a consistent conclusion: Romanian medical QA remains highly sensitive to the interaction between task format, domain supervision, and language adaptation.

Source: https://www.emergentmind.com/topics/romedqa