---
title: 'PeruMedQA: Spanish Medical QA Benchmark'
url: https://www.emergentmind.com/topics/perumedqa
type: topic
---

# PeruMedQA: Spanish Medical QA Benchmark

Searching arXiv for PeruMedQA and closely related Spanish medical QA benchmarks and systems.
PeruMedQA is a Spanish medical multiple-choice question answering benchmark built from Peru’s official residency and post-residency entrance examinations. It was introduced to evaluate whether high-performing medical large language models transfer to Spanish-language medical questions from a Latin American country with a distinctive epidemiological mix, including chronic noncommunicable diseases alongside infectious, tropical, and neglected diseases [2509.11517]. The benchmark comprises 8,380 questions from examinations publicly released by CONAREME, the Consejo Nacional de Residentado Médico, and it supports both zero-shot evaluation of open medical-domain LLMs and parameter-efficient fine-tuning of a compact medical model [2509.11517]. In the reported study, medgemma-27b-text-it was the strongest off-the-shelf model, while a LoRA-adapted medgemma-4b-it emerged as the strongest sub-10B alternative and, in multiple in-domain settings, rivaled a 70B model [2509.11517].

## 1. Origin, scope, and motivating problem

PeruMedQA was designed to address a specific benchmarking gap: medical LLM evaluation had largely focused on English and on benchmarks derived from the United States or other high-income settings, even as medical deployment was expanding in Latin America [2509.11517]. The central concern was not whether medical LLMs perform well in the abstract, but whether that performance transfers to Spanish-language medical questions from a South American country whose epidemiological profile and exam-writing conventions differ from the settings emphasized in mainstream benchmarks [2509.11517].

The source material comes from CONAREME, Peru’s official body for medical residency selection. The authors downloaded exam PDFs and answer keys for 2018, 2019, 2020, 2022, 2023, 2024, and 2025; 2021 was unavailable on the CONAREME website [2509.11517]. The examinations are all in Spanish and include both specialty-entry exams and subspecialty or fellowship-style exams [2509.11517]. This establishes PeruMedQA as a benchmark grounded in real, high-stakes physician assessment rather than synthetic question generation [2509.11517].

The benchmark contains 8,380 questions spanning 12 medical domains plus the two general specialty exams Test A and Test B [2509.11517]. The named areas visible in the paper are Pathology and Anatomical Pathology, Anesthesiology, General Surgery, Thoracic and Cardiovascular Surgery, Gynecology & Obstetrics, Neurosurgery, Ophthalmology, Pediatrics, Psychiatry, Radiology, Urology, and the general specialty exams Test A and Test B [2509.11517]. In the paper’s nomenclature, Test A and Test B are two equivalent versions of the specialty-entry examination, whereas named specialties such as pediatrics correspond to subspecialty or post-residency examinations [2509.11517].

This domain choice matters because the benchmark reflects what Peruvian exam committees choose to test, including shifts across years such as COVID-19 content and outbreak-driven attention to diseases like malaria [2509.11517]. That contextual specificity is both a strength and a source of bias. It enables evaluation under local epidemiological and linguistic conditions, but it also means that PeruMedQA is not a direct measure of the entirety of clinical competence [2509.11517].

## 2. Dataset construction and representational design

Construction proceeded by writing Python programs to parse the PDFs and extract each question, the answer options, and the correct answer, then saving these into CSV files one exam at a time [2509.11517]. A human manually verified that the extracted correct answers matched the original PDFs [2509.11517]. Only 16 of the 8,380 questions required manual correction, implying a 0.19% extraction error rate for answer keys [2509.11517]. The authors also checked that numeric answer options had not been corrupted by CSV date parsing [2509.11517].

The verification procedure was deliberately limited. The authors did not verify whether the exam questions themselves or the distractor options were medically correct; they verified extraction fidelity only [2509.11517]. This sharply defines the benchmark’s evidentiary status: it is a faithful machine-readable rendering of official exam material, not a medically adjudicated corpus [2509.11517].

A further harmonization step addressed inconsistency in option cardinality. Some exam years had four options (A–D), while others had five (A–E). To enforce a uniform five-choice format, the authors added an extra option, “NA” (“none of the above”), to four-option questions from 2023, 2024, and 2025 [2509.11517]. To preserve Spanish special characters, the final dataset was stored and used as a pickle file [2509.11517].

This harmonization introduces a stated limitation. Adding a fifth option makes four-option questions subtly harder by changing random-choice probability from \(1/4\) to \(1/5\), so direct comparison against published human exam scores is not fair [2509.11517]. Other limitations explicitly acknowledged include the absence of 2021, reliance on open-access official exams only, the benchmark’s focus on physicians entering specialty and subspecialty training rather than medical students or other health professionals, and the lack of annotation distinguishing factual recall from clinical-case reasoning questions [2509.11517].

## 3. Evaluation protocol, prompting, and model set

PeruMedQA was evaluated as a zero-shot multiple-choice task using only open medical-domain LLMs available on Hugging Face [2509.11517]. The model selection emphasized smaller systems under 10B parameters because the authors wanted the findings to be practical for Peru and other resource-constrained settings, while also including two larger models to represent a high-resource “ideal scenario” [2509.11517].

The eight evaluated base models were as follows:

| Model | Parameters | Hugging Face identifier |
|---|---:|---|
| medgemma-4b-it | 4B | `google/medgemma-4b-it` |
| BioMistral-7B-DARE | 7B | `BioMistral/BioMistral-7B-DARE` |
| MediPhi-Instruct | 3.8B | `microsoft/MediPhi-Instruct` |
| Llama3-OpenBioLLM-8B | 8B | `aaditya/Llama3-OpenBioLLM-8B` |
| JSL-MedLlama-3-8B-v2.0 | 8B | `johnsnowlabs/JSL-MedLlama-3-8B-v2.0` |
| meditron-7b | 7B | `epfl-llm/meditron-7b` |
| medgemma-27b-text-it | 27B | `google/medgemma-27b-text-it` |
| Llama3-OpenBioLLM-70B | 70B | `aaditya/Llama3-OpenBioLLM-70B` |

The prompting strategy was task-specific and zero-shot. The system message was: “Eres un asistente médico experto con entrenamiento en Perú.” The user prompt instructed the model to solve the problem step by step internally and end with “Respuesta final:” followed only by the correct option letter, explicitly forbidding visible chain-of-thought [2509.11517]. The paper states that this was done “in accordance with the standard methodology” used in related medical LLM exam evaluations and that it intentionally did not explore more advanced prompting techniques such as chain-of-thought or tree-of-thought [2509.11517].

Evaluation compared the model’s chosen option against the CONAREME answer key [2509.11517]. Because some models hallucinated or failed to follow the required output format, those cases were treated as invalid or missing answers [2509.11517]. The main metric was conditional accuracy over valid responses, described in prose as the number of questions correctly answered by the LLM divided by the number of questions with valid answers [2509.11517]. The authors also computed a model-uniqueness statistic: the percentage of all 8,380 questions answered correctly by a given model while all other models got them wrong [2509.11517].

## 4. Fine-tuning methodology and held-out testing

The study’s second major component was parameter-efficient fine-tuning of medgemma-4b-it using LoRA [2509.11517]. This model was chosen because it was described as the newest and most compact medical LLM available at the time, making it the most attractive target for efficient adaptation [2509.11517]. The paper offers a conceptual characterization of LoRA rather than a formal derivation, stating that it “represents the weights update of a LLM without updating the model,” learning small low-rank matrices that capture the adaptation needed for the new domain [2509.11517].

The train/validation/test split was year-based. All 2025 exam questions were reserved as test data, yielding 1,400 held-out questions [2509.11517]. The remaining pre-2025 data were split 90/10 into training and validation: 6,282 training questions and 698 validation questions [2509.11517]. Training lasted 10 epochs, with learning rate reported as “5-5,” which the paper’s details indicate is almost certainly a typographical rendering of \(5 \times 10^{-5}\) [2509.11517]. The LoRA hyperparameters were rank \(r = 16\), alpha \(= 16\), dropout \(= 0.05\), and target modules `"all-linear"` [2509.11517]. The authors state that they largely followed the medgemma cookbook [2509.11517].

Their custom monitoring metric during training was the same conditional-accuracy definition used at evaluation time [2509.11517]. The paper does not report batch size, optimizer, sequence length, gradient accumulation, quantization setup, exact tokenization settings, wall-clock training time, or GPU model counts [2509.11517]. It does report that all models were executed on the HyPER C3 Community Cloud HPC Cluster at Emory University [2509.11517]. In the limitations section, the authors note that medgemma-4b-it and its fine-tuned version are usable in environments like Google Colab and some consumer-grade GPU machines, whereas larger models are less accessible [2509.11517].

A plausible implication is that PeruMedQA was designed not merely as an evaluation set, but as a practical adaptation substrate for low-resource medical LLM deployment in Spanish-speaking settings. That implication follows from the explicit emphasis on sub-10B models, PEFT, and usability in constrained compute environments [2509.11517].

## 5. Empirical findings

The strongest overall result is that medgemma-27b-text-it was the best model across years, specialties, and subspecialties, and it outperformed even the larger Llama3-OpenBioLLM-70B [2509.11517]. Its headline scores included Psychiatry 2025 at 94.00%, Pathology 2024 at 92.00%, Test B 2020 at 91.11%, Test B 2018 at 91.00%, Pediatrics 2019 at 91.00%, and Test A 2024 at 91.00% [2509.11517]. It also had the highest unique-correct rate: 2.69% of all questions, or 226 of 8,380 [2509.11517].

The fine-tuned medgemma-4b-it was the strongest practical result among smaller models. It substantially outperformed the vanilla 4B model and all other models below 10B, and in many 2024 scenarios it rivaled or beat the 70B model [2509.11517]. The paper explicitly notes ties or wins for the fine-tuned 4B model in Neurosurgery 2024, Anesthesiology 2024, Gynecology 2024, Pediatrics 2024, Radiology 2024, Test A 2024, Thoracic and Cardiovascular Surgery 2024, and Urology 2024 [2509.11517]. This is the clearest evidence that targeted PEFT on PeruMedQA can make a compact model competitive with much larger medical LLMs on in-domain tasks [2509.11517].

At the other end of the performance spectrum, models under 10B parameters mostly remained below 60% accuracy and often much lower, with some exams below 50% or even below 20% [2509.11517]. The strongest sub-10B vanilla model was JSL-MedLlama-3-8B-v2.0, whose peak score was 74.73% on Psychiatry 2023; it also reached 73.91% on Psychiatry 2024 and 71.28% on Psychiatry 2020 [2509.11517]. Apart from psychiatry and a few isolated tasks, however, it mostly stayed in the 40–60% range and suffered a substantial 9.00% invalid-answer rate [2509.11517].

The paper identifies several task-level trends. Psychiatry was the easiest or at least the most favorable domain for many models, including weaker ones [2509.11517]. By contrast, ophthalmology, thoracic and cardiovascular surgery, and other surgery-heavy domains were relatively harder, particularly for smaller models [2509.11517]. Among the 278 questions that no model answered correctly, 20.50% came from 2025 versus 15.10% from 2018; by specialty, the largest shares were Ophthalmology at 12.23%, General Surgery at 11.87%, and Thoracic and Cardiovascular Surgery at 11.15% [2509.11517]. This suggests persistent difficulty in certain surgical domains and a somewhat heavier concentration of universally failed items in more recent exams [2509.11517].

## 6. Instruction-following, invalid outputs, and benchmark limitations

PeruMedQA’s analysis of hallucination and format failure shows that model differences were not only about medical knowledge but also about instruction-following [2509.11517]. Invalid-answer rates varied dramatically. Llama3-OpenBioLLM-70B and the fine-tuned medgemma-4b-it each had 0 invalid outputs out of 8,380, medgemma-27b-text-it had 2, MediPhi-Instruct had 4, vanilla medgemma-4b-it had 12, BioMistral-7B-DARE had 76, Llama3-OpenBioLLM-8B had 416, JSL-MedLlama-3-8B-v2.0 had 755, and meditron-7b had 5,562, corresponding to 66.37% [2509.11517]. The medgemma-4b-it fine-tuning therefore improved not only domain adaptation but also output compliance, reducing invalid outputs from 0.14% to 0.00% [2509.11517].

This phenomenon is methodologically important because the main metric conditions on valid responses. A model with plausible latent knowledge but weak adherence to output constraints can underperform sharply in strict MCQA evaluation [2509.11517]. The meditron-7b result illustrates this most strongly: failure to produce a valid option letter in two-thirds of cases dominates its practical utility [2509.11517].

The benchmark’s limitations are explicit and nontrivial. Because it is derived from official exam content, PeruMedQA reflects examination priorities rather than the full space of medical competence [2509.11517]. It lacks annotation distinguishing factual recall from clinical-case reasoning [2509.11517]. It is tied to Peruvian exam-writing style and topic emphasis [2509.11517]. It omits 2021 and relies only on publicly released exams [2509.11517]. The forced five-option standard changes chance performance for originally four-option questions, making comparison with human exam outcomes unfair [2509.11517].

The paper does not provide calibration studies, confidence estimates, or a formal error taxonomy beyond hallucinations and invalid answers [2509.11517]. It does not analyze answer rationales [2509.11517]. One likely failure mode discussed by the authors is incomplete alignment with Peruvian epidemiology, local clinical emphases, and exam formulation style [2509.11517]. The gains from fine-tuning medgemma-4b-it are interpreted as evidence that base medical LLMs may possess general medical knowledge while lacking sufficient exposure to Peru-specific epidemiological profiles and the specific way Peruvian medical exam questions are asked [2509.11517].

## 7. Position within Spanish and medical QA research

PeruMedQA occupies a distinct position within medical QA research. It is an exam benchmark rather than a patient-facing community QA or regulator-grounded leaflet retrieval system. That distinction becomes clear when it is contrasted with other Spanish or healthcare QA efforts on arXiv.

The paper "Medicines Question Answering System, MeQA" describes what its authors call the first Spanish system capable of answering questions about medicines for human use, grounded in official medicine leaflets and organized as a hybrid retrieval-oriented pipeline rather than an exam benchmark [2111.02760]. MeQA uses regulated leaflet content, entity normalization, section prediction, and sentence extraction, whereas PeruMedQA evaluates zero-shot and fine-tuned medical LLMs on multiple-choice specialist examinations [2111.02760]. The two projects therefore share a Spanish-language medical QA orientation but instantiate different task formulations: document-grounded medicine QA in MeQA versus benchmarked medical MCQA in PeruMedQA [2111.02760][2509.11517].

A different neighboring line is represented by "YaleNLP @ PerAnsSumm 2025: Multi-Perspective Integration via Mixture-of-Agents for Enhanced Healthcare QA Summarization," which studies perspective-aware healthcare answer summarization in community question-answering forums [2504.03932]. That work addresses multi-answer synthesis, perspective extraction, and factuality-aware summarization, not specialist exam answering [2504.03932]. Its relevance to PeruMedQA is therefore architectural rather than direct. This suggests that future PeruMedQA-based systems could combine exam-tuned medical knowledge with perspective-aware evidence synthesis for patient-facing or forum-based settings, but such an integration is not part of the PeruMedQA paper itself [2504.03932][2509.11517].

Within its own declared scope, PeruMedQA supports three principal conclusions. First, medical LLM performance does not transfer uniformly across languages and regions; benchmarking in Spanish and in local epidemiological contexts is necessary [2509.11517]. Second, for Peru and potentially for other Spanish-speaking countries with similar disease burdens and exam styles, medgemma-27b-text-it is the best off-the-shelf open medical model among those tested [2509.11517]. Third, when computational resources are limited, a fine-tuned medgemma-4b-it is a strong practical alternative because it beats all tested medical LLMs under 10B and rivals a 70B model in many settings, though it still does not surpass medgemma-27b-text-it overall and loses more performance on the held-out 2025 exam [2509.11517].

The authors propose several future directions: evaluating more advanced prompting strategies beyond zero-shot, testing larger and possibly proprietary general-purpose models such as GPT-class systems, fine-tuning stronger models like medgemma-27b-text-it if resources permit, prompting models to produce and then evaluating rationales, using more flexible answer extraction so smaller models are not penalized as much for formatting failures, extending evaluation to medical student exams and other health professions, distinguishing factual questions from clinical-case reasoning questions, and monitoring for catastrophic forgetting when specializing smaller LLMs on local datasets [2509.11517]. In that sense, PeruMedQA functions both as a benchmark and as a concrete argument for region-specific, Spanish-language medical model evaluation in the global South [2509.11517].

Source: https://www.emergentmind.com/topics/perumedqa