PeruMedQA: Spanish Medical QA Benchmark
- PeruMedQA is a Spanish medical multiple-choice QA benchmark derived from official Peruvian residency exams, reflecting local epidemiological and linguistic characteristics.
- The dataset was constructed via automated PDF parsing with a 0.19% correction rate and harmonized answer options to ensure consistent evaluation across 12 medical domains.
- Empirical findings highlight medgemma-27b-text-it as the top off-the-shelf model and a LoRA-adapted medgemma-4b-it as a competitive, resource-efficient alternative.
Searching arXiv for PeruMedQA and closely related Spanish medical QA benchmarks and systems. PeruMedQA is a Spanish medical multiple-choice question answering benchmark built from Peru’s official residency and post-residency entrance examinations. It was introduced to evaluate whether high-performing medical LLMs transfer to Spanish-language medical questions from a Latin American country with a distinctive epidemiological mix, including chronic noncommunicable diseases alongside infectious, tropical, and neglected diseases (Carrillo-Larco et al., 15 Sep 2025). The benchmark comprises 8,380 questions from examinations publicly released by CONAREME, the Consejo Nacional de Residentado Médico, and it supports both zero-shot evaluation of open medical-domain LLMs and parameter-efficient fine-tuning of a compact medical model (Carrillo-Larco et al., 15 Sep 2025). In the reported study, medgemma-27b-text-it was the strongest off-the-shelf model, while a LoRA-adapted medgemma-4b-it emerged as the strongest sub-10B alternative and, in multiple in-domain settings, rivaled a 70B model (Carrillo-Larco et al., 15 Sep 2025).
1. Origin, scope, and motivating problem
PeruMedQA was designed to address a specific benchmarking gap: medical LLM evaluation had largely focused on English and on benchmarks derived from the United States or other high-income settings, even as medical deployment was expanding in Latin America (Carrillo-Larco et al., 15 Sep 2025). The central concern was not whether medical LLMs perform well in the abstract, but whether that performance transfers to Spanish-language medical questions from a South American country whose epidemiological profile and exam-writing conventions differ from the settings emphasized in mainstream benchmarks (Carrillo-Larco et al., 15 Sep 2025).
The source material comes from CONAREME, Peru’s official body for medical residency selection. The authors downloaded exam PDFs and answer keys for 2018, 2019, 2020, 2022, 2023, 2024, and 2025; 2021 was unavailable on the CONAREME website (Carrillo-Larco et al., 15 Sep 2025). The examinations are all in Spanish and include both specialty-entry exams and subspecialty or fellowship-style exams (Carrillo-Larco et al., 15 Sep 2025). This establishes PeruMedQA as a benchmark grounded in real, high-stakes physician assessment rather than synthetic question generation (Carrillo-Larco et al., 15 Sep 2025).
The benchmark contains 8,380 questions spanning 12 medical domains plus the two general specialty exams Test A and Test B (Carrillo-Larco et al., 15 Sep 2025). The named areas visible in the paper are Pathology and Anatomical Pathology, Anesthesiology, General Surgery, Thoracic and Cardiovascular Surgery, Gynecology & Obstetrics, Neurosurgery, Ophthalmology, Pediatrics, Psychiatry, Radiology, Urology, and the general specialty exams Test A and Test B (Carrillo-Larco et al., 15 Sep 2025). In the paper’s nomenclature, Test A and Test B are two equivalent versions of the specialty-entry examination, whereas named specialties such as pediatrics correspond to subspecialty or post-residency examinations (Carrillo-Larco et al., 15 Sep 2025).
This domain choice matters because the benchmark reflects what Peruvian exam committees choose to test, including shifts across years such as COVID-19 content and outbreak-driven attention to diseases like malaria (Carrillo-Larco et al., 15 Sep 2025). That contextual specificity is both a strength and a source of bias. It enables evaluation under local epidemiological and linguistic conditions, but it also means that PeruMedQA is not a direct measure of the entirety of clinical competence (Carrillo-Larco et al., 15 Sep 2025).
2. Dataset construction and representational design
Construction proceeded by writing Python programs to parse the PDFs and extract each question, the answer options, and the correct answer, then saving these into CSV files one exam at a time (Carrillo-Larco et al., 15 Sep 2025). A human manually verified that the extracted correct answers matched the original PDFs (Carrillo-Larco et al., 15 Sep 2025). Only 16 of the 8,380 questions required manual correction, implying a 0.19% extraction error rate for answer keys (Carrillo-Larco et al., 15 Sep 2025). The authors also checked that numeric answer options had not been corrupted by CSV date parsing (Carrillo-Larco et al., 15 Sep 2025).
The verification procedure was deliberately limited. The authors did not verify whether the exam questions themselves or the distractor options were medically correct; they verified extraction fidelity only (Carrillo-Larco et al., 15 Sep 2025). This sharply defines the benchmark’s evidentiary status: it is a faithful machine-readable rendering of official exam material, not a medically adjudicated corpus (Carrillo-Larco et al., 15 Sep 2025).
A further harmonization step addressed inconsistency in option cardinality. Some exam years had four options (A–D), while others had five (A–E). To enforce a uniform five-choice format, the authors added an extra option, “NA” (“none of the above”), to four-option questions from 2023, 2024, and 2025 (Carrillo-Larco et al., 15 Sep 2025). To preserve Spanish special characters, the final dataset was stored and used as a pickle file (Carrillo-Larco et al., 15 Sep 2025).
This harmonization introduces a stated limitation. Adding a fifth option makes four-option questions subtly harder by changing random-choice probability from $1/4$ to $1/5$, so direct comparison against published human exam scores is not fair (Carrillo-Larco et al., 15 Sep 2025). Other limitations explicitly acknowledged include the absence of 2021, reliance on open-access official exams only, the benchmark’s focus on physicians entering specialty and subspecialty training rather than medical students or other health professionals, and the lack of annotation distinguishing factual recall from clinical-case reasoning questions (Carrillo-Larco et al., 15 Sep 2025).
3. Evaluation protocol, prompting, and model set
PeruMedQA was evaluated as a zero-shot multiple-choice task using only open medical-domain LLMs available on Hugging Face (Carrillo-Larco et al., 15 Sep 2025). The model selection emphasized smaller systems under 10B parameters because the authors wanted the findings to be practical for Peru and other resource-constrained settings, while also including two larger models to represent a high-resource “ideal scenario” (Carrillo-Larco et al., 15 Sep 2025).
The eight evaluated base models were as follows:
| Model | Parameters | Hugging Face identifier |
|---|---|---|
| medgemma-4b-it | 4B | [google](https://www.emergentmind.com/topics/service-weaver-google)/medgemma-4b-it |
| BioMistral-7B-DARE | 7B | BioMistral/BioMistral-7B-DARE |
| MediPhi-Instruct | 3.8B | microsoft/MediPhi-Instruct |
| Llama3-OpenBioLLM-8B | 8B | aaditya/Llama3-OpenBioLLM-8B |
| JSL-MedLlama-3-8B-v2.0 | 8B | johnsnowlabs/JSL-MedLlama-3-8B-v2.0 |
| meditron-7b | 7B | epfl-LLM/meditron-7b |
| medgemma-27b-text-it | 27B | google/medgemma-27b-text-it |
| Llama3-OpenBioLLM-70B | 70B | aaditya/Llama3-OpenBioLLM-70B |
The prompting strategy was task-specific and zero-shot. The system message was: “Eres un asistente médico experto con entrenamiento en Perú.” The user prompt instructed the model to solve the problem step by step internally and end with “Respuesta final:” followed only by the correct option letter, explicitly forbidding visible chain-of-thought (Carrillo-Larco et al., 15 Sep 2025). The paper states that this was done “in accordance with the standard methodology” used in related medical LLM exam evaluations and that it intentionally did not explore more advanced prompting techniques such as chain-of-thought or tree-of-thought (Carrillo-Larco et al., 15 Sep 2025).
Evaluation compared the model’s chosen option against the CONAREME answer key (Carrillo-Larco et al., 15 Sep 2025). Because some models hallucinated or failed to follow the required output format, those cases were treated as invalid or missing answers (Carrillo-Larco et al., 15 Sep 2025). The main metric was conditional accuracy over valid responses, described in prose as the number of questions correctly answered by the LLM divided by the number of questions with valid answers (Carrillo-Larco et al., 15 Sep 2025). The authors also computed a model-uniqueness statistic: the percentage of all 8,380 questions answered correctly by a given model while all other models got them wrong (Carrillo-Larco et al., 15 Sep 2025).
4. Fine-tuning methodology and held-out testing
The study’s second major component was parameter-efficient fine-tuning of medgemma-4b-it using LoRA (Carrillo-Larco et al., 15 Sep 2025). This model was chosen because it was described as the newest and most compact medical LLM available at the time, making it the most attractive target for efficient adaptation (Carrillo-Larco et al., 15 Sep 2025). The paper offers a conceptual characterization of LoRA rather than a formal derivation, stating that it “represents the weights update of a LLM without updating the model,” learning small low-rank matrices that capture the adaptation needed for the new domain (Carrillo-Larco et al., 15 Sep 2025).
The train/validation/test split was year-based. All 2025 exam questions were reserved as test data, yielding 1,400 held-out questions (Carrillo-Larco et al., 15 Sep 2025). The remaining pre-2025 data were split 90/10 into training and validation: 6,282 training questions and 698 validation questions (Carrillo-Larco et al., 15 Sep 2025). Training lasted 10 epochs, with learning rate reported as “5-5,” which the paper’s details indicate is almost certainly a typographical rendering of (Carrillo-Larco et al., 15 Sep 2025). The LoRA hyperparameters were rank , alpha , dropout , and target modules "all-linear" (Carrillo-Larco et al., 15 Sep 2025). The authors state that they largely followed the medgemma cookbook (Carrillo-Larco et al., 15 Sep 2025).
Their custom monitoring metric during training was the same conditional-accuracy definition used at evaluation time (Carrillo-Larco et al., 15 Sep 2025). The paper does not report batch size, optimizer, sequence length, gradient accumulation, quantization setup, exact tokenization settings, wall-clock training time, or GPU model counts (Carrillo-Larco et al., 15 Sep 2025). It does report that all models were executed on the HyPER C3 Community Cloud HPC Cluster at Emory University (Carrillo-Larco et al., 15 Sep 2025). In the limitations section, the authors note that medgemma-4b-it and its fine-tuned version are usable in environments like Google Colab and some consumer-grade GPU machines, whereas larger models are less accessible (Carrillo-Larco et al., 15 Sep 2025).
A plausible implication is that PeruMedQA was designed not merely as an evaluation set, but as a practical adaptation substrate for low-resource medical LLM deployment in Spanish-speaking settings. That implication follows from the explicit emphasis on sub-10B models, PEFT, and usability in constrained compute environments (Carrillo-Larco et al., 15 Sep 2025).
5. Empirical findings
The strongest overall result is that medgemma-27b-text-it was the best model across years, specialties, and subspecialties, and it outperformed even the larger Llama3-OpenBioLLM-70B (Carrillo-Larco et al., 15 Sep 2025). Its headline scores included Psychiatry 2025 at 94.00%, Pathology 2024 at 92.00%, Test B 2020 at 91.11%, Test B 2018 at 91.00%, Pediatrics 2019 at 91.00%, and Test A 2024 at 91.00% (Carrillo-Larco et al., 15 Sep 2025). It also had the highest unique-correct rate: 2.69% of all questions, or 226 of 8,380 (Carrillo-Larco et al., 15 Sep 2025).
The fine-tuned medgemma-4b-it was the strongest practical result among smaller models. It substantially outperformed the vanilla 4B model and all other models below 10B, and in many 2024 scenarios it rivaled or beat the 70B model (Carrillo-Larco et al., 15 Sep 2025). The paper explicitly notes ties or wins for the fine-tuned 4B model in Neurosurgery 2024, Anesthesiology 2024, Gynecology 2024, Pediatrics 2024, Radiology 2024, Test A 2024, Thoracic and Cardiovascular Surgery 2024, and Urology 2024 (Carrillo-Larco et al., 15 Sep 2025). This is the clearest evidence that targeted PEFT on PeruMedQA can make a compact model competitive with much larger medical LLMs on in-domain tasks (Carrillo-Larco et al., 15 Sep 2025).
At the other end of the performance spectrum, models under 10B parameters mostly remained below 60% accuracy and often much lower, with some exams below 50% or even below 20% (Carrillo-Larco et al., 15 Sep 2025). The strongest sub-10B vanilla model was JSL-MedLlama-3-8B-v2.0, whose peak score was 74.73% on Psychiatry 2023; it also reached 73.91% on Psychiatry 2024 and 71.28% on Psychiatry 2020 (Carrillo-Larco et al., 15 Sep 2025). Apart from psychiatry and a few isolated tasks, however, it mostly stayed in the 40–60% range and suffered a substantial 9.00% invalid-answer rate (Carrillo-Larco et al., 15 Sep 2025).
The paper identifies several task-level trends. Psychiatry was the easiest or at least the most favorable domain for many models, including weaker ones (Carrillo-Larco et al., 15 Sep 2025). By contrast, ophthalmology, thoracic and cardiovascular surgery, and other surgery-heavy domains were relatively harder, particularly for smaller models (Carrillo-Larco et al., 15 Sep 2025). Among the 278 questions that no model answered correctly, 20.50% came from 2025 versus 15.10% from 2018; by specialty, the largest shares were Ophthalmology at 12.23%, General Surgery at 11.87%, and Thoracic and Cardiovascular Surgery at 11.15% (Carrillo-Larco et al., 15 Sep 2025). This suggests persistent difficulty in certain surgical domains and a somewhat heavier concentration of universally failed items in more recent exams (Carrillo-Larco et al., 15 Sep 2025).
6. Instruction-following, invalid outputs, and benchmark limitations
PeruMedQA’s analysis of hallucination and format failure shows that model differences were not only about medical knowledge but also about instruction-following (Carrillo-Larco et al., 15 Sep 2025). Invalid-answer rates varied dramatically. Llama3-OpenBioLLM-70B and the fine-tuned medgemma-4b-it each had 0 invalid outputs out of 8,380, medgemma-27b-text-it had 2, MediPhi-Instruct had 4, vanilla medgemma-4b-it had 12, BioMistral-7B-DARE had 76, Llama3-OpenBioLLM-8B had 416, JSL-MedLlama-3-8B-v2.0 had 755, and meditron-7b had 5,562, corresponding to 66.37% (Carrillo-Larco et al., 15 Sep 2025). The medgemma-4b-it fine-tuning therefore improved not only domain adaptation but also output compliance, reducing invalid outputs from 0.14% to 0.00% (Carrillo-Larco et al., 15 Sep 2025).
This phenomenon is methodologically important because the main metric conditions on valid responses. A model with plausible latent knowledge but weak adherence to output constraints can underperform sharply in strict MCQA evaluation (Carrillo-Larco et al., 15 Sep 2025). The meditron-7b result illustrates this most strongly: failure to produce a valid option letter in two-thirds of cases dominates its practical utility (Carrillo-Larco et al., 15 Sep 2025).
The benchmark’s limitations are explicit and nontrivial. Because it is derived from official exam content, PeruMedQA reflects examination priorities rather than the full space of medical competence (Carrillo-Larco et al., 15 Sep 2025). It lacks annotation distinguishing factual recall from clinical-case reasoning (Carrillo-Larco et al., 15 Sep 2025). It is tied to Peruvian exam-writing style and topic emphasis (Carrillo-Larco et al., 15 Sep 2025). It omits 2021 and relies only on publicly released exams (Carrillo-Larco et al., 15 Sep 2025). The forced five-option standard changes chance performance for originally four-option questions, making comparison with human exam outcomes unfair (Carrillo-Larco et al., 15 Sep 2025).
The paper does not provide calibration studies, confidence estimates, or a formal error taxonomy beyond hallucinations and invalid answers (Carrillo-Larco et al., 15 Sep 2025). It does not analyze answer rationales (Carrillo-Larco et al., 15 Sep 2025). One likely failure mode discussed by the authors is incomplete alignment with Peruvian epidemiology, local clinical emphases, and exam formulation style (Carrillo-Larco et al., 15 Sep 2025). The gains from fine-tuning medgemma-4b-it are interpreted as evidence that base medical LLMs may possess general medical knowledge while lacking sufficient exposure to Peru-specific epidemiological profiles and the specific way Peruvian medical exam questions are asked (Carrillo-Larco et al., 15 Sep 2025).
7. Position within Spanish and medical QA research
PeruMedQA occupies a distinct position within medical QA research. It is an exam benchmark rather than a patient-facing community QA or regulator-grounded leaflet retrieval system. That distinction becomes clear when it is contrasted with other Spanish or healthcare QA efforts on arXiv.
The paper "Medicines Question Answering System, MeQA" describes what its authors call the first Spanish system capable of answering questions about medicines for human use, grounded in official medicine leaflets and organized as a hybrid retrieval-oriented pipeline rather than an exam benchmark (Santamaría, 2021). MeQA uses regulated leaflet content, entity normalization, section prediction, and sentence extraction, whereas PeruMedQA evaluates zero-shot and fine-tuned medical LLMs on multiple-choice specialist examinations (Santamaría, 2021). The two projects therefore share a Spanish-language medical QA orientation but instantiate different task formulations: document-grounded medicine QA in MeQA versus benchmarked medical MCQA in PeruMedQA (Santamaría, 2021, Carrillo-Larco et al., 15 Sep 2025).
A different neighboring line is represented by "YaleNLP @ PerAnsSumm 2025: Multi-Perspective Integration via Mixture-of-Agents for Enhanced Healthcare QA Summarization," which studies perspective-aware healthcare answer summarization in community question-answering forums (Jang et al., 4 Apr 2025). That work addresses multi-answer synthesis, perspective extraction, and factuality-aware summarization, not specialist exam answering (Jang et al., 4 Apr 2025). Its relevance to PeruMedQA is therefore architectural rather than direct. This suggests that future PeruMedQA-based systems could combine exam-tuned medical knowledge with perspective-aware evidence synthesis for patient-facing or forum-based settings, but such an integration is not part of the PeruMedQA paper itself (Jang et al., 4 Apr 2025, Carrillo-Larco et al., 15 Sep 2025).
Within its own declared scope, PeruMedQA supports three principal conclusions. First, medical LLM performance does not transfer uniformly across languages and regions; benchmarking in Spanish and in local epidemiological contexts is necessary (Carrillo-Larco et al., 15 Sep 2025). Second, for Peru and potentially for other Spanish-speaking countries with similar disease burdens and exam styles, medgemma-27b-text-it is the best off-the-shelf open medical model among those tested (Carrillo-Larco et al., 15 Sep 2025). Third, when computational resources are limited, a fine-tuned medgemma-4b-it is a strong practical alternative because it beats all tested medical LLMs under 10B and rivals a 70B model in many settings, though it still does not surpass medgemma-27b-text-it overall and loses more performance on the held-out 2025 exam (Carrillo-Larco et al., 15 Sep 2025).
The authors propose several future directions: evaluating more advanced prompting strategies beyond zero-shot, testing larger and possibly proprietary general-purpose models such as GPT-class systems, fine-tuning stronger models like medgemma-27b-text-it if resources permit, prompting models to produce and then evaluating rationales, using more flexible answer extraction so smaller models are not penalized as much for formatting failures, extending evaluation to medical student exams and other health professions, distinguishing factual questions from clinical-case reasoning questions, and monitoring for catastrophic forgetting when specializing smaller LLMs on local datasets (Carrillo-Larco et al., 15 Sep 2025). In that sense, PeruMedQA functions both as a benchmark and as a concrete argument for region-specific, Spanish-language medical model evaluation in the global South (Carrillo-Larco et al., 15 Sep 2025).