HealthBench-BR: Brazilian Clinical LLM Benchmark
- HealthBench-BR is a benchmark that evaluates Brazilian clinical knowledge using 1,780 balanced true/false assertions derived from 178 official SUS guidelines.
- It employs a domain adaptation pipeline where GPT-5.2 generates paired clinical assertions by modifying a single detail to ensure precise recall of protocols.
- Empirical results demonstrate that continual pre-training combined with reinforcement learning boosts accuracy, with the best model achieving 83.9% on factual verification.
HealthBench-BR is a guideline-grounded benchmark for evaluating Brazilian clinical knowledge in LLMs. Introduced alongside the domain adaptation pipeline in “Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines” (Abonizio et al., 1 May 2026), it consists of 1,780 balanced true/false clinical assertions derived from 178 official clinical guidelines that govern Brazil’s Unified Health System (SUS). Its primary function is to measure whether a model can accept or reject concrete statements about diagnosis, treatment, dosages, monitoring, contraindications, and related protocol details in Brazilian Portuguese, without relying on external retrieval (Abonizio et al., 1 May 2026).
1. Scope, rationale, and institutional grounding
HealthBench-BR was created to address three gaps in medical LLM evaluation. First, most medical LLMs and benchmarks are English-centric and built from US or European literature and exams. Second, existing Portuguese medical benchmarks, including Revalida, MedPT, HealthQA-BR, and DrBode, do not directly test recall of the official SUS clinical guidelines. Third, SUS care is governed by mandatory official protocols whose volume and complexity make at-the-point-of-care recall difficult, so any clinical-support LLM intended for SUS deployment must reliably reproduce these recommendations (Abonizio et al., 1 May 2026).
The benchmark is explicitly anchored in the official documents elaborated with CONITEC and adopted by SUS managers. These documents encode diagnostic criteria, treatment regimens, exact dosages, routes of administration, monitoring and follow-up intervals, and inclusion or exclusion criteria. HealthBench-BR operationalizes this corpus as a low-ambiguity factual verification task: each assertion must be judged true or false relative to the official guidelines, in Brazilian Portuguese, and the task is framed as parametric recall rather than retrieval-augmented consultation (Abonizio et al., 1 May 2026).
This design distinguishes HealthBench-BR from the broader HealthBench family. The original HealthBench evaluates 5,000 multi-turn conversations with 48,562 unique rubric criteria across themes such as emergencies, uncertainty, global health, and health data tasks (Arora et al., 13 May 2025). By contrast, HealthBench-BR is narrower and more factual: it targets direct protocol adherence rather than open-ended conversational quality (Abonizio et al., 1 May 2026).
2. Source corpus and construction pipeline
The benchmark is grounded in 178 official Brazilian clinical guidelines collected from the Ministry of Health’s PCDT portal as of March 2026. The corpus includes Clinical Protocols and Therapeutic Guidelines, Protocolos de Uso, Diretrizes Diagnósticas e Terapêuticas em Oncologia, Diretrizes Brasileiras, and Linhas de Cuidado. Raw extraction yielded approximately 26.9M characters; after truncation to 120k characters per document for generator context limits, the corpus size is approximately 16.6M characters, or about 5.4M tokens (Abonizio et al., 1 May 2026).
HealthBench-BR was generated automatically with GPT-5.2. For each guideline, GPT-5.2 received the full text, selected a clinical fact, and produced a true assertion faithful to that fact. It then generated a false variant by changing one critical detail, such as a dosage, an interval, or an inclusion or exclusion condition, while keeping the statement clinically plausible. The paper characterizes the result as paired generation: each true assertion is matched with a minimally modified false distractor (Abonizio et al., 1 May 2026).
The assertions are written in Brazilian Portuguese and mimic the formal register of official guidelines rather than lay language. Because they are derived directly from guideline text, they retain domain-specific terminology, Brazilian naming conventions, and SUS-specific organizational terms. The paper does not describe a manual clinical expert correction pass for HealthBench-BR, and it does not report inter-annotator agreement or manual auditing metrics. Quality control is instead indirect, relying on guideline grounding, one-detail perturbations in false statements, and downstream sanity checks such as calibration analyses (Abonizio et al., 1 May 2026).
3. Dataset structure and evaluation protocol
HealthBench-BR contains 1,780 clinical assertions, with 10 assertions per guideline. The 178 guidelines are split evenly into 89 in the train split and 89 in the test split, producing 890 assertions per split. The benchmark is perfectly balanced overall: 50% of assertions are true and 50% are false. This balance is constructed pairwise, with one true statement and one modified false variant for each clinical fact (Abonizio et al., 1 May 2026).
Each item is a binary classification problem. The input is a single Brazilian-Portuguese sentence stating a clinical assertion, and the model must conclude with an explicit verdict, “Verdadeiro” or “Falso.” During evaluation, the model is prompted to explain its reasoning in natural language and then provide the verdict. A regex extracts the last occurrence of “Verdadeiro” or “Falso,” and all evaluation runs use greedy decoding to remove sampling noise (Abonizio et al., 1 May 2026).
The benchmark targets detailed clinical content rather than generic medical reasoning. Evaluated knowledge includes diagnostic criteria, treatment indications and lines of therapy, contraindications and precautions, numerical dosages, routes of administration, follow-up and monitoring schedules, and operational details such as switching-therapy criteria. Because the false variants modify one critical detail, often numerical or categorical, the task forces discrimination between plausible near-misses such as “monitoring every 3 months” versus “every 6 months,” or “dose of 10 mg/kg” versus “15 mg/kg” (Abonizio et al., 1 May 2026).
The reported metric is accuracy:
Macro-F1 is not used, since the dataset is perfectly balanced. Evaluation is zero-shot with respect to the assertion text: no few-shot assertion examples are provided in the prompt (Abonizio et al., 1 May 2026).
4. Role in the training pipeline
HealthBench-BR is not only an evaluation set but also part of the training strategy used in the paper’s adapted Qwen2.5-14B-Instruct model. The broader pipeline begins with 178 official guidelines, from which the authors generate approximately 70M tokens of synthetic data in three formats: rephrases, wiki-style articles, and question-answer pairs. Each guideline is processed by four generator LLMs—GPT-4.1-mini, GPT-5-nano, GPT-OSS-20B, and Qwen3-235B—yielding 17,800 synthetic documents, described as a 13× expansion (Abonizio et al., 1 May 2026).
Continual pre-training is then applied to Qwen2.5-14B-Instruct on this approximately 70M-token synthetic corpus using a standard causal language-modeling objective for 3 epochs, with learning rate and sequence length 4,096. After continual pre-training, the authors apply Group Relative Policy Optimization with LoRA , using the HealthBench-BR train split as the reward environment. For each assertion, 16 completions are sampled. The reward is 1.0 if the verdict is correct and the answer contains at least 50 words of reasoning, and 0.0 otherwise. This reward design is intended to discourage bare-label reward hacking while encouraging correct verdicts and extended clinical reasoning (Abonizio et al., 1 May 2026).
The paper also evaluates supervised fine-tuning on GPT-5.2-generated answers to HealthBench-BR train questions, but reports that this underperforms reinforcement learning on HealthBench-BR test. In this sense, HealthBench-BR functions simultaneously as a benchmark for factual recall and as a task-aligned reinforcement signal for verdict-style clinical verification (Abonizio et al., 1 May 2026).
5. Empirical performance, ablations, and calibration
On the HealthBench-BR test split, base Qwen2.5-14B-Instruct scores 59.4%. RL only reaches 64.2%, and SFT only reaches 62.1%. Continual pre-training with one generator reaches 69.6%, while CPT plus RL with one generator reaches 75.6%. Using four generators improves CPT alone to 71.1%, and the final configuration—CPT with four generators plus RL—reaches 83.9% (Abonizio et al., 1 May 2026).
The same paper reports frontier proprietary baselines on HealthBench-BR test: GPT-4.1 at 73.3%, GPT-5.2 instant at 76.3%, GPT-5.2 high at 78.5%, Claude Sonnet 4.6 at 77.6%, and Gemini 3.1 Pro at 79.3%. Google AI Overview, using web-grounded retrieval, reaches 70.5% on answered questions, with approximately 11.7% abstention overall. The paper also evaluates retrieval-augmented baselines over the guidelines: baseline plus BM25 RAG reaches 90.0%, baseline plus embedding RAG 87.9%, and the final CPT-plus-RL model plus BM25 RAG reaches 95.3% (Abonizio et al., 1 May 2026).
The ablations support two main conclusions. First, continual pre-training is necessary for knowledge injection, moving performance from 59.4% to roughly 69.6–71.1%. Second, GRPO is particularly important for HealthBench-BR’s structured true/false task: with four generators, RL increases HealthBench-BR test performance from 71.1% to 83.9%, a gain of 12.8 points. Generator diversity has a modest direct effect on HealthBench-BR but a larger effect on the companion open-ended benchmark PCDT-QA (Abonizio et al., 1 May 2026).
The paper also uses HealthBench-BR to analyze calibration through the proportion of “True” answers. Because the dataset is 50/50 true and false, a well-calibrated model should output “True” about half the time. The base Qwen2.5-14B model has a true-rate of 49.9%; after continual pre-training it shifts to 58.9%, indicating a bias toward agreement; after GRPO in the final four-generator model it moves back toward balance at 53.0%. Among frontier systems, Claude Sonnet 4.6 is at 49.2%, GPT-5.2 high at 48.7%, Gemini 3.1 Pro at 63.3%, and Google AI Overview at 69.6% (Abonizio et al., 1 May 2026).
The paper cautions that 83.9% should not be interpreted as sufficient for unsupervised clinical use, because residual errors may involve consequential details such as dosages or contraindications. No fine-grained error taxonomy is reported, but dosage and contraindication failures are explicitly identified as safety-critical (Abonizio et al., 1 May 2026).
6. Benchmark position, limitations, and availability
HealthBench-BR occupies a specific niche within Brazilian and international medical benchmarking. It is complementary to global resources such as MedQA, MedMCQA, and MultiMedQA, and to Portuguese benchmarks such as Revalida, MedPT, HealthQA-BR, and DrBode, but differs in four respects: it is built from Brazilian SUS protocols, written natively in Brazilian Portuguese, formulated as true/false verification of clinical assertions, and grounded in official guidelines rather than exam questions or generic medical literature (Abonizio et al., 1 May 2026). This emphasis on jurisdiction-specific localization is consistent with broader evidence that direct benchmark translation can mismeasure local clinical appropriateness (Hisada et al., 22 Sep 2025), and with critiques arguing that health benchmarks should be anchored in clinical practice guidelines rather than only in expert-opinion rubrics (Mutisya et al., 31 Jul 2025).
Its intended uses are to evaluate and improve SUS guideline adherence, to measure whether continual pre-training and reinforcement learning actually inject protocol knowledge into model parameters, and to study calibration and sycophancy on binary clinical tasks. Its limitations are equally explicit. HealthBench-BR measures protocol recall rather than comprehensive clinical competence; real encounters involve patient-specific nuance, incomplete data, and trade-offs not fully captured by guidelines. The dataset is generated by GPT-5.2 from public guidelines, so some statements may contain subtle noise or suboptimal phrasing. Guidelines change over time, creating temporal drift. No formal manual clinical audit or inter-annotator agreement is reported. The resulting model is presented as a research artifact rather than a certified medical device, and the authors explicitly state that it cannot be used unsupervised in clinical decision-making (Abonizio et al., 1 May 2026).
The project is fully released for reproducible clinical NLP research in Brazilian Portuguese. The repository at https://github.com/hugoabonizio/clinical-protocols-br includes extracted protocol texts, the synthetic training corpus, the HealthBench-BR train and test splits, the companion PCDT-QA benchmark, model weights for the adapted Qwen2.5-14B variants, and evaluation scripts for prompting and verdict parsing (Abonizio et al., 1 May 2026). In that form, HealthBench-BR functions both as a benchmark and as an experimental scaffold for studying guideline-grounded knowledge injection in a major non-English healthcare system.