PCDT-QA: Brazilian Clinical QA Benchmark

• PCDT-QA is an open-ended clinical QA benchmark that measures if models can recall and apply official Brazilian SUS protocol details in free-form answers.
• Built from 178 clinical guidelines and 890 open-ended questions, it covers both broad clinical topics and fine-grained protocol specifics like dosages and criteria.
• Empirical results highlight that continual pre-training with diverse LLM-generated synthetic data boosts accuracy up to 86.3%, underscoring the power of targeted data augmentation.

Searching arXiv for the cited paper to ground the article and confirm bibliographic details. {"query":"arXiv (Abonizio et al., 1 May 2026) Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines","max_results":5} PCDT-QA is an open-ended clinical question-answering benchmark in Brazilian Portuguese introduced in “Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines” (Abonizio et al., 1 May 2026). It is designed to measure whether a LLM can recall and apply knowledge from Brazil’s official SUS clinical guidelines in free-form answers rather than in multiple-choice or binary-verification settings. The benchmark is grounded in the regulatory and therapeutic framework of the Brazilian Unified Health System, with questions that require protocol-specific knowledge such as inclusion and exclusion criteria, recommended dosages, monitoring intervals, pediatric dosing, and related procedural details. Within the paper, PCDT-QA complements HealthBench-BR: the latter evaluates balanced true/false clinical assertions, whereas PCDT-QA evaluates open-ended answer quality.

1. Definition and intended scope

“PCDT” refers to Protocolos Clínicos e Diretrizes Terapêuticas, the clinical protocols and therapeutic guidelines issued by the Brazilian Ministry of Health. PCDT-QA is a question-answering benchmark derived from those PCDTs and related official SUS documents. The paper does not explicitly spell out the full expansion “PCDT-QA,” but the benchmark is clearly presented as open-ended QA over Brazilian clinical protocols (Abonizio et al., 1 May 2026).

Its primary objective is to test whether a model can recall and apply Brazilian SUS guideline knowledge, rather than generic medical knowledge drawn from international textbooks, exam banks, or PubMed-style evidence summaries. The benchmark targets both broad protocol content and fine-grained factual detail. Broad topics include inclusion and exclusion criteria for access to therapies and high-level diagnostic criteria for entering a protocol. Fine-grained topics include dosages, titration schedules, monitoring intervals, follow-up examinations, laboratory testing, pediatric versus adult dosing, contraindications, and adjustments.

This scope is tightly coupled to SUS practice. The source documents are described as mandatory for SUS managers and as defining diagnostic criteria, treatments, dosages, and monitoring procedures for conditions covered by the public system. Accordingly, correct performance on PCDT-QA depends on knowledge of official national protocols rather than on general clinical fluency alone. The benchmark was introduced because, according to the paper, no prior benchmark evaluated factual recall grounded in Brazilian Portuguese official protocols.

2. Source corpus and benchmark construction

PCDT-QA is built from the full corpus of 178 clinical guidelines available from the Ministry of Health’s PCDT portal as of March 2026 (Abonizio et al., 1 May 2026). These guidelines were downloaded as PDFs, text-extracted, and truncated at 120k characters when necessary to fit context limits during generation. The resulting corpus comprises approximately 5.4 million tokens and 16.6 million characters. It includes PCDTs, Protocolos de Uso, oncology diagnostic and therapeutic guidelines, national guidelines, and care pathways.

For benchmarking, the 178 guidelines are split by guideline into 89 train guidelines and 89 test guidelines. PCDT-QA contains 890 open-ended questions, with 5 questions per guideline and therefore 445 questions per split. The paper describes the benchmark as containing questions “covering both broad clinical topics (e.g., inclusion criteria) and specific factual queries (e.g., recommended pediatric dosages),” and each question is paired with a reference answer grounded in the underlying guideline text.

Property	Value	Note
Guidelines	178	Ministry of Health corpus
Questions	890	5 per guideline
Split	445 / 445	Train / test
Language	Brazilian Portuguese	Question and reference answer
Granularity	Broad + fine-grained	Criteria, dosages, monitoring

Each guideline contributes exactly five questions, and the benchmark is intended to provide system-wide coverage rather than specialty-specific exclusion. Because the split is performed at the guideline level, test questions come from guidelines that are not used to generate QA training data. For the test guidelines, only rephrases and wiki-style articles are generated in the synthetic data pipeline; QA generation is restricted to the training-side guidelines to avoid leakage.

The paper does not explicitly state whether PCDT-QA questions were authored by humans or generated by LLMs. The broader data-generation pipeline strongly suggests guideline-grounded LLM generation, but that point remains inferential rather than explicitly documented.

3. Data representation and scoring protocol

A PCDT-QA instance consists of a question text in Brazilian Portuguese and a reference answer, also in Portuguese, derived from the relevant guideline (Abonizio et al., 1 May 2026). The benchmark also has implicit guideline grounding, since each pair corresponds to a specific protocol, although the paper’s main text does not describe a public instance-level guideline identifier. The benchmark is not presented as multiple choice, and the paper does not mention explicit difficulty labels or category tags.

Evaluation uses an LLM-as-a-judge protocol. The model under evaluation receives the question and produces a free-text answer in Portuguese. A GPT-4.1 judge then compares the candidate answer against the reference answer and returns a binary verdict, correct or incorrect. The overall score is therefore accuracy, computed as the proportion of questions judged correct over the total number of questions.

This protocol makes PCDT-QA a semantic evaluation rather than a string-matching benchmark. A model is not rewarded merely for lexical overlap; it must provide an answer that GPT-4.1 deems correct relative to the guideline-grounded reference. The paper does not provide the judge prompt, and it does not explicitly specify decoding details for PCDT-QA, although it notes greedy decoding for the true/false task.

Methodologically, this scoring design places PCDT-QA closer to rubric-based clinical answer assessment than to exact-match QA. A plausible implication is that the benchmark is sensitive both to factual recall and to the ability to articulate protocol content in a semantically faithful manner.

4. Role in the model-adaptation pipeline

PCDT-QA is a held-out evaluation benchmark rather than a training dataset (Abonizio et al., 1 May 2026). The paper adapts Qwen2.5-14B-Instruct to the Brazilian clinical domain using a two-stage procedure. First, the authors perform continual pre-training (CPT) on approximately 70 million tokens of synthetic data derived from the 178 guidelines. That synthetic corpus contains three formats: rephrases, wiki-style articles, and question-answer pairs. The data are generated by four LLMs: GPT-4.1-mini, GPT-5-nano, GPT-OSS-20B, and Qwen3-235B.

Second, the paper applies Group Relative Policy Optimization (GRPO) using the train split of HealthBench-BR as reward supervision. In that RL stage, true/false verification is treated as a verifiable-reward task, with reward equal to 1.0 only when the model’s binary verdict is correct and the explanation is at least 50 words, and 0 otherwise. GRPO is implemented with LoRA on Qwen2.5-14B.

Within that pipeline, PCDT-QA serves a diagnostic purpose: it measures whether synthetic guideline-derived training and subsequent RL have injected national clinical protocol knowledge into the model’s parameters. The benchmark shares the same knowledge source as the synthetic QA data—the official guidelines—but not the same role. Synthetic QA is part of training; PCDT-QA remains reserved for evaluation.

This separation is central to the paper’s experimental logic. The benchmark is meant to test generalization to held-out guidelines and to determine whether gains reflect genuine parameter-level acquisition of SUS protocol knowledge rather than memorization of benchmark items.

5. Empirical results and ablation patterns

On the PCDT-QA test split, the baseline Qwen2.5-14B-Instruct achieves 27.9% accuracy (Abonizio et al., 1 May 2026). The paper reports that RL only reaches 29.4%, and SFT only on pseudo-labeled HealthBench-BR data also reaches 29.4%. By contrast, CPT with 1 generator increases test accuracy to 66.3%. CPT + SFT yields 57.8%, while CPT + RL reaches 65.4%. The largest gains occur with generator diversity: CPT with 4 generators reaches 86.3%, CPT (4 generators) + SFT reaches 81.1%, and CPT (4 generators) + RL reaches 85.4%.

Configuration	PCDT-QA test
Baseline Qwen2.5-14B-Instruct	27.9%
RL only	29.4%
SFT only	29.4%
CPT, 1 generator	66.3%
CPT + RL	65.4%
CPT, 4 generators	86.3%
CPT, 4 generators + RL	85.4%

These results show that continual pre-training on synthetic guideline-derived data is the main driver of improvement on PCDT-QA, while generator diversity is critical. The paper also reports that its best joint-benchmark model achieves 85.4% on PCDT-QA and outperforms GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Pro, and Google AI Overview’s web-grounded RAG, despite using only 14B parameters.

The augmentation ablations clarify why. With 1 generator, the paper reports 52.4% for rephrase only, 60.9% for wiki only, 65.4% for wiki + rephrase, and 66.3% for rephrase + wiki + QA. With 4 generators, the corresponding numbers are 73.0%, 76.9%, 83.8%, and 86.3%. Wiki-style articles are consistently stronger than rephrases alone, and combining formats is always beneficial. The authors interpret the disproportionate gain on PCDT-QA as evidence that generator diversity primarily benefits free-form knowledge recall, where varied phrasings and explanatory styles translate into richer answers.

GRPO produces a more nuanced effect. For the 4-generator setting, PCDT-QA declines slightly from 86.3% under CPT alone to 85.4% under CPT + RL, even though RL improves HealthBench-BR more strongly. This indicates that the RL objective, tuned on true/false verification, is better aligned with binary factual judgment than with open-ended clinical answering. The benchmark therefore exposes a distinction between binary calibration and free-text protocol recall.

6. Position within clinical NLP, practical use, and limitations

PCDT-QA occupies a specific niche relative to existing clinical QA benchmarks (Abonizio et al., 1 May 2026). Compared with benchmarks such as MedQA, MedMCQA, PubMedQA, and MultiMedQA, it differs along three axes. First, it is explicitly Brazilian Portuguese, whereas many widely used benchmarks are English-centric. Second, it is grounded in official national clinical guidelines rather than exam banks, textbooks, or biomedical abstracts. Third, it is fully open-ended, with free-text questions and answers evaluated by an LLM judge, rather than multiple-choice selection.

Those design choices make the benchmark particularly suitable for evaluating models intended for SUS-aligned clinical NLP. It tests not only generic medical knowledge but also the protocol-specific treatment pathways, coverage rules, and follow-up schedules that govern care within the Brazilian public health system. In this sense, PCDT-QA and HealthBench-BR form a protocol-grounded evaluation suite: one tests open-ended answer production, and the other tests binary factual verification.

The benchmark, associated datasets, code, and model weights are released through the project repository at https://github.com/hugoabonizio/clinical-protocols-br. The source documents are public government publications, and the paper states that no private clinical notes or patient data are used. At the same time, the released models and datasets are characterized as research artifacts, not certified medical devices.

Several limitations are explicit. PCDT-QA reflects a snapshot of 178 guidelines as of March 2026; subsequent revisions to SUS protocols are not represented until the benchmark is updated. Each guideline contributes only five questions, so breadth is large but per-guideline depth is limited. The paper does not state that questions were created or vetted by clinicians, which leaves open the possibility of generator-induced stylistic or conceptual bias. The use of GPT-4.1 as judge, and likely as generator in parts of the broader pipeline, also creates a potential evaluation bias. Most importantly, high benchmark scores do not establish deployment safety: the paper warns that residual errors may still involve dosages or contraindications and that use should occur only under professional supervision.

Future extensions suggested in the paper include adding more questions per guideline, updating the benchmark as SUS protocols evolve, incorporating human clinical review, using more nuanced grading schemes such as partial credit, and complementing protocol recall with real-case vignettes and safety-constrained patient-context evaluation. Together, these directions imply that PCDT-QA is best understood as a foundational benchmark for Brazilian guideline-grounded clinical QA rather than a complete clinical safety assessment.