---
title: 'PCDT-QA: Brazilian Clinical QA Benchmark'
url: https://www.emergentmind.com/topics/pcdt-qa
type: topic
---

# PCDT-QA: Brazilian Clinical QA Benchmark

Searching arXiv for the cited paper to ground the article and confirm bibliographic details.
{"query":"arXiv 2605.01077 Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines","max_results":5}
PCDT-QA is an open-ended clinical question-answering benchmark in Brazilian Portuguese introduced in “Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines” [2605.01077]. It is designed to measure whether a language model can recall and apply knowledge from Brazil’s official SUS clinical guidelines in free-form answers rather than in multiple-choice or binary-verification settings. The benchmark is grounded in the regulatory and therapeutic framework of the Brazilian Unified Health System, with questions that require protocol-specific knowledge such as inclusion and exclusion criteria, recommended dosages, monitoring intervals, pediatric dosing, and related procedural details. Within the paper, PCDT-QA complements HealthBench-BR: the latter evaluates balanced true/false clinical assertions, whereas PCDT-QA evaluates open-ended answer quality.

## 1. Definition and intended scope

“PCDT” refers to **Protocolos Clínicos e Diretrizes Terapêuticas**, the clinical protocols and therapeutic guidelines issued by the Brazilian Ministry of Health. PCDT-QA is a question-answering benchmark derived from those PCDTs and related official SUS documents. The paper does not explicitly spell out the full expansion “PCDT-QA,” but the benchmark is clearly presented as open-ended QA over Brazilian clinical protocols [2605.01077].

Its primary objective is to test whether a model can recall and apply **Brazilian SUS guideline knowledge**, rather than generic medical knowledge drawn from international textbooks, exam banks, or PubMed-style evidence summaries. The benchmark targets both broad protocol content and fine-grained factual detail. Broad topics include inclusion and exclusion criteria for access to therapies and high-level diagnostic criteria for entering a protocol. Fine-grained topics include dosages, titration schedules, monitoring intervals, follow-up examinations, laboratory testing, pediatric versus adult dosing, contraindications, and adjustments.

This scope is tightly coupled to SUS practice. The source documents are described as mandatory for SUS managers and as defining diagnostic criteria, treatments, dosages, and monitoring procedures for conditions covered by the public system. Accordingly, correct performance on PCDT-QA depends on knowledge of official national protocols rather than on general clinical fluency alone. The benchmark was introduced because, according to the paper, no prior benchmark evaluated factual recall grounded in Brazilian Portuguese official protocols.

## 2. Source corpus and benchmark construction

PCDT-QA is built from the full corpus of **178 clinical guidelines** available from the Ministry of Health’s PCDT portal as of **March 2026** [2605.01077]. These guidelines were downloaded as PDFs, text-extracted, and truncated at **120k characters** when necessary to fit context limits during generation. The resulting corpus comprises approximately **5.4 million tokens** and **16.6 million characters**. It includes PCDTs, Protocolos de Uso, oncology diagnostic and therapeutic guidelines, national guidelines, and care pathways.

For benchmarking, the 178 guidelines are split by guideline into **89 train guidelines** and **89 test guidelines**. PCDT-QA contains **890 open-ended questions**, with **5 questions per guideline** and therefore **445 questions per split**. The paper describes the benchmark as containing questions “covering both broad clinical topics (e.g., inclusion criteria) and specific factual queries (e.g., recommended pediatric dosages),” and each question is paired with a reference answer grounded in the underlying guideline text.

| Property | Value | Note |
|---|---:|---|
| Guidelines | 178 | Ministry of Health corpus |
| Questions | 890 | 5 per guideline |
| Split | 445 / 445 | Train / test |
| Language | Brazilian Portuguese | Question and reference answer |
| Granularity | Broad + fine-grained | Criteria, dosages, monitoring |

Each guideline contributes exactly five questions, and the benchmark is intended to provide system-wide coverage rather than specialty-specific exclusion. Because the split is performed at the guideline level, test questions come from guidelines that are not used to generate QA training data. For the test guidelines, only rephrases and wiki-style articles are generated in the synthetic data pipeline; QA generation is restricted to the training-side guidelines to avoid leakage.

The paper does not explicitly state whether PCDT-QA questions were authored by humans or generated by LLMs. The broader data-generation pipeline strongly suggests guideline-grounded LLM generation, but that point remains inferential rather than explicitly documented.

## 3. Data representation and scoring protocol

A PCDT-QA instance consists of a **question text** in Brazilian Portuguese and a **reference answer**, also in Portuguese, derived from the relevant guideline [2605.01077]. The benchmark also has implicit guideline grounding, since each pair corresponds to a specific protocol, although the paper’s main text does not describe a public instance-level guideline identifier. The benchmark is not presented as multiple choice, and the paper does not mention explicit difficulty labels or category tags.

Evaluation uses an **LLM-as-a-judge** protocol. The model under evaluation receives the question and produces a free-text answer in Portuguese. A **GPT-4.1** judge then compares the candidate answer against the reference answer and returns a **binary verdict**, correct or incorrect. The overall score is therefore **accuracy**, computed as the proportion of questions judged correct over the total number of questions.

This protocol makes PCDT-QA a semantic evaluation rather than a string-matching benchmark. A model is not rewarded merely for lexical overlap; it must provide an answer that GPT-4.1 deems correct relative to the guideline-grounded reference. The paper does not provide the judge prompt, and it does not explicitly specify decoding details for PCDT-QA, although it notes greedy decoding for the true/false task.

Methodologically, this scoring design places PCDT-QA closer to rubric-based clinical answer assessment than to exact-match QA. A plausible implication is that the benchmark is sensitive both to factual recall and to the ability to articulate protocol content in a semantically faithful manner.

## 4. Role in the model-adaptation pipeline

PCDT-QA is a **held-out evaluation benchmark** rather than a training dataset [2605.01077]. The paper adapts **Qwen2.5-14B-Instruct** to the Brazilian clinical domain using a two-stage procedure. First, the authors perform **continual pre-training (CPT)** on approximately **70 million tokens** of synthetic data derived from the 178 guidelines. That synthetic corpus contains three formats: **rephrases**, **wiki-style articles**, and **question-answer pairs**. The data are generated by four LLMs: **GPT-4.1-mini**, **GPT-5-nano**, **GPT-OSS-20B**, and **Qwen3-235B**.

Second, the paper applies **Group Relative Policy Optimization (GRPO)** using the train split of HealthBench-BR as reward supervision. In that RL stage, true/false verification is treated as a verifiable-reward task, with reward equal to **1.0** only when the model’s binary verdict is correct and the explanation is at least **50 words**, and **0** otherwise. GRPO is implemented with **LoRA** on Qwen2.5-14B.

Within that pipeline, PCDT-QA serves a diagnostic purpose: it measures whether synthetic guideline-derived training and subsequent RL have injected national clinical protocol knowledge into the model’s parameters. The benchmark shares the same knowledge source as the synthetic QA data—the official guidelines—but not the same role. Synthetic QA is part of training; PCDT-QA remains reserved for evaluation.

This separation is central to the paper’s experimental logic. The benchmark is meant to test generalization to held-out guidelines and to determine whether gains reflect genuine parameter-level acquisition of SUS protocol knowledge rather than memorization of benchmark items.

## 5. Empirical results and ablation patterns

On the PCDT-QA **test split**, the baseline **Qwen2.5-14B-Instruct** achieves **27.9%** accuracy [2605.01077]. The paper reports that **RL only** reaches **29.4%**, and **SFT only** on pseudo-labeled HealthBench-BR data also reaches **29.4%**. By contrast, **CPT with 1 generator** increases test accuracy to **66.3%**. **CPT + SFT** yields **57.8%**, while **CPT + RL** reaches **65.4%**. The largest gains occur with generator diversity: **CPT with 4 generators** reaches **86.3%**, **CPT (4 generators) + SFT** reaches **81.1%**, and **CPT (4 generators) + RL** reaches **85.4%**.

| Configuration | PCDT-QA test |
|---|---:|
| Baseline Qwen2.5-14B-Instruct | 27.9% |
| RL only | 29.4% |
| SFT only | 29.4% |
| CPT, 1 generator | 66.3% |
| CPT + RL | 65.4% |
| CPT, 4 generators | 86.3% |
| CPT, 4 generators + RL | 85.4% |

These results show that **continual pre-training on synthetic guideline-derived data is the main driver** of improvement on PCDT-QA, while **generator diversity is critical**. The paper also reports that its best joint-benchmark model achieves **85.4%** on PCDT-QA and outperforms **GPT-5.2**, **Claude Sonnet 4.6**, **Gemini 3.1 Pro**, and **Google AI Overview’s web-grounded RAG**, despite using only **14B parameters**.

The augmentation ablations clarify why. With **1 generator**, the paper reports **52.4%** for rephrase only, **60.9%** for wiki only, **65.4%** for wiki + rephrase, and **66.3%** for rephrase + wiki + QA. With **4 generators**, the corresponding numbers are **73.0%**, **76.9%**, **83.8%**, and **86.3%**. Wiki-style articles are consistently stronger than rephrases alone, and combining formats is always beneficial. The authors interpret the disproportionate gain on PCDT-QA as evidence that generator diversity primarily benefits **free-form knowledge recall**, where varied phrasings and explanatory styles translate into richer answers.

GRPO produces a more nuanced effect. For the 4-generator setting, PCDT-QA declines slightly from **86.3%** under CPT alone to **85.4%** under CPT + RL, even though RL improves HealthBench-BR more strongly. This indicates that the RL objective, tuned on true/false verification, is better aligned with binary factual judgment than with open-ended clinical answering. The benchmark therefore exposes a distinction between **binary calibration** and **free-text protocol recall**.

## 6. Position within clinical NLP, practical use, and limitations

PCDT-QA occupies a specific niche relative to existing clinical QA benchmarks [2605.01077]. Compared with benchmarks such as **MedQA**, **MedMCQA**, **PubMedQA**, and **MultiMedQA**, it differs along three axes. First, it is explicitly **Brazilian Portuguese**, whereas many widely used benchmarks are English-centric. Second, it is grounded in **official national clinical guidelines** rather than exam banks, textbooks, or biomedical abstracts. Third, it is **fully open-ended**, with free-text questions and answers evaluated by an LLM judge, rather than multiple-choice selection.

Those design choices make the benchmark particularly suitable for evaluating models intended for SUS-aligned clinical NLP. It tests not only generic medical knowledge but also the protocol-specific treatment pathways, coverage rules, and follow-up schedules that govern care within the Brazilian public health system. In this sense, PCDT-QA and HealthBench-BR form a protocol-grounded evaluation suite: one tests open-ended answer production, and the other tests binary factual verification.

The benchmark, associated datasets, code, and model weights are released through the project repository at `https://github.com/hugoabonizio/clinical-protocols-br`. The source documents are public government publications, and the paper states that no private clinical notes or patient data are used. At the same time, the released models and datasets are characterized as **research artifacts**, not certified medical devices.

Several limitations are explicit. PCDT-QA reflects a **snapshot** of 178 guidelines as of March 2026; subsequent revisions to SUS protocols are not represented until the benchmark is updated. Each guideline contributes only **five questions**, so breadth is large but per-guideline depth is limited. The paper does not state that questions were created or vetted by clinicians, which leaves open the possibility of generator-induced stylistic or conceptual bias. The use of **GPT-4.1** as judge, and likely as generator in parts of the broader pipeline, also creates a potential evaluation bias. Most importantly, high benchmark scores do not establish deployment safety: the paper warns that residual errors may still involve dosages or contraindications and that use should occur **only under professional supervision**.

Future extensions suggested in the paper include adding more questions per guideline, updating the benchmark as SUS protocols evolve, incorporating human clinical review, using more nuanced grading schemes such as partial credit, and complementing protocol recall with real-case vignettes and safety-constrained patient-context evaluation. Together, these directions imply that PCDT-QA is best understood as a foundational benchmark for Brazilian guideline-grounded clinical QA rather than a complete clinical safety assessment.

Source: https://www.emergentmind.com/topics/pcdt-qa