Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

Abstract: Brazil's Unified Health System (SUS) relies on official clinical guidelines that define diagnostic criteria, treatments, dosages, and monitoring procedures for over 200 million citizens. Yet current LLMs perform poorly on this guideline-specific knowledge, and no benchmark evaluates clinical recall grounded in Brazilian Portuguese protocols. We address this gap by adapting Qwen2.5-14B-Instruct to the Brazilian clinical domain. From 178 official guidelines (~5.4M tokens), we generate ~70M tokens of synthetic data in three formats -- rephrases, wiki-style articles, and question-answer pairs -- using four generator LLMs. We then apply continual pre-training followed by Group Relative Policy Optimization (GRPO). We introduce HealthBench-BR, with 1,780 balanced true/false clinical assertions, and PCDT-QA, with 890 open-ended clinical questions scored by an LLM judge. Our best model achieves 83.9% on HealthBench-BR and 85.4% on PCDT-QA, outperforming GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Pro, and Google AI Overview's web-grounded RAG despite having only 14B parameters. Ablations show that generator diversity and reinforcement learning are critical to these gains. We release all datasets, benchmarks, and model weights to support reproducible clinical NLP research for Brazilian Portuguese. Code, data, and model weights are available at https://github.com/hugoabonizio/clinical-protocols-br

• The paper introduces a novel synthetic data pipeline that transforms 178 clinical guidelines into a diverse 70M-token corpus for effective LLM training.
• The model is adapted via continual pre-training followed by GRPO, achieving up to 85.4% accuracy on domain-specific benchmarks.
• Results show that full-parameter tuning outperforms methods like LoRA, ensuring robust in-domain performance and factual recall.

Teaching LLMs Brazilian Healthcare: Knowledge Injection from Official Clinical Guidelines

Motivation and Context

The Brazilian Unified Health System (SUS) is among the world's largest public health endeavors, serving over 200 million citizens through a set of official clinical guidelines that determine care within the system. Despite the recent surge in LLMs for medical reasoning, their parametric knowledge remains overwhelmingly English-centric and fails to encode protocol-specific details dictated by the SUS. The paper "Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines" (2605.01077) targets this domain gap. By programmatically transforming Brazil’s 178 clinical protocols into a large and diverse synthetic corpus, and following with continual pre-training and reinforcement learning, the authors systematically adapt Qwen2.5-14B-Instruct for Brazilian clinical practice in Portuguese.

Figure 1: The training pipeline includes data generation from 178 guidelines, continual pre-training, and GRPO; benchmarks evaluate protocol-specific recall.

Data Synthesis and Knowledge Injection

Direct pre-training on raw protocol text is data-limited and structurally suboptimal for LLM adaptation. Therefore, the authors implement a synthetic data pipeline, processing each clinical guideline as follows:

• Rephrase Generation: Protocols are paraphrased with preserved semantics but diverse surface forms.
• Wiki-style Article Generation: Guidelines are recast in expository style, enhancing the model's ability to reason with structured clinical knowledge.
• Question-Answer Pairs: From the train split, QA pairs with stepwise rationale are generated to explicitly supervise clinical reasoning.

Each data generation stage is replicated across four generator LLMs—GPT-4.1-mini, GPT-5-nano, GPT-OSS-20B, Qwen3-235B—to maximize lexical and stylistic diversity. The resulting synthetic dataset expands the original 5.4M-token corpus of clinical guidelines to approximately 70M tokens—critical for robust knowledge injection.

Benchmarks and Evaluation Protocol

The evaluation regime is anchored by two new benchmarks, HealthBench-BR and PCDT-QA, constructed from the guidelines:

• HealthBench-BR: 1,780 true/false assertion pairs, carefully designed with plausible distractors where a single critical detail (dose, interval, contraindication) is altered. The model must classify each as Verdadeiro (True) or Falso (False) with explicit justification.
• PCDT-QA: 890 open-ended clinical questions, evaluated via LLM-as-a-judge protocols against reference answers.

Both are split into train and test by protocol, so test performance reflects generalization to unseen guidelines encountered only as rephrased/wiki text, never QA pairs.

Figure 2: Left: HealthBench-BR uses minimal-pair assertions with subtle factual modifications. Right: PCDT-QA assesses open-ended clinical QA.

Modeling Protocol and Optimization

Parameter adaptation proceeds in two phases:

1. Continual Pre-Training (CPT): Qwen2.5-14B-Instruct undergoes additional causal LM training on the synthetic corpus, using standard AdamW optimization and a TPU v6-64.
2. Reinforcement Learning (GRPO): The CPT model is further adapted via Group Relative Policy Optimization on HealthBench-BR. The objective rewards correctness with detailed reasoning, directly targeting factual recall and anti-sycophancy.

Alternative adaptation methods such as LoRA and replay were benchmarked as catastrophic forgetting mitigation strategies, with the main results indicating full-parameter CPT yields the best in-domain and out-of-distribution trade-off.

Empirical Results

Main Findings

• Knowledge Absorption: The final model (CPT + GRPO + 4 generators) achieves 83.9% accuracy on HealthBench-BR and 85.4% on PCDT-QA (test splits). This decisively exceeds Gemini 3.1 Pro (79.3% / 80.0%), Claude Sonnet 4.6, and GPT-5.2—even with less than one-tenth the parameter count of GPT-4/5.
• Ablations: Format and generator diversity are strongly complementary; including all augmentation types and generators boosts PCDT-QA by +20 points over single-format/generator ablations.
• Reinforcement Learning Impact: GRPO adds +12.8 points to HealthBench-BR over CPT alone and mitigates sycophancy, correcting the model's increased baseline agreement tendency (from 58.9% to 53.0% "True" verdicts after RL, matching the gold 50% balance).
• Retrieval-Augmented Baselines: The parametric model outperforms web-grounded retrieval (Google AI Overview) by 13.4–15.1 points, highlighting the limitations of non-parametric access for protocol-centric medical QA.

Catastrophic Forgetting and Robustness

Full fine-tuning achieves higher in-domain knowledge injection with moderate out-of-distribution degradation, outperforming both LoRA (which exhibits detrimental low-rank interference) and replay (which helps only at lower scales). Therefore, the paradigm adopted finds a favorable point on the knowledge-retention frontier for protocol adaptation.

Implications and Limitations

Practical Consequences

By outperforming closed and web-augmented LLMs on the Brazilian clinical domain, the approach facilitates the deployment of trustworthy, low-resource, and reproducible inference for healthcare professionals who require native-language protocol compliance at the point of care. The public release of models and benchmarks will catalyze further research in Brazilian clinical NLP and lower barriers to the responsible adaptation of LLMs for non-English health systems.

Theoretical Implications

The results confirm that knowledge injection via multi-format, multi-generator synthetic corpora, coupled with targeted RL, is a highly effective strategy for clinical LLM adaptation—superior in outcome to parameter-efficient training. Moreover, this challenges prior claims regarding the general effectiveness of LoRA for continual pre-training and opens the door for additional research into the interaction between adaptation scale, format diversity, and parametric retention.

Future Directions

Key avenues for extension include scaling to real-world clinical encounters (beyond protocol recall), integrating multi-source Portuguese corpora as replay for robust alignment, and expanding evaluation with domain professionals. The framework may inform analogous adaptation strategies for healthcare LLMs in other countries with protocol-driven medical systems and less-resourced languages.

Conclusion

This work demonstrates that systematic, domain-grounded synthetic corpus expansion—leveraging both format and generator diversity—enables effective knowledge injection into open LLMs for the Brazilian clinical setting. The adapted model surpasses larger frontier baselines and retrieval-augmented solutions in protocol recall accuracy, validating both the pipeline and the new HealthBench-BR and PCDT-QA evaluations as foundational resources for Portuguese medical NLP. The public open sourcing of all assets further supports reproducible and equitable clinical AI development for Brazil and, more broadly, for healthcare NLP in non-English environments.