CPGBench: Evaluating LLMs for Clinical Guidelines

• CPGBench is an automated benchmark that evaluates LLMs’ ability to detect and apply clinical practice guidelines in multi-turn conversations.
• It leverages a decade-scale corpus from multiple countries and specialties, using structured extraction and human-validated accuracy to ensure quality.
• The benchmark separates guideline detection, title grounding, and adherence, exposing a know–do gap that underscores challenges for clinical deployment.

CPGBench is an automated benchmark for evaluating LLMs’ ability to detect and adhere to Clinical Practice Guidelines (CPGs) in realistic multi-turn clinical conversations. It was introduced to address a gap in prior evaluation settings, which were described as narrow in scope, often limited to small numbers of guidelines, a single country, question-answer format, no multi-turn dialogue, and expert curation. The benchmark operates at guideline-recommendation granularity, covers a decade-scale corpus of CPGs, and measures two distinct capabilities: whether a model can recognize which recommendation is implicated in a conversation, and whether it can correctly apply that recommendation when generating the next clinical turn (Tan et al., 26 Mar 2026).

1. Scope, motivation, and benchmark design

Clinical practice guidelines are framed as central to evidence-based decision-making and improved patient outcomes. CPGBench is motivated by the observation that clinical deployment requires both recognition of the applicable guideline and correct application of that guideline in dialogue. The benchmark therefore targets both “detection” and “adherence” in multi-turn conversations rather than limiting evaluation to isolated question answering (Tan et al., 26 Mar 2026).

Its coverage is explicitly broad. The benchmark spans guidelines published from 2015 to Aug 2025, includes material from USA, Canada, UK, Germany, Australia, Japan, Mainland China, Hong Kong, Taiwan, WHO, and ESNM, and covers all 24 specialties defined by the American Board of Medical Specialties. The stated objective is not only one-off evaluation but also automated scoring via a “Judge-LLM,” enabling continuous benchmarking of new models and newly published guidelines.

The benchmark is also positioned as a response to limitations in prior systems such as AMEGA, PromptGuide, MedGuide, and NICE-RAG. In that framing, CPGBench shifts evaluation toward large-scale, geographically diverse, specialty-spanning, recommendation-level assessment under dialogue conditions. A plausible implication is that the benchmark is intended to stress capabilities closer to clinical deployment constraints than narrow prompt-response tests.

2. Dataset construction and structured representation

The dataset construction pipeline begins with a raw crawl of 6 115 documents from official national or regional bodies, ECRI, WHO, ESNM, and related sources. An LLM-based filtering stage, identified as Prompt 2 with GPT4o, classifies bona fide CPGs versus consensus statements or position papers, yielding 4 792 high-quality CPGs. A subsequent date filter retaining only 2015+ publications produces the final set of 3 418 CPGs (Tan et al., 26 Mar 2026).

From these documents, the benchmark extracts 32 155 clinical recommendations. Extraction is described as structured, using Prompts 4–6. Each record includes the full recommendation text under “recomm,” a “strength” field containing evidence level and recommendation grade such as “Grade A, strong,” and a decomposition into “Context,” “Action,” and “Goal.” Metadata includes title, institute, publication date, country, and specialty.

The distribution is reported both by region and specialty. By country, Canada accounts for 22.3 %, the UK 21.7 %, Germany 19.2 %, the USA 13.3 %, Australia 11.9 %, and others each contribute less than 10 %. By specialty, the top three are Pediatrics at 19.4 %, Internal Medicine at 13.5 %, and Preventive Medicine at 10.1 %. Because all 24 ABMS categories are represented, the benchmark is structured for cross-specialty stratification rather than a single-domain stress test.

Human validation is used to assess extraction quality. In the reported spot-check, 106 documents and 1 160 recommendations were checked by clinicians or post-graduates, with at least 94 % accuracy across all fields. This does not remove the possibility of pipeline error, but it indicates that the structured representation was not accepted solely on automatic processing.

3. Task definitions and evaluation metrics

CPGBench defines two core tasks. The first is guideline detection. The input is a full multi-turn conversation synthesized to include exactly one target recommendation. The output is a list of detected recommendations together with their source titles. Detection is measured using two metrics:

$$\text{Accuracy}_{\text{detect}}= \frac{\#\ \text{correctly detected recommendations}}{\text{total \# recommendations}}$$

and

$$\text{Title\_grounding\_rate}= \frac{\#\ \text{correctly cited guideline titles}}{\text{total \# recommendations}}.$$

The second task is guideline adherence. Here the input is a truncated conversation ending just before the clinician’s application of the recommendation, and the evaluated model must generate the next turn. Scoring asks whether the response includes the key content of the ground-truth recommendation. The metric is:

$$\text{AdherenceRate}= \frac{\#\ \text{recommendations correctly applied}}{\text{total \# recommendations}}.$$

This separation between content detection, title grounding, and adherence is central to the benchmark’s design. Detection tests whether a model can identify the applicable recommendation. Title grounding tests whether it can correctly reference the source title. Adherence tests whether it can operationalize the recommendation in dialogue. The reported results emphasize that these are not interchangeable competencies.

4. Multi-turn conversation synthesis

For each extracted recommendation, CPGBench generates one multi-turn conversation. The conversation generation model is DeepSeek-R1 with 671 B parameters, described as open-source (Tan et al., 26 Mar 2026). Prompt design is attributed to Prompts 7–9 and includes the structured fields context, action, goal, strength, title, and country.

Generation is constrained by five criteria: communication flow and naturalness, empathy and rapport, clinical realism, patient authenticity, and geographical specificity. The output format is a JSON list of objects of the form {"role":"user"/"assistant","content":…}. A marker, <recommendation id>, is inserted at the first assistant turn that applies the recommendation. The benchmark further requires that the prior speaker at that point be the “user,” so that the truncated dialogue constitutes a valid prompt for the adherence task.

These design choices matter because the benchmark is not only about static recommendation matching. The conversations are constructed to emulate application of a recommendation within an evolving interaction. This suggests that CPGBench is intended to evaluate whether models can bridge from encoded medical knowledge to contextually appropriate conversational action, rather than merely reproduce standalone guidance statements.

Conversation quality is also subject to human review. A sample of 1 500 dialogues was evaluated by 56 clinicians across 24 specialties for inclusion, background suitability, and non-anomaly, with at least 99 % rated at least 0.5. The reported criterion is broad rather than a fine-grained realism score, but it functions as evidence that the synthetic dialogues were judged acceptable for benchmark use.

5. Evaluation pipeline and human validation

Eight leading LLMs are benchmarked: Llama3-8B-inst, Qwen3-4B-inst, Qwen3-32B, Huatuo-o1-7B, Baichuan-m2-32B, DeepSeek-R1, GPT4o, and GPT5. The benchmark groups them into open-source general models, open-source medically tuned models, and proprietary models. Automatic evaluation uses GPT4o as the Judge-LLM. Prompts 12–13 evaluate detection and title grounding, and Prompt 14 evaluates adherence.

The scale of the evaluation is large: 32 155 dialogues produce 514 480 model outputs and 771 720 Judge-LLM analyses. This scale is part of the benchmark’s stated rationale for automation. Manual adjudication at the same breadth would be difficult to sustain for continuous benchmarking over newly released models and guidelines.

Agreement with human evaluation is reported using Cohen’s $\kappa$. Judge-LLM agreement with humans is $\kappa=0.62$ for content detection, $\kappa=0.62$ for title grounding, and $\kappa=0.70$ for adherence. Inter-human agreement is described as moderate to substantial on all tasks, with $\kappa$ coefficients ranging from 0.53 to 0.81 (Tan et al., 26 Mar 2026).

These figures support the use of the automatic pipeline while also delimiting its precision. The benchmark explicitly notes subjectivity in scoring open-ended outputs and that $\kappa < 1$. Accordingly, the automated Judge-LLM is presented as validated but not as an error-free oracle.

6. Empirical findings

The overall results on 32 155 recommendations show a clear separation among the measured competencies. GPT5 attains 79.47 % content detection, 29.68 % title grounding, and 63.18 % adherence, with an average of 57.44 %. DeepSeek-R1 attains 89.62 % content detection, 13.24 % title grounding, and 50.01 % adherence. Qwen3-32B records 84.77 %, 7.96 %, and 45.89 %, while GPT4o records 82.65 %, 7.91 %, and 41.75 %. Baichuan-m2-32B, Qwen3-4B-inst, Huatuo-o1-7B, and Llama3-8B-inst all show the same qualitative pattern: materially higher content detection than title grounding, and adherence below detection (Tan et al., 26 Mar 2026).

The paper summarizes this as a persistent “know–do” gap. Detection substantially exceeds adherence for every model. The data block highlights GPT5 as an example, with 79.5 % versus 63.2 %. Title citation is especially weak, ranging from 3.6 % to 29.7 %, which is described as opening trust issues. The central empirical point is that models often recover the content of a guideline without being able to identify its source title or consistently apply it in dialogue.

Stratified analyses show that detection varies little across specialties, with variance less than 0.0033, whereas adherence varies more substantially. The highest adherence is reported in Preventive Medicine and Allergy and Immunology, and the lowest in Radiology and Thoracic Surgery. Regionally, detection is at least 65 % in all regions. Title grounding is highest in UK, USA, and WHO documents, and near zero for small institutes such as ESNM.

A separate safety-critical subset contains 6 632 recommendations. On this subset, detection ranges from 77.6 % to 93.6 %, which is higher than on the full set; title grounding is slightly lower; and adherence ranges from 16.7 % to 61.3 %, which is lower than on the full set. The benchmark interprets this as demonstrating acute risk if LLMs are used unsupervised in safety-critical scenarios. This suggests that improved factual recognition alone does not guarantee safer behavior where correct recommendation application is most consequential.

7. Limitations, failure modes, and implications for deployment

The benchmark identifies several failure modes. Post-release guidelines can sometimes appear to be “adhered” to because a model relies on prior guideline versions, general medical knowledge, socio-emotional heuristics, or looseness in Judge-LLM scoring. The pipeline may also propagate LLM processing errors, including extraction mistakes and ambiguous matching. These points matter because CPGBench is an automated end-to-end framework rather than a purely manually curated test set (Tan et al., 26 Mar 2026).

Several limitations are stated directly. The automated pipeline may introduce noise despite human spot-checks. Scoring open-ended outputs involves subjectivity. Synthetic dialogues cannot cover the infinite diversity of real patient interactions. These limitations do not negate the benchmark’s findings, but they constrain interpretation: benchmark scores should not be treated as exhaustive estimates of clinical performance in practice.

The proposed improvements are correspondingly targeted. The paper recommends enhancing multi-turn, guideline-grounded reasoning in training, incorporating explicit citation verification and penalties for hallucinations, and developing adaptive prompting or retrieval-augmented pipelines to improve title grounding. For deployment, the conclusion is explicit that LLMs should be treated as supportive tools under human supervision, not autonomous decision-makers, especially in safety-critical settings.

One common misconception would be to equate high detection with safe clinical utility. The reported results do not support that interpretation. Another would be to infer that correct guideline content reproduction implies reliable source attribution. The low title grounding rates directly contradict that. CPGBench’s main contribution is therefore not merely a benchmark scorecard, but a structured demonstration that knowledge recovery, source grounding, and conversational application are separable dimensions of model behavior in clinical guidance settings.