HPI Diagnostic Guideline Comparison

• HPI-diagnostic guideline comparison is an evaluation framework that maps patient histories to structured guidelines for assessing adherence and diagnosis accuracy.
• It employs decision-tree encoding and feature set extraction to convert narrative guidelines into measurable features for both AI and clinician evaluations.
• Benchmarking with models like GPT-4.1 demonstrates that precise guideline translation and targeted fine-tuning significantly enhance HPI scoring and diagnostic outcomes.

The HPI-diagnostic guideline comparison is an evaluation paradigm in computational clinical reasoning where the elicited History of Present Illness (HPI) from a patient (real or simulated) is explicitly compared to the structured features outlined in diagnostic guidelines. This method quantifies the extent to which an AI or clinician’s questioning surfaces diagnostically salient information, and assesses guideline adherence both directly (feature matching) and indirectly (diagnosis accuracy). Recent research has operationalized this paradigm in benchmark frameworks and auto-prompting systems that translate clinical practice guidelines into computational forms for use with LLMs, enabling rigorous multi-model and multi-specialty comparison of pre-consultation capability and adherence to expert guidelines (Deng et al., 7 Jan 2026, Seo et al., 7 Jan 2026).

1. Guideline Structuring and Representation

Diagnostic guidelines, typically authored as narrative recommendations by specialty societies, must be transformed into computationally tractable forms to enable HPI-guideline comparison. Two principal approaches have emerged:

• Decision-Tree Encoding: Narrative guidelines are converted into explicit decision trees, with nodes representing symptom checks, temporal conditions, or quantitative thresholds. Each path through the tree corresponds to a clinical decision or management pathway. Example: The prostate cancer guideline is operationalized by nodes for digital rectal exam findings and specific PSA cutoffs, leading to pathway assignments such as urgent referral or routine monitoring (Deng et al., 7 Jan 2026).
• Feature Set Extraction and Weighting: Guidelines are manually curated into sets of diagnostic features, each assigned a significance weight. For instance, in the breast cancer guideline, features like “palpable breast lump” and “bloody nipple discharge” are designated high-weight (2 points), while “asymmetry in size” and “systemic symptoms” are medium-weight (1 point). This enables granular scoring of HPI units against clinical priorities (Seo et al., 7 Jan 2026).

2. HPI-Guideline Matching Frameworks

The core evaluation mechanism involves mapping the extracted or simulated HPI to guideline features. The process typically follows these steps:

1. Simulation/Dialogue Collection: Doctor-agent LLMs conduct pre-consultation interviews with patient-agents, eliciting HPI via structured dialogue.
2. Atomic Unit Decomposition: Each doctor question and patient answer pair is decomposed into atomic presence/absence units, representing granular symptom or finding assertions.
3. Guideline Feature Alignment: A comparer model assigns each HPI unit to a guideline feature or “None of Above.”
4. Scoring: Metrics include:
   • Unweighted adherence: $S_{\text{unweighted}} = \frac{m_+}{|G|}$ where $m_+$ is the number of matched units and $|G|$ is the total feature count.
   • Weighted adherence: $S_{\text{weighted}} = \frac{W_+}{M}$, where $W_+$ is the sum of weights for matched features, and $M$ the total possible weight.

This structure enables rigorous, reproducible comparisons across models, specialties, and consultation lengths (Seo et al., 7 Jan 2026).

3. Quantitative Benchmarking and Model Performance

Benchmark datasets such as EPAG provide synthetic, clinician-vetted patient profiles spanning diseases and specialties, each linked to expert-curated guidelines. Models are evaluated on their ability to elicit high-value HPI and facilitate accurate diagnosis, with results tracked over standard dialogue turn-counts.

Model	Unweighted HPI Score	Weighted HPI Score	Diagnosis Top-1 Acc. (%)
Human Expert	4.35	7.29	68.24
GPT-4.1	4.82	8.12	74.56
Claude-3.7-Sonnet	4.59	8.12	69.23
Medgemma-4B (SFT)	4.19	7.22	65.93
Qwen2.5-7B	3.74	6.51	58.46

Key findings include:

• Proprietary LLMs (e.g., GPT-4.1) surpass human and medically fine-tuned open-source models on both adherence and diagnosis.
• Targeted supervised fine-tuning (SFT) on dialogue data greatly improves adherence and diagnostic accuracy for open-source models, sometimes enabling parity with or outperformance of larger counterparts.
• Increasing HPI unit quantity does not guarantee higher diagnostic accuracy; breadth may dilute focus unless high-weight features are prioritized (Seo et al., 7 Jan 2026).

4. Impact of Guideline Structure on LLM Reasoning

The nature of the diagnostic guideline directly influences LLM performance:

• Negation and Multi-Criteria: Guidelines for domains such as headache require correct parsing of “denies…” statements and multi-symptom thresholds (e.g., migraine diagnosis via $C_1 + C_2 + C_3 \geq 2$). Negation handling is a pronounced failure mode for LLMs, leading to safety-biased but coarse decision boundaries (e.g., perfect recall but multi-class F1 = 0.44) (Deng et al., 7 Jan 2026).
• Temporal Sequencing: Back pain guidelines require duration-aware reasoning (e.g., “pain onset within 12 weeks and not improving after 1–6 weeks”). Temporal complexity contributes to moderate reductions in fine-grained classification (multi-class F1 = 0.72) (Deng et al., 7 Jan 2026).
• Quantitative/Threshold Features: Prostate cancer and similar guidelines, operationalized via objective laboratory cutoffs (e.g., PSA $\geq 20\ \mu$g/L), yield unambiguous LLM branching and the highest pathway-assignment accuracy (F1 = 0.77) (Deng et al., 7 Jan 2026).

5. Case Studies and Error Modes

Empirical case studies illustrate subtle failure patterns:

• Modified dialogue length (e.g., one additional HPI unit describing severity) can alter model diagnostic sequence and bias away from the guideline’s correct pathway—for example, shifting from AKI to renal colic upon addition of pain severity, despite continued guideline-feature coverage (Seo et al., 7 Jan 2026).
• Exclusion vignettes (cases explicitly meant for guideline exclusion) reveal particular difficulty in negation-heavy guidelines, with F1 as low as 0.06 for exclusion in the headache domain (Deng et al., 7 Jan 2026).

A plausible implication is that models lacking hierarchical weighting or dynamic relevance scoring may overweight contextually irrelevant but guideline-matched features.

6. Cross-Linguistic and Multimodal Considerations

Language of consultation shapes model QA behavior. For example, Korean dialogues elicit broader HPI coverage and higher guideline scores, whereas English leads to higher diagnostic specificity but narrower symptom probing. This variation suggests a need for localization and tuning to optimize both breadth and diagnostic depth in multilingual settings (Seo et al., 7 Jan 2026).

Development of extensions to handle multimodal features (e.g., labs, imaging) is recommended to fully capture the complexity of guidelines outside pure text domains (Seo et al., 7 Jan 2026).

7. Recommendations and Future Directions

Key recommendations for improving HPI-guideline comparison and downstream diagnostic decision support include:

• Dynamic, real-time weighting of features to ensure models focus on high-impact symptoms during interactive HPI collection.
• Continued curation of domain-specific, expert-reviewed pre-consultation datasets for supervised fine-tuning.
• Integration of specialized modules for negation handling and temporal reasoning in guideline-to-prompt translation frameworks.
• Explicit evaluation and calibration of LLM questioning styles in multilingual and heterogeneous healthcare environments.
• Expansion of evaluation pipelines to multimodal settings for conditions whose guidelines require integration of lab, imaging, or other structured clinical data (Seo et al., 7 Jan 2026, Deng et al., 7 Jan 2026).