DocCHA: A Modular Conversational Health Agent

• DocCHA is a confidence-aware, modular conversational health agent that uses structured, multi-turn dialogue to enhance clinical diagnosis.
• It decomposes the diagnostic process into symptom elicitation, history acquisition, and causal graph construction, with confidence scores guiding adaptive follow-up questions.
• DocCHA demonstrates significant improvements in diagnostic accuracy and information recall by dynamically clarifying uncertain clinical inputs and reducing reasoning errors.

DocCHA is a confidence-aware, modular conversational health agent (CHA) framework that augments LLMs to perform transparent, adaptive, and efficient online clinical diagnosis via multi-stage, structured dialogue. Designed to overcome the static and brittle behavior of prior LLM-based CHAs, DocCHA systematically decomposes the diagnostic workflow into interpretable modules that enable multi-turn reasoning, dynamic symptom clarification, and explicit causal inference, with confidence scores guiding the dialogue strategy throughout. DocCHA has demonstrated significant improvements in diagnostic accuracy and information recall over strong LLM prompting baselines using real-world clinical consultation datasets (Liu et al., 10 Jul 2025).

1. High-Level Architecture

DocCHA embeds off-the-shelf LLMs such as GPT-4o or LLaMA-3 within a three-stage diagnostic agent, orchestrated by a controller that monitors and manages global dialogue state and module-specific confidence. The main system components are:

• Symptom Elicitation: Extraction and dynamic refinement of the patient's symptom set from user utterances.
• History Acquisition: Gathering contextual factors (medical, epidemiological, and social history).
• Causal Graph Construction & Refinement: Construction of a directed acyclic graph to explicitly chain the evidence across symptoms, history, and disease hypotheses.
• Final Diagnosis: Application of an LLM-based classifier over the explicit reasoning graph.

Each module calculates an interpretable confidence score. If this score falls below a configurable threshold and the dialogue turn quota for the module is not exhausted, the system triggers targeted follow-up questions to clarify or complete information. This iterative cycle is designed to emulate clinical reasoning, producing transparent and modifiable decision chains.

2. Modular Reasoning Pipeline and Confidence Metrics

2.1 Symptom Elicitation

• Inputs: The extracted symptom set $S_u = \{s_1, \ldots, s_n\}$ from the latest user utterances and the top-$K$ candidate diagnoses $\mathcal{D} = \{d_1, \ldots, d_K\}$ generated via LLM prompting.
• Processing:
  • Compute each symptom's discriminative power: $$DP(s_i) = \mathrm{Var}_{d_k \in \mathcal{D}}P(s_i|d_k)$$.
  • Assess coverage against canonical symptom lists $S_{d_k}$ for each candidate $d_k$.
  • Evaluate attribute completeness by comparing collected vs. required attributes for each symptom.
• Confidence Calculation:
  • Coverage: $$C_{\mathrm{cov}} = \max_{d_k \in \mathcal{D}} \frac{|S_u \cap S_{d_k}|}{|S_{d_k}|}$$
  • Detail: $$C_{\mathrm{det}} = \frac{1}{|S_u|} \sum_{s_i \in S_u} \frac{|A(s_i)|}{|A_{\mathrm{req}}(s_i)|}$$
  • Combined: $$C_{\mathrm{sym}} = 0.5\,C_{\mathrm{cov}} + 0.5\,C_{\mathrm{det}}$$
• Adaptive Strategy: If $C_{\mathrm{sym}} < \tau_{\mathrm{sym}}$, solicit information on the highest $K$0 symptom or missing attribute.

2.2 History Acquisition

• Inputs: Collected history facts $K$1 and the set of expected history categories $K$2 aggregated over candidate diagnoses.
• Metrics:
  • Coverage, relevance (cosine embedding similarity), and certainty (classification score from an auxiliary LLM model).
  • Confidence: $K$3
• Adaptive Strategy: If $K$4, generate clarifying questions.

2.3 Causal Graph Construction

• Input: Symptoms $K$5, history $K$6, and candidate $K$7. The LLM produces a directed acyclic graph $K$8 representing the hypothesized diagnostic reasoning chain.
• Scoring Components:
  • Coherence: Automated chain scoring (BARTScore).
  • Medical plausibility: $K$9 where $\mathcal{D} = \{d_1, \ldots, d_K\}$0 is the UMLS edge set.
  • Entailment: NLI model score.
  • Aggregate: $\mathcal{D} = \{d_1, \ldots, d_K\}$1
• Adaptive Strategy: When $\mathcal{D} = \{d_1, \ldots, d_K\}$2, query about the weakest edge in the inferred graph.

3. Dialogue Control and Adaptive Follow-Up

DocCHA’s dialogue orchestration follows a prioritized, confidence-driven intervention policy, implemented in a loop over modules:

$\mathcal{D} = \{d_1, \ldots, d_K\}$3

This approach ensures that follow-up questions are targeted to areas of uncertainty, yielding structured, efficient, and clinician-mimetic dialogues.

4. Dataset Construction and Implementation Details

• Training and Testing Data:
  • IMCS21 (100 tele-consultation transcripts, pediatric, Chinese origin)
  • DX (120 consultation cases, Chinese, annotated with UMLS categories)
• Processing: Transcripts translated to English (Google Translate API), then mapped to UMLS concepts.
• Model Backbone: No fine-tuning; DocCHA layers on GPT-4o and LLaMA-3 as plug-in LLMs.
• Infrastructure:
  • Prompt templates for each module (elicit, clarify, infer).
  • Controller enforces per-module quota (≤4 rounds/module), manages API calls, and computes confidence.
  • Embedding (mpnet-base-v2), inference models (MedNLI), and external resource integration (UMLS, SemMedDB).

5. Quantitative Performance and Ablation Analysis

Comparative Results on IMCS21 and DX

Method	Acc. (%)	Recall_info (%)	Turns (n)
LLaMA-3 Prompting	66.23 / 65.72	47.72 / 46.90	5.6 / 5.1
GPT-3.5 Prompting	87.69 / 85.44	52.04 / 51.75	6.4 / 5.9
GPT-4o Prompting	90.68 / 89.96	52.12 / 51.49	6.8 / 6.4
DocCHA (GPT-4o)	95.86 / 94.14	54.49 / 54.07	7.1 / 7.1

• On IMCS21: DocCHA outperforms GPT-4o by +5.18 percentage points in accuracy and +2.37 points in information recall, with only 0.3 more dialogue turns.
• On DX: Gains are +4.18 and +2.58, respectively.

Modular Ablation (IMCS21):

Ablation	Acc. (%)	Cosine sim.	Recall_info (%)	Turns
Full DocCHA	95.86	65.68	54.49	7.1
– Module 1 (symptom)	93.56	64.18	53.09	6.2
– Module 2 (history)	95.07	63.85	54.27	6.8
– Module 3 (causal)	94.29	63.62	54.06	6.6

Findings: The symptom elicitation module is most critical for overall performance. The causal graph module reduces hallucinated associations and sharpens reasoning quality. All modules contribute synergistically to accuracy and information recall.

6. Error Modes and Limitations

DocCHA’s empirical analysis reveals that:

• Over-focus may occur on weakly discriminative symptoms when discriminative power is erroneously estimated.
• Translation quality (Chinese→English) impacts overall performance.
• Fixed confidence thresholds and question quotas may need customization across medical specialties.

Formal uncertainty quantification (p-values, intervals) was not reported; instead, performance trends were robust across datasets.

7. Outlook and Prospective Directions

Key avenues for future research include:

• Real-world deployment beyond Chinese/English, including low-resource and multilingual environments.
• Integration of multimodal evidence (e.g., imaging, biosignals) into the structured reasoning pipeline.
• Adaptive calibration of thresholds and per-patient complexity adaptation.
• Human factors evaluation—prospective clinician trust and usability studies.

DocCHA establishes a generalizable, interpretable framework for LLM-powered clinical agents, unifying confidence-scored elicitation, history collection, and explicit causal reasoning toward trustworthy medical dialogue systems (Liu et al., 10 Jul 2025).