Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications

Abstract: Recent advances in LLMs and multi-agent systems have driven the rise of Agentic AI, showing promise for medical reasoning. However, open-ended conversational agents remain prone to two critical failure modes: premature diagnostic handoff and silent clinical hallucinations that may go undetected before reaching the patient. In this work, we propose a multi-agent framework that addresses both issues by replacing ``LLM-as-a-judge'' routing with deterministic orchestration constraints. The framework incorporates two safety mechanisms. First, a neuro-symbolic state-tracking gate enforces completeness of the OLDCARTS clinical protocol (Onset, Location, Duration, Character, Aggravating/Alleviating factors, Radiation, Timing, and Severity) by blocking diagnostic transitions until all required dimensions are collected. Second, an epistemic uncertainty quantification (UQ) gate computes semantic entropy (H) across K=5 independent diagnostic samples to identify and intercept divergent outputs before delivery. We evaluate the system using simulated patient agents powered by the llama-3.1-70b-instruct model on 150 test cases. The full architecture achieves 49.3% diagnostic precision, representing an absolute improvement of 11.3 percentage points over an unconstrained baseline. Additionally, we observe a statistically significant negative correlation (r = -0.181, p < 0.05) between OLDCARTS completeness (σ) and semantic entropy (H), suggesting that structured information gathering is associated with reduced diagnostic uncertainty.

• The paper presents a novel multi-agent framework that integrates neuro-symbolic intake gating with semantic entropy-based uncertainty quantification to ensure complete symptom collection before diagnostics.
• The paper demonstrates that coupling structured OLDCARTS protocols with parallel diagnostic agents improves accuracy by 11.3 percentage points compared to baseline methods.
• The paper shows that increased completeness in clinical intake significantly reduces diagnostic uncertainty, addressing premature handoff and silent hallucination issues.

Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications

Introduction

The deployment of LLM-driven multi-agent frameworks in healthcare is hampered by critical reliability deficits in open-ended conversational agents, notably premature diagnostic handoff and silent hallucination. The former occurs when an agent transitions to diagnostic reasoning before capturing all clinically requisite symptom attributes, while the latter refers to undetected, potentially unsafe output errors such as incorrect medication recommendations absent explicit uncertainty signals. This work proposes and evaluates a deterministic multi-agent workflow integrating neuro-symbolic intake gating and uncertainty quantification gates to systematically mitigate these failure modes.

Figure 1: Failure modes typical to unconstrained LLM-based clinical agents, including (left) incomplete OLDCARTS coverage before handoff and (right) hallucinated recommendations without epistemic uncertainty screening.

System Architecture and Methodology

The proposed architecture consists of three orchestrated phases:

1. Structured History Taking with OLDCARTS Neuro-Symbolic Gate (M1): Intake is driven by the OLDCARTS protocol, with a state-tracking gate enforcing the collection of all eight OLDCARTS symptom attributes (Onset, Location, Duration, Character, Aggravating/Alleviating factors, Radiation, Timing, Severity) before allowing transition to diagnostic reasoning.
2. Parallel Uncertainty-Aware Diagnosis with Semantic Entropy Gate (M2): After intake completion, $K=5$ independent diagnostician agents generate differential diagnoses in parallel. Their outputs are clustered by NLI entailment to form semantically distinct diagnostic label clusters, from which normalized Shannon entropy ($H$) is computed to quantify epistemic uncertainty as disagreement across diagnostic samples.
3. Recursive Safety Supervision:

Diagnostic outputs flagged with high $H$ (above a fixed threshold $\tau$) are routed for explicit review and possible revision by a Medical Safety Supervisor, enabling a controlled recursive refinement loop with up to three attempts.

Figure 2: Modular architecture illustrating the three-stage pipeline—structured symptom collection with the OLDCARTS gate [M1], semantic entropy-based uncertainty quantification [M2], and recursive safety review—implemented via deterministic multi-agent orchestration.

Unlike purely prompt-controlled or LLM-as-a-Judge approaches, this design enforces deterministic intake completeness, explicit uncertainty quantification, and gated supervision, ensuring critical transitions and outputs are subject to symbolic and probabilistic constraints.

Experimental Setup

The evaluation leverages 150 diagnostic vignette-style cases from the MedQA-USMLE dataset, using llama-3.1-70b-instruct for all agents. Diagnostic precision, completeness of OLDCARTS intake (mean $\sigma$-score), and the correlation between symptom completeness and downstream entropy are the primary endpoints. Experimental conditions include:

• Baseline: No symbolic gate, $K=1$ deterministic diagnosis.
• Ablation A: OLDCARTS gate enabled, $K=1$ deterministic diagnosis.
• Full Architecture: OLDCARTS gate enabled, $K=5$ stochastic diagnosis and semantic entropy-based uncertainty quantification.

Results

Failure Mode Mitigation and Intake Completeness

Without OLDCARTS gating, mean $\sigma$-score stagnates at $\sim$2.4/8 (30%), signifying that unconstrained agents transition to diagnosis after collecting only a minority of requisite attributes. Enabling the symbolic gate increases intake completeness to 6.67/8 (83.3%), with the remaining gap attributed primarily to the maximum turn constraint during triage dialogues.

Diagnostic Accuracy and Ablation Analysis

The Full Architecture yields an 11.3 percentage-point gain in mean diagnostic accuracy relative to baseline (49.3% vs 38.0% at $H$0). Notably, enforcing OLDCARTS completeness in isolation (Ablation A) does not improve accuracy and can in fact degrade it, highlighting the necessity of coupling structured intake and uncertainty-aware gating to realize performance gains—multi-agent orchestration is necessary for synergistic improvements.

Figure 3: Diagnostic accuracy and intake completeness across varying architecture components, showing substantial gains only when both gates are active; accuracy does not improve with intake gating alone.

Relationship Between Symptom Completeness and Diagnostic Uncertainty

Statistical analysis reveals a significant negative correlation (Pearson $H$1, $H$2) between OLDCARTS completeness and semantic entropy $H$3. This suggests that more complete clinical histories constrain the diagnostic space, reducing divergent model generations and thus epistemic uncertainty. The effect size is small, consistent with the indirect, mediated relationship between intake protocol adherence and output calibration.

Figure 4: Regression of OLDCARTS completeness ($H$4) against semantic entropy ($H$5), confirming significant inverse correlation; greater intake completeness produces lower diagnostic uncertainty.

Theoretical and Practical Implications

This framework demonstrates that pure prompt- or LLM-as-a-judge strategies cannot assure reliable clinical intake or adequately filter hallucination. The integration of symbolic gating (OLDCARTS state tracker) and semantic uncertainty quantification addresses structural gaps in LLM-based clinical systems, allowing for explicit system-level guarantees over workflow transitions and enabling the detection and interception of high-uncertainty outputs.

From an applied perspective, such deterministic multi-agent architectures are critical for operationalizing LLMs in safety-critical domains. They offer modularity for future extension (e.g., richer clinical ontologies, real-world patient-agent interaction) and provide a foundation for more robust clinical reasoning pipelines. On the theoretical side, the observed linkage between structured data acquisition and output entropy quantifies how protocol-driven agentic workflows can reduce LLM epistemic uncertainty, with implications for future agent calibration mechanisms.

Future Directions

Open directions include optimizing the triage dialog turn budget, systematically tuning $H$6 thresholds for uncertainty routing, and integrating clinician-in-the-loop evaluation for real-world deployment. Adapting this deterministic multi-agent paradigm to other complex domains (e.g., multi-modal diagnostic reasoning, longitudinal care trajectories) may further enhance safe AI adoption in healthcare.

Conclusion

This work advances the practical and methodological foundation of LLM-driven agentic clinical systems. By enforcing deterministic intake completeness and quantifying epistemic uncertainty via semantic entropy, the proposed architecture demonstrably reduces critical failure modes—premature diagnostic handoff and silent hallucination. Diagnostic accuracy improvements and reduced output divergence support the efficacy of explicit neuro-symbolic orchestration. This approach marks a substantial progression toward robust, safety-critical multi-agent AI in clinical decision support.

---

Reference: "Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications" (2606.18068)