- The paper demonstrates the viability of LLM-based agents in healthcare using deterministic pre-LLM filtering and schema constraints to ensure safety.
- It details CareConnect’s multi-layered architecture, achieving a 91.8% task completion rate and 96.0% safety compliance in 680 simulated scenarios.
- The findings highlight substantial operational cost reductions and scalability benefits over traditional human receptionists in administrative tasks.
Toward Trustworthy LLM Agents in Healthcare: Technical Essay
Introduction
The deployment of LLMs as agentic systems in high-stakes domains necessitates rigorous safety and operational constraints, particularly in healthcare. "Toward Trustworthy LLM Agents in Healthcare" (2607.05055) addresses the significant gap between the capabilities of general-purpose LLMs and the requirements for reliably automating administrative tasks such as healthcare appointment scheduling. The paper presents CareConnect, a conversational AI agent designed to automate scheduling workflows while providing explicit safety and scope guarantees suitable for non-clinical healthcare environments.
System Architecture and Methodology
CareConnect integrates four principal components: a GPT-4o-based agent orchestrator with native function calling; a deterministic, pre-LLM intent classifier for safety-critical filtering; a hybrid RAG pipeline for operational information retrieval; and a tightly scoped suite of eight domain-specific tools for transactional scheduling and facility operations.
A distinctive feature of CareConnect is the multi-layered safety architecture. Pre-LLM pattern-matching blocks emergency, diagnostic, and medical advice intents in a multilingual manner, fundamentally reducing reliance on prompt-based or model-internal heuristics and the associated risks of hallucination and prompt injection. This deterministic filtering aligns with regulatory and auditable requirements and ensures that scope violations or inappropriate responses cannot reach users in the workflow.
The agent orchestrator employs schema-constrained tool invocation. Each tool—spanning appointment availability, booking, modification, cancellation, user data retrieval, provider lookup, and notification—is governed by strict JSON schemas and parameter validation against cached agent state. This enables stateful, multi-turn interaction with robust transactional integrity, preventing unauthorized, hallucinated, or inconsistent operations even under adversarial inputs or ambiguous dialogue flows.
For non-transactional informational queries, the RAG subsystem indexes facility logistics, provider metadata, and laboratory preparation texts, grounded via vector retrieval (using ChromaDB and text-embedding-3-large) and metadata filtering. CareConnect employs a fixed top-K document retrieval (K=5) to optimize contextual coverage and precision, integrating RAG results via the structured agent pipeline rather than direct model prompting.
Multi-modal interfaces—text (web and WhatsApp) and voice (web)—extend accessibility while unifying safety and tool orchestration logic on the backend. Speech-to-text is powered by Whisper, and TTS by OpenAI API, with latency and context management tailored per modality.
Experimental Results
Benchmarking and Scenario Design
Evaluation was conducted on a synthetic, privacy-compliant dataset with 30 patients and 60 providers, covering 680 systematically constructed scenarios. Scenarios include single- and multi-turn standard workflows, modification and cancellation, informational queries, safety enforcement, and edge-case handling with varying constraint complexity and ambiguity.
Quantitative Results
- Task Completion Rate: CareConnect achieved a 91.8% overall task completion rate (624/680), with performance across standard workflows (90.6%), modifications (88.5%), information retrieval (92.3%), safety compliance (96.0%), and edge cases (90.9%).
- Tool Call Accuracy: Tool invocation correctness was measured at 94.8% (3,283/3,462), with all operational errors intercepted by schema validation, preventing data inconsistency or corruption.
- Safety Compliance: On the 150-scenario safety-critical subset, CareConnect demonstrated 96.0% compliance, correctly identifying all emergency conditions, blocking medical advice requests, and rejecting diagnostic queries via deterministic refusal pathways.
- Latency: Median per-request end-to-end latency was 2.2 seconds, with 90th and 99th percentiles at 4.2 and 8.7 seconds, respectively. RAG queries exhibited modestly higher variance. Voice interaction added between 0.5 and 1.2 seconds due to ASR/TTS processing.
- Operational Cost: Per-appointment cost was measured at $0.0324, based on tokenized API usage, in contrast to an estimated$0.75 per interaction for human receptionists. This represents a 23x reduction in per-appointment cost, alongside 24/7 scalability and zero peak-time wait times.
Comparative Baseline
Relative to human receptionists (industry-estimated 85% success rate, 180s call duration, business-hour only), CareConnect surpasses human performance in completion rate and cost metrics while offering unbounded scalability and continuous availability.
Failure Mode Analysis
Of 56 scenario failures (8.2%), the predominant issues were unresolved constraint conflicts (37.5%), context drift in extended dialogues (32.1%), and ambiguous temporal references (19.6%). Critically, all failures were benign with respect to data and safety due to the layered validation architecture.
Discussion
The results substantiate the core proposition that strict, deterministic safety mechanisms—implemented before LLM invocation—are necessary and sufficient to robustly enforce operational scope in high-stakes agentic systems. The separation of transactional and informational workflows via hybrid tool-RAG architecture ensures both operational reliability and user-context richness. High compliance with safety and task correctness metrics demonstrates the viability of LLM-based agentic automation in healthcare logistics, provided structural validation and intent filtering are deeply integrated.
The cost structure and scalability profile reveal strong practical potential for deployment, though the synthetic nature of evaluation and lack of clinical data integration (e.g., HL7 FHIR, IRB governance) limit the immediate applicability in live health systems. The observed failures in constraint negotiation and long-turn context management suggest avenues for integrating more formal constraint satisfaction algorithms or extended dialogue state modeling in future iterations.
Theoretically, CareConnect’s architecture affirms that LLM agentic systems can, with deterministic safety and schema-enforcement, be safely tuned towards administrative automation tasks without incurring the risks endemic to clinical reasoning applications. As LLM capabilities grow, such architectures will likely generalize to a broader range of semi-structured operational domains where transactional safety is a hard requirement.
Implications and Future Work
CareConnect’s demonstrated safety, functional correctness, and efficiency have clear operational implications for healthcare scheduling and analogous administrative functions. The embedded safety guarantees—in contrast with model-based or learned guardrails—represent a significant step toward regulatory-compliant LLM agents deployable in environments with strict privacy and safety requirements.
Future directions include clinical pilot studies with live EHR integration, fine-grained preference and context learning for personalized scheduling, extension to multimodal and vision-based safety filtering, and methodological advances in constraint reasoning within conversational agents. System performance could further be optimized through fine-tuning of intent classification, more extensive linguistic coverage, and adaptive tool orchestration strategies for dynamic environments.
Conclusion
"Toward Trustworthy LLM Agents in Healthcare" formalizes a robust, safety-first approach to LLM-based administrative automation in healthcare, achieving high task completion, strong safety compliance, and substantial operational cost savings. The architecture—combining deterministic, pre-LLM filtering with schema-constrained agentic tool use and RAG—constitutes a compelling blueprint for safe, auditable LLM agent design in other regulated domains. Continued real-world validation and extension toward broader workflow automation remain critical next steps for the practical adoption of such systems.