Exploring the Feasibility and Acceptability of AI-Mediated Serious Illness Conversations in the Emergency Department

Abstract: Serious illness conversations (SICs) align care with patients' values, goals, and preferences, yet they rarely occur in emergency departments (EDs), where time constraints and emotional burden often leave clinicians making high-stakes decisions without documented insight into what matters most to patients. We present a case study of ED GOAL-AI, a voice-based conversational agent for brief, structured values discussions with older adults in the ED, evaluated with 55 patients for feasibility and acceptability. Most participants completed the conversation and reported the interaction as acceptable and feasible, with ratings of feeling heard and understood comparable to clinicians. However, we also observed critical failure modes, including boundary violations such as hallucinated diagnostic statements, highlighting ethical and emotional risks. This work points to early promise for AI-mediated SICs while underscoring the need for careful boundary setting and participatory design before broader deployment.

• The paper demonstrates that ED GOAL-AI can mediate five-minute structured serious illness conversations in the ED, achieving 94.5% respectful ratings.
• It employs a fine-tuned, locally hosted LLM with integrated ASR and TTS to ensure a safe and efficient conversational workflow.
• The study shows 89% dialogue completion and comparable patient-reported communication quality, though rare lapses highlight the need for further refinement.

AI-Mediated Serious Illness Conversations in the Emergency Department: Feasibility and Acceptability of ED GOAL-AI

Introduction

Serious Illness Conversations (SICs) are clinically critical yet underdelivered interventions intended to elicit patient values, goals, and preferences, thus aligning medical interventions with what matters most to seriously ill individuals. Despite extensive evidence that early SICs reduce unwanted aggressive interventions, support end-of-life care quality, and curtail healthcare costs, their frequency and timing in acute care settings, particularly the Emergency Department (ED), remain suboptimal due to severe time constraints and emotional burden on clinicians. This paper introduces ED GOAL-AI—a locally hosted, fine-tuned LLM-driven voice conversational agent—designed to facilitate brief and structured values discussions with older adults in the ED, and rigorously evaluates its feasibility and acceptability through a clinical case study with 55 participants (2604.26214).

System Architecture and Workflow

ED GOAL-AI operationalizes a closed-domain, voice-only conversational agent leveraging Qwen2.5 7B fine-tuned via QLoRA with 500 synthetic dialogues modeled on authentic SIC transcripts. The pipeline integrates Whisper-based ASR for speech-to-text conversion, in-session conversational context via efficient key–value cache management, and Kokoro TTS with prosody modifications attuned for older adults. The agent is governed by a rigid system prompt: it defines agent boundaries (no diagnostic, prognostic, or treatment statements), embeds ethical and safety constraints, and institutes procedure for redirecting medical queries to clinicians with transitional acknowledgment.

The conversational structure tightly adheres to a condensed SIC script: five core questions adapted from the SIC guide, each delivered in a three-turn micro-dialogue (solicitation, acknowledgment, at most two clarification loops). The model is trained not only to instantiate the SIC protocol but also to provide brief, affect-appropriate empathetic responses when patients express emotional distress, reflecting established palliative care communication practices.

The system was deployed locally on a tablet device, optimizing for patient privacy, data governance, and latency, and sessions were administered in situ with research staff present solely for technical support, not content mediation.

Figure 1: A patient uses ED GOAL-AI on a tablet in the ED, with research staff available for technical support but not content intervention.

Study Design and Measures

Fifty-five English-speaking older adults (mean age: 74.3, 53% male, 89% White) were recruited from a tertiary hospital ED based on serious illness criteria, prior hospitalization, or age threshold. Inclusion rigor follows prior ED-based SIC feasibility studies. After consent, each participant engaged with ED GOAL-AI in a single, five-minute structured SIC session. Post-interaction measures included:

• Primary Outcome: Acceptability on four 5-point Likert items (acceptability, respectfulness, clarity, ease).
• Secondary Outcomes: Completion rate, episode duration, interruptions, non-responses, and the NQF-endorsed “Feeling Heard and Understood” scale comparing the AI and clinicians.

Procedural integrity was preserved: the research staff intervened only for technical failures or participant distress. Survey data was supported by ethnographic observation of breakdowns and misalignments.

Empirical Findings

Acceptability: Participant ratings showed a robust right-skew, with 94.5% reporting “completely respectful” interactions and over 90% giving positive (≥4) ratings for all dimensions except ease (65% “completely” easy), the latter likely reflecting topic difficulty over interface design.

Figure 2: Acceptability ratings across four dimensions reveal high perceived acceptability and respectfulness; most participants rated ED GOAL-AI as completely acceptable.

Feasibility: 89% of participants completed full dialogues; mean duration was 4.5 minutes (SD=2.3), aligned with pragmatic workflow requirements. Early technical failures (hallucinated medical statements, repetition loops attributable to cache implementation bugs) were contained post refinement, validating the agent’s session stability after initial tuning.

Feeling Heard and Understood: All participants reported at least moderate levels of feeling heard by the system, with no statistical difference from their primary physicians (mean difference not significant, p=0.129).

Failure Modes and Emotional Alignment: Notable boundary violation included one hallucinated diagnostic statement, exposing the risk for LLM-driven hallucination even under fine-tuned constraints. Some agent acknowledgments (“I completely understand”) were perceived as emotionally inadequate—mirroring HCI literature on “illusory empathy” and underscoring the difficulty of aligning LLM-generated affect with clinical empathy standards.

Discussion

This study establishes that, in a highly time-pressured ED environment, most older adult patients can meaningfully participate in brief, AI-mediated SICs using a locally maintained, domain-constrained LLM agent. Patient acceptability and perceived respectfulness are on par with, or in some dimensions numerically comparable to, clinician-facilitated SICs. These results position AI-powered CAs as scalable adjuncts for values elicitation in contexts where human bandwidth is inconsistent, confirming previous HCI findings that CAs can augment—but not replace—critical aspects of emotionally fraught patient communication [milne-ives_effectiveness_2020, huynh_applications_2026, hsu_precare_2025].

Yet, the emotionally and ethically fraught nature of SICs exposes a high-consequence risk landscape: even rare LLM hallucinations (diagnostic statements, repeated questioning) constitute critical boundary violations, with outsized impact on patient safety, autonomy, and trust. Emotional misattunement, though less frequent, highlights the intrinsic challenge of encoding nuanced empathy heuristics into LLM conversational flows. Observed failures reinforce existing taxonomies of LLM breakdowns in clinical dialogue and urge further work in both robust constraint engineering and fine-grained emotional response modeling [huang_survey_2025, lee_benefits_2023].

Practically, these findings advocate for rigorous, participatory design, extensive scenario fine-tuning (particularly for value-laden and emotionally charged clinical communication), and post-hoc review for rare but severe breaches. The case further elevates the importance of local hosting for privacy/security and empirically demonstrates the feasibility of rapid, high-fidelity deployment in the ED—provided strict boundary-adherence protocols and contingency plans are in place.

Theoretical and Practical Implications

The study provides operational evidence that LLM-powered agents, when locally deployed and aggressively fine-tuned via scenario-specific data, can mediate high-stakes, values-focused patient–AI dialogues with high user acceptability in real-world clinical environments. This constitutes a significant step towards the automation or semi-automation of parts of the SIC workflow currently infeasible in acute care without dedicated human resources.

However, the work forces a recalibration of expectations: technical “feasibility” alone is insufficient as a deployment criterion for high-stakes conversational AI. The ethical and emotional cost of rare failures, especially hallucinations and empathy misalignments, must be explicitly modeled, tested, and constrained. Participatory codesign and real-world post-deployment monitoring become mandatory adjuncts to traditional technical validation.

Future Directions

The work highlights several critical pathways for further research and implementation:

• Expansion to Diverse Clinical Settings: Generalizability must be tested in resource-heterogeneous EDs and across more diverse patient populations.
• Emotion Modeling and Empathetic Alignment: Future iteration should include real-world (not synthetic) SIC transcripts and advanced training on emotional attunement frameworks used in palliative care [childers_beyond_2023].
• Technical Safeguards: Stronger runtime bounding strategies, conversational state checking, and real-time “human-in-the-loop” fail-safe triggers are needed.
• Participatory Co-Design: Involving patients, clinicians, and ethicists in scenario generation and real-world testing is necessary to align system constraints with clinical values and patient autonomy.

Conclusion

ED GOAL-AI demonstrates that LLM-based conversational agents can achieve high feasibility and acceptability for mediating structured values discussions with seriously ill older adults in the ED, replicating human interaction in several core dimensions. However, even with fine-tuned, system-purpose-trained local models, the observed risks—rare but highly consequential hallucinations and emotional misalignments—require substantive controls beyond technical performance metrics. Responsible progress in clinical conversational AI must prioritize the minimization of ethical and emotional boundary violations through rigorous constraint architectures, context-specific fine-tuning, and participatory, human-centered design.

This case study serves as both a demonstration of technical viability and a cautionary example advocating for the design of AI systems as narrowly scoped tools within well-defined boundaries, supporting but not supplanting clinician and patient agency and safety.

(2604.26214)