DischargeSim: Benchmark for Post-Visit Education

• DischargeSim is a simulation benchmark that evaluates LLMs as personalized discharge educators in the post-visit phase of clinical care.
• It models multi-turn dialogues between DoctorAgent and PatientAgent, incorporating structured discharge notes and patient profiles to assess communication and personalization.
• Key evaluations focus on dialogue quality, personalized document generation, and downstream patient comprehension, highlighting trade-offs between fluency and strategic communication.

DischargeSim is a simulation benchmark for evaluating LLMs as personalized discharge educators in the post-visit phase of clinical care, where the central problem is not diagnosis but whether patients understand what to do after leaving the hospital. It models multi-turn conversations between an LLM-based DoctorAgent and an LLM-based PatientAgent, and evaluates performance along three linked axes: dialogue quality, personalized document generation, and patient comprehension. In the benchmark’s framing, this shifts clinical LLM evaluation from in-visit diagnostic reasoning toward discharge education, personalization, and downstream understanding (Yao et al., 8 Sep 2025).

1. Clinical rationale and benchmark scope

DischargeSim was introduced to address a specific gap in clinical LLM evaluation: existing benchmarks such as AgentClinic, AMIE, and HealthBench emphasize diagnostic reasoning, uncertainty handling, and clinical inference during the visit, but do not meaningfully evaluate whether a model can explain discharge instructions clearly, adapt to literacy and emotional state, sustain a multi-turn educational dialogue, or generate useful written discharge materials. The benchmark also distinguishes itself from prior discharge-related work such as PaniniQA, which is described as limited to static, single-turn QA and as not modeling realistic dialogue, patient diversity, or downstream comprehension (Yao et al., 8 Sep 2025).

This positioning is substantive rather than nominal. The benchmark treats discharge as an educational setting with its own communicative demands: medication explanation, return precautions, follow-up planning, and self-management guidance. A plausible implication is that DischargeSim operationalizes discharge quality as a joint property of factual completeness, interpersonal adaptation, and teach-back-like retention, rather than as mere extraction of discharge-note content.

2. Scenario construction and agent architecture

The benchmark is built from 49 discharge notes sampled from MIMIC-IV. Each note is converted into a structured representation inspired by OSCE and AHRQ discharge guidance. The DoctorAgent has access to the structured discharge record and is responsible for educating the patient, while the PatientAgent is initialized with a psychosocial profile and may ask questions, misunderstand information, or react emotionally. The PatientAgent may also receive a partially masked memory representation to simulate imperfect recall (Yao et al., 8 Sep 2025).

The psychosocial profile varies along three controlled dimensions:

Dimension	Values
Health literacy	low; high
Education level	no high school; high school/GED; college graduate / bachelor’s degree
Emotional style	neutral; anxious; deflective / avoidant

These controlled profiles let the benchmark test whether models adapt to different patient needs and whether performance is equitable across populations.

The discharge interaction is organized around six clinically grounded topics:

Topic	Scope
Indications to return to the hospital / ED	return precautions
Medication information	prescribed drugs and instructions
Diagnosis	explanation of the condition
Post-discharge treatments / activity restrictions / home care	recovery and self-management
Treatments/tests during the hospital stay	recap of in-hospital care
Follow-up plans	appointments and monitoring

The paper notes that these topics are listed in slightly different order in different sections, but they are the same six discharge domains.

3. Dialogue protocol and communication objectives

DischargeSim is explicitly multi-turn and multi-stage rather than a one-shot question-answering task. The DoctorAgent initiates each stage. Each stage is capped at 5 dialogue turns, except medication, which has no upper limit to ensure complete coverage of all prescribed drugs. Because the PatientAgent’s responses depend on its profile, the same discharge note can generate materially different conversational trajectories (Yao et al., 8 Sep 2025).

The benchmark does not treat discharge quality as content transmission alone. It also requires the DoctorAgent to deploy six communication strategies adapted from patient-centered communication literature:

1. Fostering relationship
2. Gathering information
3. Providing information
4. Decision making
5. Enabling disease- and treatment-related behavior
6. Responding to emotions

This design embeds a strong distinction between factual coverage and communicative method. The reported results later show that these dimensions can diverge: models with strong fluency and coverage are not always the models with the strongest strategy use. That distinction is central to the benchmark’s interpretation of discharge education.

4. Evaluation framework

DischargeSim evaluates systems along three linked axes: dialogue quality, personalized document generation, and patient comprehension (Yao et al., 8 Sep 2025).

For dialogue quality, the benchmark uses an LLM-as-judge framework with DeepSeek-V3 as the judge. Two major dimensions are scored. Language and Delivery covers Linguistic clarity, Coherence, and Repetitiveness. Human-Centered Communication covers Personalization and empathy and Interaction appropriateness. The evaluation prompt is conservative and states that scores above 4.0 should be rare. The rubric bands are given as 5.0 = perfect, 4.0–4.9 = exceptional, 3.0–3.9 = competent, 2.0–2.9 = weak, and 1.0–1.9 = very poor. The LLM-as-judge setup was validated on 49 simulated cases with two licensed clinicians, yielding human inter-rater reliability of Cohen’s $\kappa = 0.715$, Percent agreement $= 85.7\%$, and Spearman’s $\rho = 0.720$ with $p < 0.001$; alignment between the LLM judge and experts was Cohen’s $\kappa = 0.591$, Percent agreement $= 79.6\%$, and Spearman’s $\rho = 0.593$ with $p < 0.001$.

For personalized document generation, the DoctorAgent produces two artifacts after the dialogue: a free-text personalized discharge summary and a structured AHRQ-style checklist. The free-text summary is evaluated with ROUGE-L, BLEURT, and UMLS-F1, with UMLS concepts extracted using scispaCy. The appendix defines UMLS-based factual overlap by precision and recall over the generated and reference concept sets, with F1 as the harmonic mean. The summary is also judged by DeepSeek-V3-Judge on eight dimensions: Fluency, Coherence, Informativeness, Personalization, Semantic Coverage, Factuality, Relevance, and Consistency. The structured output is a JSON-formatted checklist modeled after the AHRQ “Going Home: Questions to Ask After the Hospital” guide, and is scored with AHRQ ACC.

For patient comprehension, the benchmark uses a downstream multiple-choice comprehension exam. For each discharge note, clinical experts created 5–10 multiple-choice questions; the benchmark uses a 10-question exam covering all six discharge categories. The appendix describes the authoring process as involving PhD CS students who drafted items and three medical experts who reviewed and revised them: two nursing professors from the U.S. and one physician from South Korea. Comprehension is tested in two conditions: Dialogue Only, where the patient answers using only the simulated conversation, and Discharge Summary, where the patient answers using only the generated discharge summary. The metric is ACC.

5. Experimental results and model ranking

The benchmark evaluates 18 LLMs, including OpenAI, Qwen 2.5, and LLaMA 3 family models under unified prompts and agent settings. Across these experiments, performance varies widely across axes and across patient profiles, and the paper emphasizes that model size does not always yield better education outcomes (Yao et al., 8 Sep 2025).

The strongest reported results are summarized below.

Axis	Leading result(s)	Value
Dialogue quality	LLaMA3.3-70B	4.13 / 4.26
Dialogue quality	GPT-5-mini	4.07 / 4.28
Content coverage	GPT-5-mini	4.96
Content coverage	GPT-5-nano	4.82
Strategy average	GPT-4.1-mini	4.01
Dialogue-only exam ACC	GPT-5-mini	84.89%
Dialogue-only exam ACC	LLaMA3.1-70B	84.29%
AHRQ checklist ACC	GPT-5-mini	93.80%
Summary overlap	LLaMA3.1-70B	ROUGE-L = 0.3419; UMLS-F1 = 0.4251
Judged summary quality	GPT-5-mini	4.19 / 4.84; Exam ACC = 83.38%

Several patterns are notable. In dialogue quality, frontier models dominate: LLaMA3.3-70B achieves 4.13 / 4.26, GPT-5-mini achieves 4.07 / 4.28, Qwen2.5-72B reaches 4.03 / 4.26, and GPT-4.1-mini scores 3.92 / 4.27. At the lower end, Qwen2.5-0.5B scores 1.47 / 1.50. Content coverage is particularly strong for the GPT-5 models, with GPT-5-mini at 4.96 and GPT-5-nano at 4.82; the paper notes near-saturation on coverage metrics for return precautions, medications, diagnosis, and follow-up.

At the same time, the benchmark records a trade-off between fluent information delivery and patient-centered communication strategy. The highest strategy avg. is not achieved by GPT-5 but by GPT-4.1-mini at 4.01, followed by Qwen2.5-72B at 3.93, LLaMA3.3-70B at 3.83, and LLaMA3.1-70B at 3.74. The GPT-5 models are lower on this measure, with GPT-5-mini: 3.62 and GPT-5-nano: 3.65, with weaker performance particularly in Responding to Emotions, Gathering Information, and sometimes Enabling Behavior Change.

For downstream comprehension, which the paper treats as the hardest and most consequential metric, GPT-5-mini leads the Conversation-only exam accuracy at 84.89%, followed closely by LLaMA3.1-70B at 84.29%, GPT-4.1-mini at 82.78%, LLaMA3.3-70B at 82.18%, and GPT-5-nano at 79.15%. Within the Qwen family, accuracy rises from 52.87% at 0.5B to 74.02% at 72B, indicating a strong family-level scaling trend, though not one that eliminates broader trade-offs.

The summary-generation results also separate lexical overlap from judged clinical usefulness. LLaMA3.1-70B leads the automatic overlap metrics with ROUGE-L = 0.3419 and UMLS-F1 = 0.4251, whereas GPT-5-mini leads the judged summary metrics with Language appropriateness avg. = 4.19, Factuality/completeness avg. = 4.84, and Exam ACC = 83.38%. The paper presents this as evidence that lexical agreement with a reference summary is not equivalent to patient-facing educational quality.

6. Profile sensitivity, trade-offs, and limitations

One of the benchmark’s most important findings is that performance changes systematically with health literacy, education level, and emotional style (Yao et al., 8 Sep 2025).

Across models, performance is generally better with high-literacy patients than with low-literacy patients, especially in expressive dimensions such as Language & Delivery, Human-Centered Communication, and Strategy. The paper summarizes the relative literacy difference as

$(\text{HL2} - \text{HL1}) / \text{HL1},$

where HL2 denotes high literacy and HL1 denotes low literacy. Within the Qwen family, the average HL2-vs-HL1 difference correlates positively with model size at $r = 0.80$ with $= 85.7\%$0; for Language & Delivery, the correlation is $= 85.7\%$1 with $= 85.7\%$2, and for Human-Centered Communication, $= 85.7\%$3 with $= 85.7\%$4. The paper interprets this as indicating that larger models adapt more when the patient is easier to communicate with, while raising concern about robustness in low-literacy settings.

Performance also increases as education level rises from no high school to high school/GED to college graduate / bachelor’s degree. Here again, factual dimensions are more stable, while social and interactional dimensions vary more strongly. For emotional style, models respond differently to deflective, neutral, and anxious patients. In general, anxious patients elicit more supportive and strategic responses. Within the Qwen family, larger models are less reactive to emotional style in some anxious-vs-neutral comparisons, with negative correlations for Strategy: $= 85.7\%$5, Language and Appropriateness: $= 85.7\%$6, and Factuality and Completeness: $= 85.7\%$7. For neutral-vs-deflective comparisons, larger models are more effective at recognizing avoidant behavior, with Human-Centered Communication: $= 85.7\%$8 and Content: $= 85.7\%$9.

The benchmark identifies several non-obvious trade-offs. High fluency does not guarantee better education; best summary overlap is not the same as best downstream comprehension; some medium-sized models strike a better balance between strategy use and content completeness; and larger models are not always better at personalized discharge education. These findings are not presented as anomalies but as evidence that discharge education is a multidimensional task in which content prioritization, interpersonal calibration, and retention can come apart.

The paper also states several limitations. PatientAgents are scripted and LLM-generated, so they may not fully capture real-patient unpredictability. The benchmark is English-only and MIMIC-IV-centric. LLM-as-judge and MCQ exams are treated as incomplete proxies that may miss trust, nuance, and long-term understanding. Finally, the benchmark’s structured topics may constrain natural dialogue, since real discharge conversations are often more open-ended and patient-driven. Suggested future directions include human-in-the-loop evaluation, standardized patient actors, multilingual and cross-domain testing, real patient feedback, long-term behavioral outcome measures, and adaptive dialogue management that preserves safety coverage while allowing patient-driven flow.

In sum, DischargeSim defines discharge communication as a benchmarkable problem of personalized clinical education. Its core contribution is methodological: it evaluates whether an LLM can explain, adapt, document, and teach, rather than merely whether it can diagnose.