---
title: 'DischargeSim: Benchmark for Post-Visit Education'
url: https://www.emergentmind.com/topics/dischargesim
type: topic
---

# DischargeSim: Benchmark for Post-Visit Education

DischargeSim is a simulation benchmark for evaluating large language models as **personalized discharge educators** in the post-visit phase of clinical care, where the central problem is not diagnosis but whether patients understand what to do after leaving the hospital. It models multi-turn conversations between an LLM-based **DoctorAgent** and an LLM-based **PatientAgent**, and evaluates performance along three linked axes: dialogue quality, personalized document generation, and patient comprehension. In the benchmark’s framing, this shifts clinical LLM evaluation from in-visit diagnostic reasoning toward discharge education, personalization, and downstream understanding [2509.07188].

## 1. Clinical rationale and benchmark scope

DischargeSim was introduced to address a specific gap in clinical LLM evaluation: existing benchmarks such as **AgentClinic**, **AMIE**, and **HealthBench** emphasize diagnostic reasoning, uncertainty handling, and clinical inference during the visit, but do not meaningfully evaluate whether a model can explain discharge instructions clearly, adapt to literacy and emotional state, sustain a multi-turn educational dialogue, or generate useful written discharge materials. The benchmark also distinguishes itself from prior discharge-related work such as **PaniniQA**, which is described as limited to static, single-turn QA and as not modeling realistic dialogue, patient diversity, or downstream comprehension [2509.07188].

This positioning is substantive rather than nominal. The benchmark treats discharge as an educational setting with its own communicative demands: medication explanation, return precautions, follow-up planning, and self-management guidance. A plausible implication is that DischargeSim operationalizes discharge quality as a joint property of factual completeness, interpersonal adaptation, and teach-back-like retention, rather than as mere extraction of discharge-note content.

## 2. Scenario construction and agent architecture

The benchmark is built from **49 discharge notes** sampled from **MIMIC-IV**. Each note is converted into a structured representation inspired by **OSCE** and **AHRQ discharge guidance**. The **DoctorAgent** has access to the structured discharge record and is responsible for educating the patient, while the **PatientAgent** is initialized with a psychosocial profile and may ask questions, misunderstand information, or react emotionally. The PatientAgent may also receive a partially masked memory representation to simulate imperfect recall [2509.07188].

The psychosocial profile varies along three controlled dimensions:

| Dimension | Values |
|---|---|
| Health literacy | low; high |
| Education level | no high school; high school/GED; college graduate / bachelor’s degree |
| Emotional style | neutral; anxious; deflective / avoidant |

These controlled profiles let the benchmark test whether models adapt to different patient needs and whether performance is equitable across populations.

The discharge interaction is organized around six clinically grounded topics:

| Topic | Scope |
|---|---|
| Indications to return to the hospital / ED | return precautions |
| Medication information | prescribed drugs and instructions |
| Diagnosis | explanation of the condition |
| Post-discharge treatments / activity restrictions / home care | recovery and self-management |
| Treatments/tests during the hospital stay | recap of in-hospital care |
| Follow-up plans | appointments and monitoring |

The paper notes that these topics are listed in slightly different order in different sections, but they are the same six discharge domains.

## 3. Dialogue protocol and communication objectives

DischargeSim is explicitly **multi-turn** and **multi-stage** rather than a one-shot question-answering task. The **DoctorAgent initiates each stage**. Each stage is capped at **5 dialogue turns**, **except medication**, which has **no upper limit** to ensure complete coverage of all prescribed drugs. Because the PatientAgent’s responses depend on its profile, the same discharge note can generate materially different conversational trajectories [2509.07188].

The benchmark does not treat discharge quality as content transmission alone. It also requires the DoctorAgent to deploy six communication strategies adapted from patient-centered communication literature:

1. **Fostering relationship**
2. **Gathering information**
3. **Providing information**
4. **Decision making**
5. **Enabling disease- and treatment-related behavior**
6. **Responding to emotions**

This design embeds a strong distinction between factual coverage and communicative method. The reported results later show that these dimensions can diverge: models with strong fluency and coverage are not always the models with the strongest strategy use. That distinction is central to the benchmark’s interpretation of discharge education.

## 4. Evaluation framework

DischargeSim evaluates systems along **three linked axes**: **dialogue quality**, **personalized document generation**, and **patient comprehension** [2509.07188].

For **dialogue quality**, the benchmark uses an **LLM-as-judge** framework with **DeepSeek-V3** as the judge. Two major dimensions are scored. **Language and Delivery** covers **Linguistic clarity**, **Coherence**, and **Repetitiveness**. **Human-Centered Communication** covers **Personalization and empathy** and **Interaction appropriateness**. The evaluation prompt is conservative and states that scores above **4.0** should be rare. The rubric bands are given as **5.0 = perfect**, **4.0–4.9 = exceptional**, **3.0–3.9 = competent**, **2.0–2.9 = weak**, and **1.0–1.9 = very poor**. The LLM-as-judge setup was validated on **49 simulated cases** with two licensed clinicians, yielding human inter-rater reliability of **Cohen’s \(\kappa = 0.715\)**, **Percent agreement \(= 85.7\%\)**, and **Spearman’s \(\rho = 0.720\)** with \(p < 0.001\); alignment between the LLM judge and experts was **Cohen’s \(\kappa = 0.591\)**, **Percent agreement \(= 79.6\%\)**, and **Spearman’s \(\rho = 0.593\)** with \(p < 0.001\).

For **personalized document generation**, the DoctorAgent produces two artifacts after the dialogue: a **free-text personalized discharge summary** and a **structured AHRQ-style checklist**. The free-text summary is evaluated with **ROUGE-L**, **BLEURT**, and **UMLS-F1**, with UMLS concepts extracted using **scispaCy**. The appendix defines UMLS-based factual overlap by precision and recall over the generated and reference concept sets, with F1 as the harmonic mean. The summary is also judged by DeepSeek-V3-Judge on eight dimensions: **Fluency**, **Coherence**, **Informativeness**, **Personalization**, **Semantic Coverage**, **Factuality**, **Relevance**, and **Consistency**. The structured output is a JSON-formatted checklist modeled after the **AHRQ “Going Home: Questions to Ask After the Hospital”** guide, and is scored with **AHRQ ACC**.

For **patient comprehension**, the benchmark uses a downstream **multiple-choice comprehension exam**. For each discharge note, clinical experts created **5–10 multiple-choice questions**; the benchmark uses a **10-question exam** covering all six discharge categories. The appendix describes the authoring process as involving **PhD CS students** who drafted items and **three medical experts** who reviewed and revised them: **two nursing professors from the U.S.** and **one physician from South Korea**. Comprehension is tested in two conditions: **Dialogue Only**, where the patient answers using only the simulated conversation, and **Discharge Summary**, where the patient answers using only the generated discharge summary. The metric is **ACC**.

## 5. Experimental results and model ranking

The benchmark evaluates **18 LLMs**, including OpenAI, Qwen 2.5, and LLaMA 3 family models under unified prompts and agent settings. Across these experiments, performance varies widely across axes and across patient profiles, and the paper emphasizes that **model size does not always yield better education outcomes** [2509.07188].

The strongest reported results are summarized below.

| Axis | Leading result(s) | Value |
|---|---|---|
| Dialogue quality | LLaMA3.3-70B | 4.13 / 4.26 |
| Dialogue quality | GPT-5-mini | 4.07 / 4.28 |
| Content coverage | GPT-5-mini | 4.96 |
| Content coverage | GPT-5-nano | 4.82 |
| Strategy average | GPT-4.1-mini | 4.01 |
| Dialogue-only exam ACC | GPT-5-mini | 84.89% |
| Dialogue-only exam ACC | LLaMA3.1-70B | 84.29% |
| AHRQ checklist ACC | GPT-5-mini | 93.80% |
| Summary overlap | LLaMA3.1-70B | ROUGE-L = 0.3419; UMLS-F1 = 0.4251 |
| Judged summary quality | GPT-5-mini | 4.19 / 4.84; Exam ACC = 83.38% |

Several patterns are notable. In dialogue quality, frontier models dominate: **LLaMA3.3-70B** achieves **4.13 / 4.26**, **GPT-5-mini** achieves **4.07 / 4.28**, **Qwen2.5-72B** reaches **4.03 / 4.26**, and **GPT-4.1-mini** scores **3.92 / 4.27**. At the lower end, **Qwen2.5-0.5B** scores **1.47 / 1.50**. Content coverage is particularly strong for the GPT-5 models, with **GPT-5-mini** at **4.96** and **GPT-5-nano** at **4.82**; the paper notes near-saturation on coverage metrics for return precautions, medications, diagnosis, and follow-up.

At the same time, the benchmark records a trade-off between fluent information delivery and patient-centered communication strategy. The highest **strategy avg.** is not achieved by GPT-5 but by **GPT-4.1-mini** at **4.01**, followed by **Qwen2.5-72B** at **3.93**, **LLaMA3.3-70B** at **3.83**, and **LLaMA3.1-70B** at **3.74**. The GPT-5 models are lower on this measure, with **GPT-5-mini: 3.62** and **GPT-5-nano: 3.65**, with weaker performance particularly in **Responding to Emotions**, **Gathering Information**, and sometimes **Enabling Behavior Change**.

For downstream comprehension, which the paper treats as the hardest and most consequential metric, **GPT-5-mini** leads the **Conversation-only exam accuracy** at **84.89%**, followed closely by **LLaMA3.1-70B** at **84.29%**, **GPT-4.1-mini** at **82.78%**, **LLaMA3.3-70B** at **82.18%**, and **GPT-5-nano** at **79.15%**. Within the Qwen family, accuracy rises from **52.87%** at **0.5B** to **74.02%** at **72B**, indicating a strong family-level scaling trend, though not one that eliminates broader trade-offs.

The summary-generation results also separate lexical overlap from judged clinical usefulness. **LLaMA3.1-70B** leads the automatic overlap metrics with **ROUGE-L = 0.3419** and **UMLS-F1 = 0.4251**, whereas **GPT-5-mini** leads the judged summary metrics with **Language appropriateness avg. = 4.19**, **Factuality/completeness avg. = 4.84**, and **Exam ACC = 83.38%**. The paper presents this as evidence that lexical agreement with a reference summary is not equivalent to patient-facing educational quality.

## 6. Profile sensitivity, trade-offs, and limitations

One of the benchmark’s most important findings is that performance changes systematically with **health literacy**, **education level**, and **emotional style** [2509.07188].

Across models, performance is generally better with **high-literacy patients** than with **low-literacy patients**, especially in expressive dimensions such as **Language & Delivery**, **Human-Centered Communication**, and **Strategy**. The paper summarizes the relative literacy difference as
\[
(\text{HL2} - \text{HL1}) / \text{HL1},
\]
where **HL2** denotes high literacy and **HL1** denotes low literacy. Within the Qwen family, the average HL2-vs-HL1 difference correlates positively with model size at **\(r = 0.80\)** with **\(p = 0.053\)**; for **Language & Delivery**, the correlation is **\(r = 0.97\)** with **\(p < 0.01\)**, and for **Human-Centered Communication**, **\(r = 0.86\)** with **\(p < 0.05\)**. The paper interprets this as indicating that larger models adapt more when the patient is easier to communicate with, while raising concern about robustness in low-literacy settings.

Performance also increases as **education level** rises from **no high school** to **high school/GED** to **college graduate / bachelor’s degree**. Here again, factual dimensions are more stable, while social and interactional dimensions vary more strongly. For **emotional style**, models respond differently to **deflective**, **neutral**, and **anxious** patients. In general, anxious patients elicit more supportive and strategic responses. Within the Qwen family, larger models are less reactive to emotional style in some anxious-vs-neutral comparisons, with negative correlations for **Strategy: \(r = -0.91\)**, **Language and Appropriateness: \(r = -0.73\)**, and **Factuality and Completeness: \(r = -0.64\)**. For neutral-vs-deflective comparisons, larger models are more effective at recognizing avoidant behavior, with **Human-Centered Communication: \(r = 0.45\)** and **Content: \(r = 0.59\)**.

The benchmark identifies several non-obvious trade-offs. **High fluency does not guarantee better education**; **best summary overlap is not the same as best downstream comprehension**; **some medium-sized models strike a better balance** between strategy use and content completeness; and **larger models are not always better at personalized discharge education**. These findings are not presented as anomalies but as evidence that discharge education is a multidimensional task in which content prioritization, interpersonal calibration, and retention can come apart.

The paper also states several limitations. **PatientAgents are scripted and LLM-generated**, so they may not fully capture real-patient unpredictability. The benchmark is **English-only and MIMIC-IV-centric**. **LLM-as-judge** and **MCQ exams** are treated as incomplete proxies that may miss trust, nuance, and long-term understanding. Finally, the benchmark’s structured topics may constrain natural dialogue, since real discharge conversations are often more open-ended and patient-driven. Suggested future directions include **human-in-the-loop evaluation**, **standardized patient actors**, **multilingual and cross-domain testing**, **real patient feedback**, **long-term behavioral outcome measures**, and **adaptive dialogue management** that preserves safety coverage while allowing patient-driven flow.

In sum, DischargeSim defines discharge communication as a benchmarkable problem of personalized clinical education. Its core contribution is methodological: it evaluates whether an LLM can explain, adapt, document, and teach, rather than merely whether it can diagnose.

Source: https://www.emergentmind.com/topics/dischargesim