NoteAid-Chatbot: Conversational EHR Education AI

• NoteAid-Chatbot is a patient-education conversational AI that transforms EHR discharge summaries into interactive, multi-turn dialogues.
• It utilizes a lightweight LLaMA 3.2-3B model fine-tuned with LoRA and PPO to optimize explanations and answer patient follow-up questions.
• Evaluations show improved comprehension scores nearing expert educator levels, despite challenges like occasional hallucinations.

NoteAid-Chatbot is a patient-education conversational AI system centered on helping patients understand electronic health record (EHR) notes, especially discharge summaries, through interactive dialogue rather than passive reading. In the NoteAid line of work, it extends the original web-based NLP system that links medical terms to lay definitions into a conversational framework that can both explain selected EHR spans and answer patient follow-up questions, and it later becomes a “learning as conversation” system built on a lightweight LLaMA 3.2-3B model trained with synthetic conversations and Proximal Policy Optimization (PPO)-based reinforcement learning (Zhang et al., 2023, Jang et al., 6 Sep 2025).

1. Lineage, scope, and conceptual identity

The NoteAid lineage begins with a web-based NLP system whose original purpose was to link medical terms in EHR notes to lay definitions. Its core components were CoDeMed, a vocabulary of lay definitions for medical terms, and MedLink, a component that identifies medical terms in text and links them to CoDeMed definitions. Later work added MedJEx and README to better identify jargon and generate definitions. The later conversational systems arise from a stated limitation of that original design: it automatically selects jargon to define, remains essentially one-way and word- or phrase-level, and does not support follow-up questions or multi-turn explanation (Zhang et al., 2023).

Within this progression, two distinct but connected systems define the modern meaning of NoteAid-Chatbot. The NoteAid EHR Interaction Pipeline (NIP) frames the problem as two patient-centered tasks: Explanation, in which a patient-selected EHR span is explained in patient-friendly language, and Q&A, in which patient questions about the EHR are answered consistently with the note. The later NoteAid-Chatbot reframes the same general objective as an outcome-optimized conversational educator trained to maximize patient comprehension in simulated discharge scenarios (Zhang et al., 2023, Jang et al., 6 Sep 2025).

Aspect	NoteAid EHR Interaction Pipeline	NoteAid-Chatbot
Primary task	Explanation and Q&A around EHR notes	Multi-turn patient education from discharge notes
Core mechanism	Mock Patient Agent + Assistant Agent	Educator agent + patient agent + PPO
Main model basis	GPT-3.5-Turbo and GPT-4	LLaMA 3.2-3B + LoRA + PPO

This suggests a transition from using large models primarily as generators of patient-centered explanations and synthetic interaction data to using a lightweight, domain-specific model as the deployable educator policy. The continuity across both stages is the insistence that note understanding is not merely a terminology-linking problem, but a conversational one.

2. NoteAid EHR Interaction Pipeline

The NoteAid EHR Interaction Pipeline simulates patient–assistant interaction around a single EHR note using two LLM agents: a Mock Patient Agent and an Assistant Agent. The Mock Patient reads the EHR note and either asks questions that a patient might ask or selects spans that a patient might find challenging. The Assistant also sees the note and produces either an answer or an explanation. Each instance contains three rounds of interaction, with previous dialogue context carried across rounds (Zhang et al., 2023).

The two formal tasks are defined over a note $N$, patient requests $X$, responses $Y$, and prompts $P$. The pipeline defines prompt construction recursively:

$$\begin{cases} p_1^i = f_{\text{init}}(n_i, x_1^i) \ p_k^i = f(p_{k-1}^i, y_{k-1}^i, x_k^i), \quad k \geq 1 \end{cases}$$

and response generation:

$$\begin{cases} y_1^i = M_{\text{large}}(f_{\text{init}}(n_i, x_1^i)) \ y_k^i = M_{\text{large}}(f(p_{k-1}^i, y_{k-1}^i, x_k^i)), \quad k \geq 1 \end{cases}$$

where $x_k^i$ is either a patient question or a selected EHR span, and $M_{\text{large}}$ is instantiated with GPT-3.5-Turbo or GPT-4 (Zhang et al., 2023).

Prompting is explicitly patient-centered. For Q&A, the Mock Patient is told: “Try to mock as the a patient and ask one question that the patient may not understand.” For Explanation, it is told: “Try to mock as the a patient and select one sentence from the medical note that the patient may not understand.” The Assistant is instructed to answer or explain “based on the reference content and use concise language that people are easy to understand,” to avoid conflict with the note, and to “Mark answers you are not sure about.” These prompts encode three enduring NoteAid design commitments: grounding in the note, patient-friendly language, and explicit uncertainty marking (Zhang et al., 2023).

3. Data construction and quality of the EHR interaction dataset

The NoteAid EHR Interaction Dataset is built from 10,000 MIMIC-III discharge instructions and 876 MADE 1.0 notes. For both Explanation and Q&A, the pipeline is run with both Turbo NIP and GPT-4 NIP. Each task therefore contains 21,752 instances, and the entire dataset contains 43,504 instances, with each instance containing three rounds of dialogue (Zhang et al., 2023).

This dataset is unusual in that it is conditioned on real EHR notes while remaining explicitly patient-centered. In a typical Q&A instance, the Mock Patient asks a realistic follow-up such as why an allergic reaction occurred even though the medication had been taken before, and the Assistant explains sensitization in lay language. In a typical Explanation instance, the Mock Patient selects a sentence such as “Your INR was found to be high so your coumadin was stopped and you were transfused blood and clotting factors,” and the Assistant rewrites it as an explanation of what INR measures, why a high value matters, and what the treatment implies for clotting (Zhang et al., 2023).

Evaluation uses five dimensions scored from 0 to 5: Relevance, Factuality, Sufficiency, Concision, and Fluent. LLM evaluation covers 2,100 cases drawn from MADE and MIMIC, and human evaluation covers 84 instances. On LLM evaluation, GPT-4 eval GPT-4 NIP yields 92.48% level-5 quality for Q&A and 89% level-5 quality for Explanation. On human evaluation, GPT-4 NIP yields 95.96% level-5 quality for Q&A and 80.81% level-5 quality for Explanation (Zhang et al., 2023).

The reported failure cases are equally important. One explanation incorrectly states that steroid use occurred because hemoglobin and hematocrit fell, even though that causal relation is not supported by the note. One Q&A answer about ophthalmology follow-up is judged too generic and not sufficiently case-specific. These cases establish a recurring NoteAid theme: conversational fluency does not eliminate hallucination or genericity, and fidelity to note-level causality remains a hard constraint (Zhang et al., 2023).

4. “Learning as conversation” and PPO-based NoteAid-Chatbot

The later NoteAid-Chatbot reformulates patient education as a reinforcement learning problem in which the chatbot is rewarded not for stylistic similarity to reference answers but for the patient’s measured understanding after a conversation. The paper defines a Gold dataset,

$$\text{Comp}_G = \{ (\mathcal{N}_G^i, Q_G^i) \mid i \in [1, 100] \},$$

and a Silver dataset,

$$\text{Comp}_S = \{ (\mathcal{N}_S^i, \text{Conv}_S^i, Q_S^i) \mid i \in [1, 10000] \},$$

where $X$0 is a discharge note, $X$1 is a synthetic conversation, and $X$2 is the associated comprehension QA set (Jang et al., 6 Sep 2025).

The deployed educator is LLaMA 3.2-3B-Instruct, adapted first by LoRA supervised fine-tuning and then by PPO. Training uses a multi-agent setup with three roles: an Educator agent (the trainable NoteAid-Chatbot), a Patient agent implemented with GPT-4o-mini, and an Evaluator agent also implemented with GPT-4o-mini. In each episode, a discharge note is selected, the educator and patient converse for up to 20 turns, and the patient then takes a multiple-choice comprehension test about the note. The educator’s reward is the patient’s test accuracy:

$X$3

where $X$4 is the number of test questions and $X$5 if question $X$6 is answered correctly and $X$7 otherwise (Jang et al., 6 Sep 2025).

The synthetic pipeline preceding RL is itself extensive. GPT-4o-mini generates 10,000 synthetic discharge notes, associated multiple-choice comprehension questions, and full educator–patient conversation transcripts. The conversations are constrained to include six medical content categories—Return to ED/hospital indications, Medication information, Diagnosis, Post-discharge treatments, Tests/treatments during stay, and Follow-up instructions—and to instantiate six medical conversation strategies from King and Hoppe: Fostering relationship, Gathering information, Providing information, Decision making, Responding to emotions, and Enabling disease- and treatment-related behavior (Jang et al., 6 Sep 2025).

This architecture operationalizes the paper’s “learning as conversation” claim in a precise way: the dialogue is not merely a medium for answer delivery but the environment in which understanding is optimized. A plausible implication is that NoteAid-Chatbot occupies a middle position between note-grounded question answering and outcome-driven educational dialogue policy learning.

5. Performance, emergent behaviors, and human comparison

On held-out synthetic conversations, PPO materially improves the LLaMA 3.2-3B educator over both the untuned base model and the SFT-only version. The reported metrics are as follows (Jang et al., 6 Sep 2025):

Metric	LLaMA 3.2-3B baseline	LLaMA + LoRA	NoteAid-Chatbot
BLEU	0.024	0.033	0.153 ± 0.017
ROUGE-L	0.118	0.127	0.325 ± 0.021
BERTScore	0.851	—	0.893 ± 0.002
FKGL	~10.8	7.636	7.237 ± 0.134

The paper also reports that NoteAid-Chatbot achieves the highest content scores across the six discharge-information categories on both Gold and Silver datasets. On the Gold set, for example, Diagnosis reaches 0.287 ± 0.004 and Return to Hospital/ED reaches 0.301 ± 0.004. Strategy scores are similarly improved; on the Gold set, Fostering relationship reaches 0.059 ± 0.001, Providing information reaches 0.061 ± 0.001, and Responding to emotions reaches 0.046 ± 0.001. The paper explicitly interprets the learning curves as showing that, as RL progresses, comprehension reward rises while token length falls and Flesch-Kincaid Grade Level decreases, yielding shorter but more effective dialogue (Jang et al., 6 Sep 2025).

The human comparison is framed as a Turing-test-style experiment with three groups: non-expert human educator, NoteAid-Chatbot, and expert educator. Patients in the NoteAid-Chatbot group achieve a comprehension score of 0.719, compared with 0.650 for non-expert humans and 0.750 for expert educators. The chatbot therefore surpasses non-expert humans and approaches expert educator performance. On perceived humanness, however, the chatbot remains detectable: in the chatbot condition, participants report 1 “Yes”, 2 “No”, and 1 “Not sure” to whether the educator felt like a human (Jang et al., 6 Sep 2025).

The 2025 paper describes these outcomes as emergent behaviors rather than directly supervised style properties. Clarity, relevance, structured dialogue, lower reading level, and higher information density are attributed to optimization against patient understanding rather than explicit labels for those attributes. This distinguishes NoteAid-Chatbot from systems that supervise lay simplification directly.

6. Limitations, safety, and methodological context

The NoteAid systems are explicit about their limitations. In the 2023 pipeline, hallucinations remain possible, some answers are generic rather than case-specific, testing is confined to MIMIC discharge instructions and MADE 1.0, and patient comprehension is not guaranteed merely because experts judge an answer high-quality. Human evaluation also remains limited in scale (Zhang et al., 2023).

The 2025 NoteAid-Chatbot retains a different but related set of constraints. The authors explicitly state that no explicit hallucination detection or correction mechanism is implemented yet. They report both “Dangerous” hallucinations and “Benign” or “positive” hallucinations, and they restrict the intended use to low-risk discharge scenarios where the chatbot is strictly reading from a known discharge note. They further note that the patient agent is GPT-4o-mini roleplay rather than a real patient population, that conversation length is capped at 20 turns, and that only basic PPO is used, without broader exploration of RLHF, RLAIF, or richer reward modeling (Jang et al., 6 Sep 2025).

Within the broader research landscape, NoteAid-Chatbot sits near several adjacent methodological strands. NoteChat generates synthetic doctor–patient dialogues conditioned on clinical notes through a Planning module, a Roleplay module, and a Polish module, and its note-grounded synthetic dialogues improve both note-to-conversation and conversation-to-note modeling (Wang et al., 2023). A plausible methodological parallel is the framework in which a guiding chatbot is optimized through reinforcement learning to induce targeted properties in the interlocutor’s next response, such as sentence length, emotion, or topical words (Su et al., 2021). These neighboring lines of work do not define NoteAid-Chatbot, but they clarify its position: it belongs to a class of systems that treat dialogue not only as language generation, but as controlled interaction with measurable downstream effects.

Taken together, the NoteAid trajectory defines NoteAid-Chatbot as a specialized patient-education system grounded in EHR notes, shaped first by synthetic two-agent generation and then by outcome-based policy optimization. Its central claim is not that lay definitions are unnecessary, but that patient understanding is better modeled as an interactive, multi-turn, note-grounded educational process than as a static glossary lookup (Zhang et al., 2023, Jang et al., 6 Sep 2025).