Note-to-Dial: Synthesizing Dialogue from Notes

• Note-to-Dial is a framework that transforms structured, note-like inputs into realistic dialogues, exemplified in medical and multimodal applications.
• It employs intermediate representations such as SOAP notes and CUI checklists to enforce factual coverage, dialogue coherence, and turn-level control.
• Synthetic datasets like MedSynth, NoteChat, and DiagNote demonstrate multi-stage LLM pipelines using planning, roleplaying, and grounding for controlled generation.

Note-to-Dialogue (Note-2-Dial) denotes a class of generation problems in which a note-like representation conditions the production of a dialogue. In recent arXiv work, the term appears in two closely related but not identical senses. In medical documentation, it refers to generating realistic doctor–patient conversations from clinical notes, typically SOAP notes, to support data augmentation and documentation automation. In multimodal dialogue learning, it refers to models that write and consume intermediate “notes” during multi-turn reasoning, using those notes to drive grounded answers. The common core is the use of structured intermediate representations to enforce factual coverage, dialogue coherence, and turn-level control (Mianroodi et al., 2 Aug 2025, Wang et al., 2023, Liu et al., 10 Mar 2025).

1. Task definition and scope

In MedSynth, the Note-2-Dial task is explicitly framed as generating dialogues from medical notes. The dataset comprises “10 035 fully synthetic medical dialogue–SOAP note pairs,” spanning “2001 distinct ICD-10 codes, each represented by exactly five note–dialogue pairs,” with examples conditioned on the ICD-10 description and then expanded into realistic doctor–patient exchanges (Mianroodi et al., 2 Aug 2025). The target dialogue is not a compressed paraphrase of the note; prompts explicitly request “SOAP adherence, realistic patient speech registers (no jargon on the patient side), and comprehensive coverage of note facts.”

In NoteChat, the same general task is formulated as note-conditioned synthetic doctor–patient conversation generation. The system decomposes note-to-dialogue into “Planning, Roleplay and Polish,” and the Planning module extracts “a sequence of CUIs (Clinical Uniform Identifiers) from the note” and organizes them into an ordered checklist,

$K = \{c_1, c_2, \ldots, c_m\},$

which is updated after each turn to enforce progression through “symptoms → history → tests → assessment → plan” (Wang et al., 2023). This makes Note-2-Dial a controlled generation problem in which the note is both semantic source and discourse plan.

In DiagNote, Note-to-Dialogue is realized differently. The model operates over an image, a multi-turn dialogue context, and two internal buffers: accumulated Deliberate steps and accumulated Gaze outputs. At each turn, Deliberate produces a reasoning snippet and a region-query, Gaze grounds that query to a bounding box, and the loop continues until Deliberate emits “END,” after which the final answer is produced (Liu et al., 10 Mar 2025). Here, “notes” are explicit intermediate reasoning traces rather than clinical notes.

System	Note-2-Dial formulation	Core control mechanism
MedSynth	SOAP note $\rightarrow$ doctor–patient dialogue	ICD-10-conditioned multi-stage GPT-4o pipeline
NoteChat	Clinical note $\rightarrow$ synthetic conversation	CUI checklist with Planning, Roleplay, Polish
DiagNote	Intermediate notes $\rightarrow$ multi-turn answer dialogue	Deliberate × Gaze loop with region queries

A recurrent source of confusion is that these formulations share a label but not a single canonical input modality. The literature instead uses Note-2-Dial for a broader family of note-mediated dialogue generation problems.

2. Data construction and benchmark design

MedSynth is built around breadth and controlled coverage. It “uniformly sampl[es] the top 2000 most frequent ICD-10 codes (drawn from an 800 million-record subset of IQVIA PharMetrics claims),” thereby avoiding skew toward common conditions while retaining real-world prevalence cues (Mianroodi et al., 2 Aug 2025). Each code is represented by exactly five note–dialogue pairs. The resulting corpus is intended as “a valuable resource in a field where open-access, privacy-compliant, and diverse training data are scarce.”

The note-first generation procedure is central to that design. Notes are generated before dialogues, which allows later dialogue synthesis to be constrained by a complete SOAP note. This suggests a deliberate reversal of the usual clinical workflow for synthetic-data purposes: the note becomes the canonical information store, and the dialogue is generated as a faithful conversational realization of that store.

NoteChat constructs two synthetic collections. “Ours-PMC” is sourced from “167 K human-curated case reports from PubMed Central,” filtered to exclude “ultra-rare or animal cases,” and yields “synthetic, multi-round dialogues (≈30 turns each on average).” “Ours-MTS” is built from “1.7 K short SOAP-formatted note-dialogue pairs + 87 long SOAP notes,” where each note is split by SOAP section, sub-dialogues are generated, and the sections are recombined through a “Combine Prompt” (Wang et al., 2023). The dataset design therefore combines large-scale case-report synthesis with note-structure-aware recomposition.

MMDiag, the benchmark for DiagNote, is not medical and not note-conditioned in the clinical sense, but it is directly relevant to note-mediated dialogue learning. It contains “three complementary scenarios”: “Everyday scenes,” “Tabular charts,” and “Minigrid grid worlds.” Dialogues have “on average 2.19 turns/dialogue (up to 4),” and every turn includes a “key region query” with a “ground-truth bounding box or OCR token locations” (Liu et al., 10 Mar 2025). Independent QA pairs are filtered out so that later questions depend on prior turns through subgraph overlaps, pronoun references, or sequenced subgoals. That benchmark design makes note-taking operational rather than metaphorical: the model must accumulate and reuse turn-level notes.

3. Generation pipelines and architectures

MedSynth uses a multi-stage, LLM-driven pipeline. Notes are generated first through a “four-agent GPT-4o system.” The “Scenario Provider” selects a clinical role such as “Family Medicine Physician” or “Orthopedic Specialist” and instantiates variables including “symptoms, history, demographics, planned investigations, medication details, etc.” The “Scenario Judge” checks diversity, requiring “≥4 variable changes from prior scenarios,” as well as “medical accuracy, and plausibility,” and returns either “Go” or structured feedback. The “Note Writer” uses “one-shot in-context examples from Aci-Bench to draft a full SOAP note,” and the “Note Polisher” relocates facts to the correct SOAP sections and standardizes terminology (Mianroodi et al., 2 Aug 2025).

Dialogue synthesis then occurs in two steps. The “Dialogue Generator” uses “few-shot in-context learning (n=3 exemplar Aci-Bench pairs) to map a SOAP note back to a realistic doctor–patient exchange.” The “Dialogue Polisher” adds “social chit-chat,” enforces “non-linear turn-taking,” and verifies inclusion of every note detail. This staged design separates content realization from conversational naturalization.

NoteChat also uses a multi-agent design, but with a different decomposition. The Planning module acts as master controller; two ChatGPT instances role-play physician and patient; and a final Polish pass edits the full draft dialogue (Wang et al., 2023). The physician must “only introduce professional terms and numerical information,” must “never synonyms of checklist keywords,” and must “never ask questions whose answers do not appear in the note or checklist.” The patient must respond “colloquially in lay language” and is forbidden to introduce “test results, medical jargon or dosages on its own.” The checklist update rule can be written as

$K_{t+1} = K_t \setminus \mathrm{CUIs}(H_{t+1}),$

where covered concepts are removed turn by turn.

DiagNote is organized as two interacting modules, “Deliberate” and “Gaze.” At dialogue turn $t$, the model consumes the image $I_v$, the context

$C^t = \{Q^1, A^1, \ldots, Q^{t-1}, A^{t-1}, Q^t\},$

and two buffers: $B_d^t$ for Deliberate steps and $B_g^t$ for Gaze outputs (Liu et al., 10 Mar 2025). Deliberate, “an LLaVA-based LLM + ViT encoder,” emits a reasoning snippet $\rightarrow$0 and a region-query $\rightarrow$1; Gaze, with “same backbone, separate weights,” grounds $\rightarrow$2 to a bounding box $\rightarrow$3. The process repeats until Deliberate emits $\rightarrow$4. In the Minigrid example, the model first queries “purple key,” then “blue door,” and only then answers: “It’s to the left of the blue door on the bottom row.”

Across these systems, Note-2-Dial is not implemented as a single-pass decoder. It is mediated by planning structures, intermediate notes, or explicit grounding loops.

4. Prompting, conditioning, and optimization

MedSynth uses ICD-10 conditioning at the start of every example: “Each example begins with the ICD-10 description, guiding the Scenario Provider’s choice of physician role and scenario variables” (Mianroodi et al., 2 Aug 2025). Prompting combines “Chain-of-Thought and role-play triggers in every GPT-4o prompt,” with “zero-shot to define tasks, one-shot for note drafting, and few-shot for dialogue generation (n=3 examples drawn from Aci-Bench).” The downstream Note-2-Dial models are “Llama-3-8B-Instruct (and, in additional experiments, Mistral-7B-Instruct),” fine-tuned “using Low-Rank Adapters (LoRA) in 4-bit quantization.” The training objective is standard maximum-likelihood cross-entropy,

$\rightarrow$5

where $\rightarrow$6 is the SOAP note and $\rightarrow$7 the target dialogue tokens. Training is performed “for four epochs (one epoch in lower-compute benchmarks) with gradient accumulation, AdamW-8bit optimizer, linear learning-rate warmups, and mixed-precision BF16/FP16.”

NoteChat relies heavily on prompt specialization. Its Planning prompt explicitly instructs the model to “Follow the logical sequence of a physician’s inquiry: symptoms → history → testing → conclusion.” The Physician prompt requires order consistency with the provided keywords, and the Patient prompt forbids test results and specialized terms (Wang et al., 2023). The Polish prompt applies “10 human-engineered rules,” including “make colloquial,” “increase rounds,” “preserve professional vocabulary in physician turns only,” and “do not generate duplicate information.” For model training, NoteChat fine-tunes “LLaMA2-7B” with “LR = 2e-5, batch = 16, max tokens = 4048, epochs = 1,” using “A100 GPUs, DeepSpeed ZeRO-2.” The objective remains standard next-token cross-entropy.

DiagNote also uses supervised autoregressive cross-entropy, but factorized by module. For Deliberate,

$\rightarrow$8

and for Gaze,

$\rightarrow$9

with an additional $\rightarrow$0 at the final round if defined (Liu et al., 10 Mar 2025). These terms are unified as

$\rightarrow$1

The notable design decision is that grounding targets are themselves tokenized outputs encoding bounding-box coordinates, which keeps both modules within an autoregressive training regime.

5. Evaluation methodology and reported performance

MedSynth evaluates Note-2-Dial with both traditional overlap metrics and an LLM-based jury. The jury uses “three judge models (Prometheus, GPT-4o, Qwen2.5-32B-Instruct)” that vote on pairwise outputs using a rubric covering “Completeness, Accuracy, Naturalness & Flow, Use of Medical Terminology, and Evidence-Based Support” (Mianroodi et al., 2 Aug 2025). On the Aci-Bench test set, Note-2-Dial preference is reported as “MS + AT vs NC + AT: 55.0%” and “MS Only vs NC Only: 87.5%.” MedSynth also reports that the “Scenario Judge Agent alone contributes a ~55% jury preference uplift in Note-2-Dial.” These results are presented alongside traditional “BLEU, ROUGE-1/2/L/Summ, and METEOR,” though the paper notes that such metrics “correlate less well with clinical quality.”

NoteChat evaluates note-to-dialogue intrinsically, extrinsically, and with human raters. Automatic metrics include “ROUGE-N, ROUGE-L_sum,” factuality via “Concept-P/R/F1,” extractiveness, and “Self-BLEU.” For concept overlap, with $\rightarrow$2 and $\rightarrow$3 denoting CUI sets,

$\rightarrow$4

On MTS-Dialog zero-shot intrinsic evaluation, NoteChat reports “ROUGE-1: ChatGPT 48.56 / GPT-4 53.29 / NoteChat 56.48,” “ROUGE-Lsum: 46.36 / 50.81 / 53.41,” and “Self-BLEU↓: 0.017 / 0.019 / 0.014” (Wang et al., 2023). On extrinsic tasks, “Note2Conversation: adding NoteChat to MTS raises ROUGE-1 from 31.09→42.54 (+37%),” and “Conversation2Note: ROUGE-1 from 38.35→43.84 (+14%).” Human ranking reports “Expert MRR (higher = better): NoteChat ≈ 0.78, GPT-4 ≈ 0.66, ChatGPT ≈ 0.42,” and the abstract states that NoteChat “substantially surpasses state-of-the-art models like ChatGPT and GPT-4 up to 22.78% by domain experts.”

DiagNote evaluates both grounding and answer quality. Grounding uses mean Intersection-over-Union on “MMDiag-GND subsets (everyday, tabular, Minigrid), plus MSCOCO and RefCOCO.” “DiagNote (MMDiag + COCO fine-tuning) achieves ≈0.65 avg IoU, outperforming LLaVA (~0.23) and VisualCoT (~0.21)” (Liu et al., 10 Mar 2025). On multi-turn reasoning and answer accuracy, scored on a “0–10 scale via Gemini-1.5-Pro,” DiagNote reports “Everyday avg (reasoning / answer): 6.35 / 5.97 vs VisualCoT 4.15 / 4.90,” “Tabular: 3.95 / 4.30 vs 1.23 / 1.95,” and “Minigrid: 5.75 / 4.93 vs 1.09 / 2.50.” Ablations show that “Gaze annotations significantly improve later reasoning steps, especially in longer dialogues (turn 3 & 4).”

Taken together, these evaluations show a methodological divide. Medical Note-2-Dial work emphasizes pairwise preference, ROUGE-style overlap, CUI factuality, and human expert ranking, whereas multimodal note-mediated dialogue learning emphasizes grounding IoU and multi-turn reasoning scores.

6. Qualitative behavior, limitations, and future directions

MedSynth reports several qualitative properties of successful outputs. Dialogues include “realistic turn-taking, social fillers (‘hmm,’ ‘okay’), patient lay descriptions vs. doctor technical explanations,” and they “weav[e] all note facts across 80–150 utterances” (Mianroodi et al., 2 Aug 2025). In Dial-2-Note, MedSynth-trained models “reliably generate four distinct SOAP sections with accurate medication, test, and follow-up entries,” whereas models fine-tuned on NoteChat alone “often collapse SOAP structure or omit Plan details.” For Note-2-Dial, the corresponding claim is that the synthetic dialogues achieve broad factual coverage while remaining conversational.

The limitations are also explicit. MedSynth notes “occasional hallucinations in rare procedural details or misplacement of facts,” although these are “largely mitigated by the Polisher agents.” It further states that “synthetic dialogues may still lack the full unpredictability of human speech disfluencies or unplanned interruptions,” that “medical accuracy is not formally validated by domain experts,” and that “real-world noise (e.g., ASR errors) and multi-speaker interactions beyond doctor/patient are not fully captured.” The paper is explicit that MedSynth “is expressly not clinical advice.”

NoteChat’s qualitative analysis identifies different but related failure modes in baselines: they “dump all note information in a single turn (over-extractive), let patients speak too ‘professionally,’ or break conversational logic (doctor reveals history vs. eliciting it)” (Wang et al., 2023). NoteChat attributes improvements to “Checklist grounding with CUIs,” “Role-adaptive prompts,” and the split between “high-level planning + low-level roleplay.” Its limitations include English-only evaluation, reliance on “LLM API costs and human-engineered prompts,” and a tendency for synthetic data to “over-cover some note details (e.g. exact dosages) that real conversations may omit for brevity.”

DiagNote’s qualitative gains are framed around error reduction in grounded reasoning. In Minigrid, “GPT-4o hallucinated the key’s position; DiagNote’s intermediate notes and bounding-box queries kept it focused.” In tabular settings with multiple similar bars or wedges, explicit queries such as “Cyprus slice 2019” prevented guessing from context, and in everyday scenes, note-like reasoning steps such as “count red plates, then count bowls” reduced miscounts (Liu et al., 10 Mar 2025). A plausible implication is that explicit note-taking is most useful when ambiguity accumulates across turns and cannot be resolved by a single forward pass.

The most concrete future directions are given by MedSynth: “Expanding beyond the top 2000 codes to cover rarer conditions and specialties,” “incorporating alternate note structures (e.g., POMR, EPIC-style formats) and real ASR transcripts for robustness,” “conducting formal clinician assessments of medical correctness and readability in true clinical workflows,” and “exploring simulation of multi-party encounters (nurse handoffs, family members) and acute/emergent scenarios” (Mianroodi et al., 2 Aug 2025). These directions indicate that Note-2-Dial research is moving from synthetic pair generation toward workflow realism, note-format diversity, and richer interaction structures.