Dialogue-to-Note: Structured Dialogue Summaries

• Dialogue-to-Note is the transformation of multi-turn dialogue into structured, concise notes with strict constraints on speaker attribution and domain-specific formats.
• It integrates multimodal elements and third-person narrative approaches, adapting to various use cases like clinical documentation and general dialogue summarization.
• Key challenges include preserving dialogue structure, managing turn-level dependencies, and mitigating errors such as omissions and hallucinations.

Searching arXiv for recent Dial-2-Note related papers to ground the article. Dialogue-to-Note (Dial-2-Note) denotes the transformation of multi-turn dialogue into concise notes or summaries, typically with stronger structural constraints than generic summarization. In the cited literature, the task appears in several technically distinct forms: multimodal dialogue converted into persistent textual and visual notes for grounded reasoning (Liu et al., 10 Mar 2025), general dialogue converted into third-person summaries or notes whose fidelity must be evaluated at speaker and turn granularity (Ramnath et al., 8 Feb 2026), and clinical encounters converted into structured visit documentation such as SOAP notes (Mianroodi et al., 2 Aug 2025, Yim et al., 2023). Across these settings, the central difficulty is not merely compression. It is the preservation of dialogue structure, referential continuity, speaker attribution, and domain-specific organization while avoiding unsupported content, viewpoint distortion, and omissions.

1. Conceptual scope and task definition

Dial-2-Note is framed differently across subfields, but the shared formal problem is to map a dialogue history into a note representation that is more structured, more compact, and often written from a different discourse perspective than the source dialogue. In multimodal dialogue, the output may include not only textual notes but also visual annotations associated with image regions; in this formulation, notes are explicit intermediate artifacts used during reasoning rather than only final outputs (Liu et al., 10 Mar 2025). In dialogue summarization, the output is a concise third-person summary whose correctness depends on turn ordering, speaker fidelity, and viewpoint handling (Ramnath et al., 8 Feb 2026). In clinical NLP, the output is a structured medical note, often in SOAP format, generated from a multi-turn physician–patient interaction (Mianroodi et al., 2 Aug 2025, Yim et al., 2023).

The multimodal formulation in "Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning" defines notes as two coupled forms: textual Chain-of-Thought traces and visual annotations of salient image regions (Liu et al., 10 Mar 2025). Specifically, textual notes are stepwise CoT traces $S_i^t$ at turn $t$ and interaction round $i$, while visual notes are region annotations $o_i^t$ tied to natural language queries $Q_i^t$. These are stored in buffers $B_d^t$ and $B_g^t$ and fed back into later rounds, making note-taking part of the inference loop rather than only a final summarization step (Liu et al., 10 Mar 2025).

In clinical settings, Dial-2-Note is defined as generating a complete visit note from a doctor–patient conversation. ACI-BENCH partitions the note into four contiguous divisions inspired by SOAP: Subjective, Objective_exam, Objective_results, and Assessment_and_plan (Yim et al., 2023). MedSynth uses standard SOAP format with Subjective, Objective, Assessment, and Plan, and states that the Subjective section may be further broken down into Chief Complaint, History of Present Illness, and Review of Systems (Mianroodi et al., 2 Aug 2025). These formulations make explicit that note generation is often a structured generation task with section-level content constraints, not a free-form abstractive summary.

A notable generalization appears in DIAL-SUMMER, which treats dialogue-to-note conversion as a transformation involving both structural shift and viewpoint shift (Ramnath et al., 8 Feb 2026). Structural shift refers to the fact that information is dispersed across alternating turns and must be compressed into sentences or bullets. Viewpoint shift refers to the change from first- or second-person dialogue to standardized third-person narration. This framework makes clear that Dial-2-Note is not reducible to standard summarization metrics or taxonomies.

2. Structural and semantic challenges

The literature identifies several recurring failure modes intrinsic to dialogue-to-note conversion. In multimodal multi-turn settings, real-world conversations require sustained attention across turns, especially saliency tracking and saliency recall (Liu et al., 10 Mar 2025). Saliency tracking means identifying and following multiple relevant regions, while saliency recall means retaining focus on the same regions in later turns. The hypothesis underlying DiagNote is that explicit intermediate notes guide attention and stabilize reasoning across turns, analogous to highlighting and jotting notes (Liu et al., 10 Mar 2025).

In text-only dialogue summarization, DIAL-SUMMER formalizes hierarchical errors at two levels (Ramnath et al., 8 Feb 2026). Dialogue-level errors include Wrong Turn Sequence, Missed Turn, Speaker Misattribution, Speaker Identity Bias, and Viewpoint Distortion. Within-turn-level errors include Wrong Linking, Changed Meaning, Extrinsic Conversation, Extrinsic Context, and Missed Conversation. The framework emphasizes that speaker attribution and turn-sequence integrity are central for notes, and that conventional metrics such as ROUGE and BERTScore do not explicitly evaluate speaker identities, turn boundaries, or narration viewpoint integrity (Ramnath et al., 8 Feb 2026).

The clinical literature exposes additional task-specific constraints. ACI-BENCH notes that clinical notes contain domain-specific language and content not necessarily verbatim from the dialogue, including templated phrases and references to prior results (Yim et al., 2023). It also reports that unsupported EHR injections were systematically identified and removed, indicating that factual traceability to the dialogue is a core benchmark design principle (Yim et al., 2023). MedSynth similarly enforces section placement and factual consistency through a Note Polisher agent: orders belong in Plan, completed tests in Objective, prescriptions in Plan, and referral details must be filled with reason, specialty, and physician name (Mianroodi et al., 2 Aug 2025).

Several findings suggest that omissions are at least as consequential as hallucinations. DIAL-SUMMER reports that Missed Conversation occurs in 73.4% of summaries and Missed Turn in 29.7%, while at least one hallucination of any type appears in 70.3% of summaries (Ramnath et al., 8 Feb 2026). It further reports that turns occurring in the middle of the dialogue are the most frequently missed in the summary, and that extrinsic hallucinations largely occur at the end of the summary (Ramnath et al., 8 Feb 2026). A plausible implication is that dialogue-to-note systems need explicit coverage control in addition to hallucination mitigation.

3. Note representations and modeling paradigms

A major distinction in the literature is between note generation as a final output task and note-taking as an explicit computational mechanism. DiagNote exemplifies the latter. It comprises two MLLMs without parameter sharing, both based on LLaVA-1.5-7B with CLIP ViT-L/14 at 336 resolution, a Vicuna-like LLM, and an MLP projector denoted mlp2x_gelu (Liu et al., 10 Mar 2025). The Deliberate module performs reasoning, CoT generation, and query generation; the Gaze module performs grounding by mapping natural-language queries to bounding boxes (Liu et al., 10 Mar 2025).

The per-turn process is iterative. Given image $I_v$ and dialogue context $C^t=(q^1,a^1,\dots,q^{t-1},a^{t-1},q^t)$, Deliberate generates a note $S_i^t$ and query $t$0, Gaze returns a box $t$1, and the buffers are updated until Deliberate outputs END (Liu et al., 10 Mar 2025). The finalization step generates the answer $t$2 and an optional final salient region. Textual notes are plain-text CoT steps, while visual notes are bounding boxes $t$3 serialized as tokens; associated queries are stored for traceability (Liu et al., 10 Mar 2025). The buffers thereby function as a persistent notebook at turn level.

Clinical systems more commonly adopt structured output schemas than explicit intermediate notebooks. ACI-BENCH advocates division-based generation, in which each note division is modeled separately and concatenated, with missing sections flagged as EMPTY (Yim et al., 2023). This is partly motivated by long-sequence degradation and by the observation that later divisions such as Objective_results and Assessment_and_plan are particularly challenging for full-note models (Yim et al., 2023). MedSynth, although focused on synthetic data generation, likewise centers the task around a stable output schema: SOAP notes with possible CC/HPI/ROS decomposition within Subjective (Mianroodi et al., 2 Aug 2025).

DIAL-SUMMER contributes a different kind of note representation: an evaluative decomposition rather than a generative architecture (Ramnath et al., 8 Feb 2026). For practical adaptation to note-style outputs, it proposes organizing notes by participants, timeline or turn coverage, content bullets, and action items or decisions. It also describes a suggested evaluation pipeline based on turn alignment, grounding checks, viewpoint normalization, speaker attribution validation, turn coverage audit, and end-of-notes extrinsic scan (Ramnath et al., 8 Feb 2026). These are presented as adaptations rather than the paper’s core experimental method, but they define a useful schema for structured Dial-2-Note auditing.

4. Datasets and benchmarks

The recent literature spans multimodal, general-dialogue, and clinical datasets, each emphasizing different dimensions of Dial-2-Note.

Resource	Domain	Scale or size
MMDiag	Multi-turn multimodal dialogue	639k multimodal QA items; 1,139k grounding annotations
DIAL-SUMMER dataset	Dialogue summary evaluation	192 dialogue–summary pairs
MedSynth	Synthetic medical dialogue-note pairs	10,035 dialogue–note pairs; 2001 unique ICD-10 codes
ACI-BENCH	Ambient clinical intelligence	207 encounters

MMDiag was introduced as a multi-turn multimodal dialogue dataset built from everyday scenes, tabular scenes, Minigrid scenes, and a common visual-text subset (Liu et al., 10 Mar 2025). It contains 639k multimodal QA items, 1,139k grounding annotations, and an average of 2.19 turns per dialogue (Liu et al., 10 Mar 2025). Its construction enforces multi-turn dependencies. For everyday scenes, each image yields a graph $t$4, QA pairs are subgraphs $t$5, and dependency is guaranteed by overlap conditions such as $t$6 such that $t$7 (Liu et al., 10 Mar 2025). This creates correlated, interdependent turns rather than isolated VQA instances.

DIAL-SUMMER’s dataset is an inference and evaluation set rather than a training corpus (Ramnath et al., 8 Feb 2026). It consists of 192 dialogue–summary pairs derived from Anthropic-Test helpful-base conversations with at least four turns, and all summaries are LLM-generated by Llama-3.3-70B-Instruct with greedy decoding (Ramnath et al., 8 Feb 2026). The resource includes 1,920 summary-level error labels across 10 error categories and sentence-level labels over 835 summary sentences (Ramnath et al., 8 Feb 2026). No explicit train/dev/test split is provided.

MedSynth is positioned as a large open synthetic dataset for both Dial-2-Note and Note-2-Dial (Mianroodi et al., 2 Aug 2025). It contains 10,035 dialogue–note pairs covering 2001 unique ICD-10 codes, with average dialogue length 932 tokens and 55 sentences, and average note length 621 tokens and 23 sentences (Mianroodi et al., 2 Aug 2025). Its construction uses a multi-agent GPT-4o-based pipeline with Scenario Provider, Scenario Judge, Note Writer, and Note Polisher for notes, and Dialogue Generator plus Dialogue Polisher for dialogues (Mianroodi et al., 2 Aug 2025). Uniform sampling across the top 2000 ICD-10 codes with five pairs per code is used to counter heavy-tailed disease prevalence (Mianroodi et al., 2 Aug 2025).

ACI-BENCH is a smaller but highly structured benchmark for automatic visit note generation (Yim et al., 2023). It contains 207 encounters, with average transcript length 1302 tokens and 55 turns, and average note length 490 tokens and 49 sentences (Yim et al., 2023). The dataset is split into train 67, validation 20, and three test sets of 40 each (Yim et al., 2023). It models three interaction modes—virtassist, virtscribe, and aci—and includes ASR and ASR-corrected transcript variants for some subsets (Yim et al., 2023). The benchmark’s division-aware evaluation and unsupported-span annotation distinguish it from generic summarization corpora.

5. Training objectives, evaluation methods, and metrics

The modeling objectives differ across paradigms. DiagNote uses an autoregressive language-model objective for both Deliberate and Gaze (Liu et al., 10 Mar 2025). If $t$8 is the target output sequence of module $t$9 at round $i$0 with input $i$1, then

$i$2

The cross-entropy loss per round is

$i$3

with totals

$i$4

and combined objective

$i$5

Bounding boxes are generated as text tokens; no separate regression loss is added (Liu et al., 10 Mar 2025).

In clinical Dial-2-Note, ACI-BENCH primarily evaluates with ROUGE-1/2/L, BERTScore, BLEURT, and UMLS concept F1 (Yim et al., 2023). It gives explicit formulas for ROUGE-N recall and concept-set precision/recall/F1. Because BERT-based metrics typically have 512-subtoken limits, division-level evaluation is emphasized for long notes (Yim et al., 2023). The final combined score for division-level evaluation is the average of mean ROUGE, BERTScore, BLEURT, and UMLS concept F1 (Yim et al., 2023).

MedSynth uses a different evaluation design. Its primary metric is an LLM jury with majority voting across Prometheus, GPT-4o, and Qwen2.5-32B-Instruct (Mianroodi et al., 2 Aug 2025). Judges compare two candidate notes against a reference note using a rubric containing Hallucination, Critical Omissions, Professional Tone, Logical Structure, Adherence to SOAP format, and Section Relevance (Mianroodi et al., 2 Aug 2025). Secondary metrics include BLEU, ROUGE-1/2/L/Lsum, and METEOR (Mianroodi et al., 2 Aug 2025).

DIAL-SUMMER focuses on error detection rather than note generation metrics (Ramnath et al., 8 Feb 2026). For LLM-as-a-judge experiments, it uses summary-level balanced accuracy and sentence-level balanced accuracy. Sentence labels are aggregated to summary-level predictions by marking an error present if any sentence is labeled YES; for Missed Turn, the same logic operates over turns rather than sentences (Ramnath et al., 8 Feb 2026). The paper finds that prompting judges with the Dial-SummEr taxonomy improves coarse hallucination detection relative to Basic, TofuEval, and FineSurE prompts for GPT-5, Claude-Sonnet-4.5, and Qwen2.5-32B-Instruct (Ramnath et al., 8 Feb 2026).

These evaluation choices reflect different assumptions about what a note should preserve. MMDiag prioritizes grounding and stepwise reasoning; ACI-BENCH emphasizes section-aware overlap and concept fidelity; MedSynth uses comparative rubric-based preference judgments; DIAL-SUMMER formalizes turn- and speaker-centric error taxonomies. This suggests that a single universal metric for Dial-2-Note remains absent.

6. Empirical findings and error patterns

The empirical record across these papers indicates that explicit structure improves performance, but the relevant structure depends on domain.

In multimodal dialogue, DiagNote reports gains in both grounding and reasoning over baselines (Liu et al., 10 Mar 2025). For grounding, average IoU is 0.356 for Grounding DINO, 0.233 for LLaVA, and 0.213 for Visual CoT, while DiagNote trained on MMDiag + COCO reaches 0.648 average IoU (Liu et al., 10 Mar 2025). The paper also reports that training on MMDiag only yields 0.471 average IoU, reflecting the dataset’s focus on correlated, multi-region tasks (Liu et al., 10 Mar 2025). In reasoning and answer consistency evaluated by Gemini-1.5-Pro on a 0–10 scale, LLaVA averages 2.21, Visual CoT 2.81, DiagNote without Gaze 4.32, DiagNote with Gaze on MMDiag 4.92, and DiagNote with Gaze on MMDiag+COCO 5.18; an upper bound with ground-truth notes injected reaches 7.00 (Liu et al., 10 Mar 2025). Turn-level tabular results show that Gaze helps more on longer contexts, especially turns 3 and 4, which the paper interprets as improved persistent focus via annotations (Liu et al., 10 Mar 2025).

The same work also identifies a major limitation: tiny regions degrade grounding and can misguide reasoning (Liu et al., 10 Mar 2025). In tabular scenes, 87.17% of targets occupy at most 0.2% of image area and 99.80% at most 3%; in Minigrid, 66.61% are at most 1% and 96.99% at most 3% (Liu et al., 10 Mar 2025). The paper attributes limited or negative Gaze gains in such settings to low image resolution and tiny boxes and mentions abstention policies, higher-resolution vision towers, and multi-scale features as possible remedies (Liu et al., 10 Mar 2025).

In dialogue-summary evaluation, DIAL-SUMMER shows that fidelity problems are widespread and structured (Ramnath et al., 8 Feb 2026). Viewpoint Distortion occurs in 20.8% of summaries, Missed Turn in 29.7%, and at least one Dialogue-level error in 24% (Ramnath et al., 8 Feb 2026). Changed Meaning is identified as the hardest fine-grained error for LLM judges, with models performing around 48–59% balanced accuracy, whereas Missed Turn can be detected much more reliably by GPT-5 (Ramnath et al., 8 Feb 2026). The positional analyses are especially notable: viewpoint distortion tends to occur at the beginning, Wrong Turn Sequence and several semantic errors in the middle, and extrinsic conversation or context at the end (Ramnath et al., 8 Feb 2026).

Clinical benchmarks show that output structuring materially affects results. In ACI-BENCH test set 1, the best ROUGE results come from BART+FT_SAMSum with division-based generation, with ROUGE-1 53.46, ROUGE-2 25.08, ROUGE-L 48.62, and UMLS-F1 48.23 (Yim et al., 2023). GPT-4 attains ROUGE-1 51.76, ROUGE-2 22.58, ROUGE-L 45.97, and the highest UMLS-F1 at 57.78 (Yim et al., 2023). The paper reports that division-based generation consistently improves BART and LED performance over full-note generation and that later divisions are especially challenging because of sequence position, sparsity, and structural variability (Yim et al., 2023).

MedSynth reports gains from synthetic data both alone and when combined with Aci-Bench training data (Mianroodi et al., 2 Aug 2025). In LLM-jury evaluation for Dial-2-Note, MS+AT beats NC+AT 60.0% of the time, MS Only beats NC Only 95.0%, and MS+AT beats AT Only 52.5% (Mianroodi et al., 2 Aug 2025). The paper highlights that models trained exclusively on NoteChat fail to preserve SOAP structure, whereas MedSynth-trained models preserve SOAP (Mianroodi et al., 2 Aug 2025). Traditional metrics move in the same direction: for MS Only vs NC Only, BLEU is 0.1151 vs 0.0873, ROUGE-1 0.5170 vs 0.3791, ROUGE-2 0.2267 vs 0.1547, ROUGE-L 0.2975 vs 0.2207, ROUGE-LSum 0.4838 vs 0.3340, and METEOR 0.3812 vs 0.3351 (Mianroodi et al., 2 Aug 2025).

7. Applications, limitations, and future directions

Dial-2-Note now spans at least three application regimes: multimodal conversational agents that maintain persistent grounded notes (Liu et al., 10 Mar 2025), dialogue summarization systems whose outputs are evaluated for turn and speaker fidelity (Ramnath et al., 8 Feb 2026), and clinical documentation systems that convert encounters into structured visit notes (Mianroodi et al., 2 Aug 2025, Yim et al., 2023). Despite the domain differences, several common engineering themes recur.

First, explicit structure is repeatedly beneficial. DiagNote uses persistent buffers of textual and visual notes (Liu et al., 10 Mar 2025). ACI-BENCH shows advantages for division-based generation (Yim et al., 2023). MedSynth uses SOAP-constrained generation and a polishing stage to enforce section correctness (Mianroodi et al., 2 Aug 2025). DIAL-SUMMER suggests sentence- or turn-granular evaluation anchored to dialogue evidence (Ramnath et al., 8 Feb 2026). This suggests that dialogue should not be treated as an undifferentiated input stream when the target note has strong organizational constraints.

Second, note fidelity depends on more than semantic similarity. DIAL-SUMMER explicitly argues that conventional metrics miss speaker attribution, turn coverage, and viewpoint integrity (Ramnath et al., 8 Feb 2026). ACI-BENCH notes that uniform metrics may underweight clinically critical facts such as dosages, allergies, and laterality (Yim et al., 2023). MedSynth’s use of a rubric with hallucination, omissions, section relevance, and SOAP adherence reflects the same concern (Mianroodi et al., 2 Aug 2025). A plausible implication is that future Dial-2-Note evaluation will remain multi-dimensional and domain-specific.

Third, current systems remain vulnerable to cascading errors. In multimodal settings, grounding errors can misguide subsequent reasoning (Liu et al., 10 Mar 2025). In general dialogue summarization, viewpoint distortion and extrinsic context remain common (Ramnath et al., 8 Feb 2026). In clinical note generation, unsupported claims, section misplacement, and omission of clinically salient details remain central risks (Mianroodi et al., 2 Aug 2025, Yim et al., 2023). All four works therefore converge on the need for stronger grounding, better memory or coverage mechanisms, and more discriminating evaluation.

The future directions named in the sources are correspondingly concrete. DiagNote points to explicit entity memory, spatial memory, cross-turn attention with retrieval, formal note schemas, multi-scale grounding, and evaluation protocols for note quality (Liu et al., 10 Mar 2025). DIAL-SUMMER calls for future work to enhance LLM judges’ performance and mentions larger, diverse domains such as meetings, customer support, and clinical notes, as well as multilingual evaluation and fine-tuned evaluators (Ramnath et al., 8 Feb 2026). MedSynth emphasizes mixed training on synthetic and real privacy-compliant data, domain adaptation, and clinician review before deployment (Mianroodi et al., 2 Aug 2025). ACI-BENCH highlights long-context modeling, structured EHR grounding, clinically weighted metrics, and human-in-the-loop safety workflows (Yim et al., 2023).

Taken together, these studies define Dial-2-Note as a family of structure-sensitive generation problems in which the note is not merely a shorter restatement of dialogue. It is a reformulation that must preserve evidence, discourse roles, temporal ordering, and domain conventions under compression and viewpoint change. The current research trajectory suggests that progress depends on making those constraints explicit in both model design and evaluation.