Synthesis and Evaluation of Long-term History-aware Medical Dialogue

Abstract: An effective healthcare agent must be able to recall and reason over a patient's longitudinal medical history. However, the absence of datasets with realistic long-term dialogue timelines limits systematic evaluation. Real clinical text is constrained by privacy and ethics, while existing benchmarks focus on isolated interactions, failing to capture cross-session reasoning. We introduce a framework for synthesizing high-quality, long-term medical dialogues with LLMs. Our approach entails a knowledge-guided decomposition into three stages: constructing synthetic patient profiles with diverse disease and complication trajectories, generating multi-turn dialogues per encounter, and integrating them into a coherent longitudinal history dataset, MediLongChat. We establish three benchmark tasks-In-dialogue Reasoning, Cross-dialogue Reasoning, and Synthesis Reasoning-to evaluate the memory capabilities of healthcare agents. To assess data quality, we introduce a multi-dimensional evaluation framework combining vector-based metrics with LLM-as-a-judge assessments. Specifically, we define automatic measures-Faithfulness, Coherence, and Diversity-together with two LLM-based evaluations: Correctness and Realism. Benchmark experiments show that even state-of-the-art LLMs struggle with MediLongChat. These findings highlight the benchmark's applicability and underscore the need for tailored methods to advance healthcare agents.

• The paper introduces a knowledge-guided, task-decomposed pipeline that generates 50,000-token longitudinal patient histories while improving faithfulness, coherence, and dialogue diversity.
• The paper finds that current LLMs perform poorly on cross-session reasoning, with best open-ended F1 scores of 33.49 for in-dialogue reasoning and 24.25 for cross-dialogue reasoning, despite 80–83.75% accuracy on multiple-choice synthesis tasks.
• The paper shows that knowledge guidance primarily improves faithfulness and realism, task decomposition strengthens long-range coherence, and diversity controls increase linguistic variety, while synthetic-data shift and judge dependence remain important limitations.

This paper introduces a framework for synthesizing long-term, history-aware medical dialogue datasets and a benchmark, MediLongChat, for evaluating the longitudinal memory and reasoning capabilities of healthcare agents (2605.19766). The work addresses a concrete gap: existing medical dialogue corpora are dominated by isolated, single-session consultations, while real longitudinal clinical text is difficult to obtain due to privacy constraints, imperfect de-identification, and restrictive data governance. The paper's response is a knowledge-guided synthesis pipeline, a five-dimensional data quality evaluation protocol, and three benchmark tasks that expose substantial weaknesses in current LLMs when reasoning over multi-session patient histories.

Motivation and problem formulation

The authors frame history-aware longitudinal clinical dialogue as a core requirement for safe healthcare agents: an agent that recalls a prior breast-cancer diagnosis and incomplete follow-up imaging will produce materially different diagnoses and safety actions for "headache + blurred vision" than a history-agnostic model defaulting to migraine. Existing resources fall short on two fronts. Public dialogue benchmarks such as PubMedQA and MedMCQA emphasize static knowledge rather than dynamic, context-dependent reasoning, and multi-session open-domain corpora lack clinical grounding. The paper identifies three bottlenecks for synthetic approaches: LLM hallucination and long-text inconsistency (compounded, they argue, by potential instability in Mixture-of-Experts architectures), finite context windows that make full-history generation expensive or infeasible, and the absence of standardized evaluation methods for synthetic data quality.

Synthesis pipeline

MediLongChat is produced through a three-stage pipeline grounded in task decomposition guided by structured knowledge.

Stage 1: knowledge-guided patient record generation. Each fictional patient receives a persona (demographics, lifestyle, family history), fused with human-reviewed disease-and-complication metadata that verifies evidence-based associations, clinically plausible temporal ordering, and realistic event spacing. The output is a chronologically ordered medical event timeline serving as the narrative backbone. These intermediate records are deliberately withheld from evaluated models.

Stage 2: task-decomposed dialogue generation. Rather than asking an LLM to emit a complete ~50K-token history in one pass, the pipeline extracts individual medical events and generates one encounter per event under context-isolated prompting: the prompt contains the patient persona plus only event-local facts (condition, time, interventions), preventing cross-visit information leakage. Each dialogue comprises roughly 50 exchanges (~3,000 tokens); a full history spans 15–20 dialogues (~50K tokens). Controlled variation is introduced via per-encounter physician personas, style directives, and elevated decoding temperatures to avoid stylistic monoculture. Encounters are then stitched in temporal order.

Stage 3: benchmark generation. Three tasks are derived: In-dialogue Reasoning (IDR), which extracts salient clinical facets (date, complaint, medications, treatment plan) from a single encounter; Cross-dialogue Reasoning (CDR), which links facts across two or more dialogues (temporal ordering, recurrence versus first onset, therapy changes), including adversarial formulations, and intentionally exceeds typical context windows; and Synthesis Reasoning (SR), a multiple-choice task requiring diagnosis of a secondary disease or complication by integrating the full history, with distractors selected by symptom similarity. The final corpus covers 80 patients with per-patient transcripts and task annotations.

Evaluation framework

Data quality is assessed along five dimensions. Three are computable with deterministic vector- or topic-based formulas: Faithfulness (average cosine similarity between source context and utterances via sentence embeddings), Coherence (a penalty on abrupt changes in adjacent-pair similarity), and Diversity (a combination of BERTopic cluster coverage and normalized Shannon entropy). Two require semantic and domain judgment and are scored via G-Eval-style LLM-as-a-judge with 5-point rubrics normalized to $[0,1]$: Correctness (clinical factuality independent of the source) and Realism (human-likeness, empathy, natural turn-taking). The paper is explicit that no reliable purely lexical proxy exists for Correctness or Realism. To mitigate judge-specific bias, the authors aggregate scores across four heterogeneous judges (Gemini 2.5, GPT-5 mini, Qwen3-235B, DeepSeek-R1), observing that individual judges diverge considerably—for example, Correctness ranges from 3.80 (GPT-5 mini) to 4.95 (Qwen3-235B) for their own data—while the ensemble yields a more stable assessment. LLM-judged trends are sanity-checked against human annotations reported for Conversation Chronicles.

Dataset quality results

MediLongChat is substantially larger than prior corpora: an average of 50,217.3 tokens and 960.9 turns per conversation, roughly 16× the tokens and 17× the turns of MSC, and 3.6× its sessions. Against LoCoMo, Conversation Chronicles, MSC, and NoteChat, the ensemble G-Eval scores show the synthetic data leading on Diversity (4.858 vs. a 3.14–3.69 range), Coherence (4.838 vs. 3.48–4.46), and especially Realism (4.505 vs. 1.98–2.39 for baselines). Notably, LoCoMo scores higher on Correctness (4.965 vs. 4.545); the authors attribute this to baselines containing more curated, single-turn factual statements, whereas long generated dialogues carry higher error risk. This concession is worth flagging: the correctness advantage of the proposed data is not established, only its competitiveness. Automatic metrics corroborate the picture—Stage-2 Coherence (0.925) exceeds LoCoMo and NoteChat, and Diversity (0.5447) is the highest among compared datasets—while Faithfulness, computed only for their own data (0.601 for Stage 2, 0.635 for Stage 1), declines modestly as expressive freedom increases.

Benchmark results and ablations

Benchmarking GPT-4o mini, DeepSeek-R1, Qwen3-235B, ERNIE-4.5-turbo, and GPT-4.1 mini without memory augmentation yields uniformly weak open-ended performance. On IDR, the best F1 is 33.49 (DeepSeek-R1); on CDR it drops to 24.25 (GPT-4o mini), with BLEU-1 at or below 2.46 across all models and tasks. SR accuracy is comparatively high (80.00–83.75), indicating that multiple-choice formatting partially compensates for long-context errors without resolving the underlying memory deficit. The central empirical claim is that current frontier LLMs can recover facts within a single encounter but degrade markedly when linking facts across sessions—precisely the capability longitudinal clinical safety depends on. These results are baselines rather than evidence that the tasks are saturated or unsolvable; the paper reports no memory-augmented or retrieval-based systems.

Ablations quantify each pipeline component. Removing knowledge guidance in Stage 1 drops Faithfulness from 0.6353 to 0.4415 and Realism from 0.720 to 0.540. Removing task decomposition in Stage 2 reduces Coherence (0.924 to 0.8689) and Diversity (0.5447 to 0.4590). Removing diversity controls leaves Coherence nearly intact (0.921) but cuts Diversity sharply (to 0.3134) and lowers Realism. The authors conclude that knowledge guidance primarily secures faithfulness and plausibility, task decomposition secures long-range coherence, and diversity controls affect linguistic variety and perceived naturalness with limited effect on factuality.

Limitations and open questions

The paper concedes three limitations directly. Synthetic data inevitably deviates from real clinical distributions, particularly for rare diseases, complex comorbidities, and behavioral health factors, so benchmark performance may not transfer to authentic clinical dialogue. The LLM-as-a-judge protocol's reliability depends on prompt design and judge robustness, and fine-grained agreement with clinical experts remains unverified beyond trend-level sanity checks. Finally, the corpus is text-only; imaging, laboratory curves, and structured EHR data are absent. An additional methodological caveat is that the quality comparison against baselines applies a relaxed medical-factuality rubric to non-clinical corpora, which is reasonable but means the Correctness comparison is not strictly like-for-like. The scale is also modest—80 patients—which raises the open question of whether the observed LLM failure patterns hold at larger corpus sizes and across disease spectrums. Whether retrieval-augmented generation or episodic memory mechanisms can close the IDR-to-CDR gap on this benchmark is left explicitly open.

Conclusion

This work contributes a knowledge-guided, task-decomposed synthesis framework for longitudinal medical dialogue, a reproducible five-dimensional quality evaluation protocol combining deterministic metrics with multi-judge LLM assessment, and a three-task benchmark demonstrating that state-of-the-art LLMs perform poorly on cross-session clinical reasoning in open-ended formats. The ablation results cleanly attribute faithfulness to knowledge guidance and coherence to task decomposition. The benchmark's principal value lies in providing a controlled testbed for memory-augmented healthcare agents, with its principal caveats being synthetic-data distribution shift, judge-dependence of quality scores, and text-only scope.