Progressive Dialogue Synthesis

• Progressive Dialogue Synthesis is a method that incrementally generates dialogue by decomposing complex tasks into manageable stages to enhance long-term coherence.
• It employs techniques such as semantic scaffolding, knowledge graph-guided expansion, and temporal synchronization to maintain contextual fidelity and reduce drift.
• This approach mitigates challenges like hallucination and weak domain precision by structuring dialogue generation into discrete, controlled phases.

Searching arXiv for recent and relevant papers on progressive dialogue synthesis and closely related dialogue/speech synthesis frameworks. Progressive dialogue synthesis can be understood as a family of methods in which dialogue, dialogue-conditioned speech, or dialogue-guided media are constructed incrementally rather than through a single undifferentiated generation pass. Recent work instantiates this idea in several technically distinct ways: semantic scaffolding for multi-turn augmentation, knowledge-graph-guided path expansion, encounter-level decomposition for longitudinal medical timelines, progressive expert pre-training for multimodal dialogue, synchronous chunk-level prediction for full-duplex speech interaction, and iterative clarification loops for dialogue-driven image refinement (Jiang et al., 2024, Liang et al., 3 Aug 2025, Hu et al., 19 May 2026, Li et al., 2023, Veluri et al., 2024, Wang et al., 21 Apr 2025). Across these settings, the shared objective is to improve contextual coherence, grounding, controllability, and long-range consistency.

1. Conceptual basis and defining properties

The central contrast in this literature is between one-shot or strictly cascade generation and staged or synchronized generation. In spoken dialogue, the conventional pipeline is “dialogue model → text response → TTS front-end → acoustic model → vocoder,” whereas unified alternatives seek to “think how to respond” and “think how to speak” at the same time (Zhou et al., 2023). In multimodal dialogue, monolithic end-to-end training is presented as brittle because multi-modal dialogue data is limited and the task family is heterogeneous, spanning intent prediction, retrieval, state tracking, and response generation (Li et al., 2023). In full-duplex spoken interaction, turn-based systems are treated as structurally inadequate because human conversation is not merely sequential but synchronous, with overlaps, backchannels, and latency-sensitive turn initiation (Veluri et al., 2024).

A plausible classification is that progression can occur at four loci. First, it can be structural, where intermediate representations or latent steps constrain later dialogue turns. Second, it can be curricular, where capabilities are acquired in stages and earlier competencies supervise later ones. Third, it can be temporal, where inference is aligned to real-world time rather than to end-of-turn boundaries. Fourth, it can be interactive, where the system and its interlocutor co-adapt over multiple rounds. Each of these forms appears explicitly in the recent literature, although not always under identical terminology (Jiang et al., 2024, Li et al., 2023, Veluri et al., 2024, Wang et al., 21 Apr 2025).

The underlying motivation is consistent across domains. Direct long-form generation is described as prone to hallucination, semantic drift, context-window failure, weak domain precision, or poor interactional realism. Progressive synthesis therefore serves as a control strategy: it decomposes a hard global objective into smaller units whose local coherence can be maintained more reliably (Liang et al., 3 Aug 2025, Hu et al., 19 May 2026, Xie et al., 9 Oct 2025).

2. Textual scaffolding, domain grounding, and longitudinal construction

In domain-sensitive text dialogue, progression is often introduced through explicit intermediate scaffolds. The KPT method for multi-turn psychological dialogue augmentation decomposes generation into a Progressive Thought Generator, a Psychological Knowledge Generator, and a Multi-turn Dialogue Generator (Jiang et al., 2024). Progressive thought is used as a semantic stabilizer: keywords are extracted with a kw-prompt, clustered by K-means, and filtered with BERT attention; matched dialogue-thought combinations are then retrieved from a progressive thought database and used to prompt new thought generation. The resulting thought sequence is intended to preserve semantics across turns and reduce drift. Psychological knowledge is retrieved from KW-KG implemented with OwnThink, converted into structured dialogue history, and ordered by BERT attention weights before final dialogue generation. A punishment evaluation framework regenerates samples if similarity to the source is above 90% or below 35%, enforcing both novelty and fidelity (Jiang et al., 2024).

Knowledge graphs provide a different form of progressive structure. ProKG-Dial constructs domain-specific multi-turn dialogue by first partitioning a domain KG into semantically cohesive communities, then expanding each dialogue incrementally along graph structure (Liang et al., 3 Aug 2025). The KG is embedded with GraphSAGE, communities are initialized at the node level, merged when $\text{Sim}(e_u, e_v) > \theta$, and refined with Louvain modularity optimization. Within each subgraph, the Adaptive Relationship-guided Graph Walk (ARGW) algorithm selects a target entity and iteratively expands the dialogue path: relations are collected, dynamically weighted, and followed to the next entity; questions and answers are then generated turn by turn, with redundant paths pruned and optional expansion to similar entities. Redundancy is controlled both semantically and structurally, including subgraph filtering by the Jaccard index

$$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$

On the Chinese medical KG CMeKG, the final dataset contains 7,200 dialogues, split 4,800 / 800 / 1,600, with 8.2 average speaker turns, 131.1 average tokens, 349,840 total key entities, and 48.6 average key entities per dialogue. The ablation study reports BLEU-1 / ROUGE-L of 0.305 / 0.298 for the baseline, 0.332 / 0.319 for partitioning only, 0.344 / 0.326 for ARGW only, and 0.391 / 0.384 for the full model (Liang et al., 3 Aug 2025).

Longitudinal medical dialogue introduces a further level of progression: the unit is not merely the turn, but the patient timeline. MediLongChat explicitly avoids generating a full lifetime of dialogue in one pass and instead decomposes synthesis into three stages: construction of synthetic patient profiles and medical records, multi-turn dialogue generation per encounter, and integration into a coherent longitudinal dataset (Hu et al., 19 May 2026). Stage 1 creates hidden patient records with personal information, lifestyle habits, past medical history, and additional information such as family history; Stage 2 extracts independent medical events and generates full clinician–patient encounters using context-isolated prompting; Stage 3 converts the result into benchmark tasks. The paper reports that a full patient history may total around 50K tokens, that each encounter-level dialogue is typically about 50 exchanges and around 3000 tokens, and that a patient history contains 15–20 dialogues. The resulting corpus covers 80 patients and averages about 960.9 turns and 50,217.3 tokens per conversation (Hu et al., 19 May 2026).

These systems share a concrete methodological claim: long-range coherence is improved when the generator is not asked to solve all aspects of the dialogue simultaneously. In KPT, semantic continuity is externalized as thought; in ProKG-Dial, topic progression is externalized as a KG walk; in MediLongChat, cross-session realism is externalized as a hidden event timeline (Jiang et al., 2024, Liang et al., 3 Aug 2025, Hu et al., 19 May 2026).

3. Joint dialogue–speech modeling and spoken dialogue realization

A major branch of progressive dialogue synthesis concerns spoken dialogue, where progression includes not only content selection but prosody, speaker identity, and turn timing. One line of work replaces the standard cascade spoken dialogue pipeline with a unified LLM that jointly models dialogue response and speech-related linguistic features (Zhou et al., 2023). The paper studies Prosodic Structure Prediction (PSP) as a Chinese TTS front-end task, with hierarchical boundaries PW = #1, PPH = #2, and IPH = #3. PSP is reformulated as sequence-to-sequence generation, with loss

$$\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p_{\theta}(y_i \mid x_i, c_i).$$

Using the DataBaker Chinese Standard Mandarin Speech Corpus with an 8:1:1 split, the best fine-tuned LLM achieves 82.38 average F-score, exceeding the BERT-based SpanPSP baseline at 79.80 average F-score. The same work then unifies dialogue response and linguistic features in a single autoregressive target sequence using JSON-style dictionaries that encode characters, duration, pinyin, prosodic hierarchy, and pitch extrema via the D-value

$\mathcal{D} = 5 \times \log_2(F/F_0).$

On training data, this joint generator reaches 95.90% parsable, 86.88% matched characters, 98.79% matched pinyin, and 97.75% matched prosody; on test data, the corresponding figures are 89.70%, 69.26%, 86.29%, and 77.70%, which the paper interprets as clear overfitting (Zhou et al., 2023).

Later speech-generation systems move from linguistic-feature planning to direct spoken dialogue synthesis. MOSS-TTSD targets long-form dialogue scripts with explicit speaker tags, using Qwen3-8B-base, MOSS-Audio-Tokenizer, a multi-head delay pattern inspired by MusicGen, and only the first 16 RVQ layers of the tokenizer, yielding about 2 kbps at 12.5 Hz (Zhang et al., 20 Mar 2026). Training uses contexts up to 65,536 tokens, clips up to 3600 s, and a three-stage curriculum from single-speaker and two-speaker data to real and synthetic 3–5 speaker data. At inference, the model supports up to 60 minutes of single-pass synthesis, up to 5 speakers, and zero-shot voice cloning through voice_clone_and_continuation. Its TTSD-eval framework uses MMS-FA forced alignment and wespeaker-SimAMResNet100 embeddings rather than diarization-based metrics. On TTSD-eval, MOSS-TTSD reports Chinese ACC 0.9587, SIM 0.7949, WER 4.85% and English ACC 0.9626, SIM 0.7326, WER 9.88% (Zhang et al., 20 Mar 2026).

DialoSpeech addresses dual-speaker conversational speech with a two-stage architecture: Text-to-token (T2T) via the LLM-based DiaLM, and Token-to-waveform (T2W) via Chunked Flow Matching (Xie et al., 9 Oct 2025). Dialogue text is BPE-tokenized, speech is discretized with S3Tokenizer, speaker prompts are extracted by ECAPA-TDNN, and a LLaMA-based Transformer predicts two semantic streams in parallel with causal cross-attention. Special tokens such as [spkchange] mark turn boundaries, and <SIL> allows the model to represent silence for the inactive speaker, so overlap becomes an emergent property of dual-track token generation. Acoustic reconstruction is performed with conditional flow matching and BigVGAN-v2. The training pipeline produces about 5,000 hours of natural conversational data at the processing stage and uses 10,000 hours in total, including 3,000 hours of Chinese dialogue from Biaobei, 5,000 hours of spontaneous Chinese podcasts, and 2,000 hours of English Fisher telephone dialogue. The model reports best Spontaneity on both English (3.71) and Chinese (3.96) benchmarks, best CER 2.27 in Chinese, and competitive coherence, intelligibility, SIM-O, and UTMOS (Xie et al., 9 Oct 2025).

A recurring misconception is that progressive spoken dialogue synthesis is simply long-form TTS. The cited work uses progression more specifically: simultaneous response and prosody planning, curriculum-based expansion to multi-party dialogue, or token-level coordination of parallel speaker streams (Zhou et al., 2023, Zhang et al., 20 Mar 2026, Xie et al., 9 Oct 2025).

4. Synchronous and full-duplex dialogue agents

Progression can also be temporal rather than merely structural. SyncLLM is designed for full-duplex spoken dialogue, where the model must generate while listening and remain aligned to a real-world clock rather than to end-of-turn boundaries (Veluri et al., 2024). The system is based on Llama3-8B and predicts interleaved chunks of discrete HuBERT units at 25 Hz, so one token corresponds to 40 ms. Dialogue is chunked at 160 ms, 200 ms, or 240 ms and augmented with synchronization tokens [S0] and [S1]. Deduplication removes repeated HuBERT units within a chunk to increase semantic density; the paper states that repeated units can cut semantic content per token by about 50% compared to deduplicated sequences.

The decoding formulation is explicitly synchronous. At time step $N$, the model predicts the next chunk as

$p(x_{N+1} \mid x_{\le N}, y_{\le N}),$

and, when the current user chunk is unavailable because of latency, it estimates

$\hat{y}_N \sim p(y_N \mid x_{\le N}, y_{<N}),$

then generates

$p(x_{N+1} \mid x_{\le N}, y_{<N}, \hat{y}_N).$

Once the real user chunk arrives, the estimate is discarded and replaced. This mechanism is the paper’s solution to internet-scale delays of 160 ms, 200 ms, and 240 ms (Veluri et al., 2024).

Training proceeds in three stages: 193k hours of text-derived synthetic speech in Stage 1, 20k hours of synthetic full-duplex dialogue in Stage 2, and about 2000 hours of real spoken dialogue from Fisher in Stage 3. Relative to dGSLM, SyncLLM reduces the perplexity gap from ground truth from about 70 to about 15. Human ratings report Meaningfulness 3.74 and Naturalness 3.90 for SyncLLM-F, compared with 1.55 and 3.95 for dGSLM, while ground truth is about 4.96 on both scales. Robustness is strongest at 160–200 ms latency and degrades at 240 ms (Veluri et al., 2024).

This full-duplex line broadens the notion of progressive dialogue synthesis. Progression need not mean only “more turns” or “staged prompting”; it can mean chunk-wise prediction under synchronized temporal constraints, with overlap and backchanneling treated as first-class modeling targets (Veluri et al., 2024).

5. Progressive pre-training and co-adaptive multimodal refinement

In multimodal dialogue, progression often appears as staged capability acquisition. PaCE is a unified multi-modal dialogue pre-training framework that replaces the standard Transformer FFN with multiple semantic experts and trains them progressively (Li et al., 2023). Each block preserves shared self-attention,

$$\boldsymbol{H}_l^{\prime}=\text{MSA}(\text{LN}(\boldsymbol{H}_{l-1}))+\boldsymbol{H}_{l-1},$$

while expert-specific FFNs produce

$$\boldsymbol{H}_l^{\text{expert}_k}=\text{FFN}^{\text{expert}_k}(\text{LN}(\boldsymbol{H}_l^{\prime}))+\boldsymbol{H}_l^{\prime}.$$

PaCE defines five experts—Caption, Image, Grounding, Context, and Generation—with the bottom $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$0 layers using modality experts and the top $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$1 layers using capability experts, where experiments set $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$2 and $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$3. Training unfolds in three stages: Image-Text Matching, Image-Context Matching, and Generation Modeling. Stage II introduces teacher-guided transfer with

$$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$4

so the older Caption expert guides the new Context expert. PaCE pre-trains on about 4 million multi-modal non-dialogue samples and about 1.4 million multi-modal dialogue samples, achieving reported state-of-the-art performance on eight multi-modal dialogue benchmarks, including PhotoChat F1 63.8, MMDialog F1 77.6, MMConv overall accuracy 39.2, and SIMMC2.0 BLEU 34.1 (Li et al., 2023).

Twin-Co shows that dialogue can itself be the progressive control mechanism for another generator, in this case text-to-image diffusion (Wang et al., 21 Apr 2025). Starting from an initial prompt $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$5, the system forms an updated prompt summary

$$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$6

where $$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$7 is implemented with GPT-4, and then generates

$$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$8

An implicit optimization pathway captions generated images with Qwen-VL, computes ambiguity via CLIP similarity,

$$\text{Sim}(G_1, G_2) = \frac{|E_{G_1} \cap E_{G_2}|}{|E_{G_1} \cup E_{G_2}|}.$$9

and triggers clarification if $$\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p_{\theta}(y_i \mid x_i, c_i).$$0. Training combines supervised fine-tuning on 2000 curated ImageReward pairs, 496 selected prompt-image pairs for dialogue construction, at least 4 dialogue rounds per pair, and over 2000 prompt variations. Reported results are T2I CLIPscore 0.338, T2I BLIPscore 0.336, I2I CLIPscore 0.812, I2I BLIPscore 0.833, and Human voting 33.6%. In a user study with 20 participants, 30 sessions each, and 600 dialogue traces, user-perceived intent capture peaks around the third round, and the most common satisfactory interaction length is 4 rounds (21.1%) (Wang et al., 21 Apr 2025).

These multimodal systems show that progressive dialogue synthesis is not restricted to producing dialogue transcripts. It can also denote progressive acquisition of dialogue capabilities or the use of dialogue to iteratively disambiguate another generative process (Li et al., 2023, Wang et al., 21 Apr 2025).

6. Evaluation regimes, limitations, and open problems

Evaluation in this area is notably heterogeneous because the object being synthesized varies: dialogue datasets, speech streams, long-term histories, or dialogue-guided media. MediLongChat proposes a five-dimensional quality framework—Faithfulness, Coherence, Diversity, Correctness, and Realism—with automatic metrics

$$\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p_{\theta}(y_i \mid x_i, c_i).$$1

$$\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p_{\theta}(y_i \mid x_i, c_i).$$2

and

$$\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p_{\theta}(y_i \mid x_i, c_i).$$3

Its benchmark tasks—In-dialogue Reasoning, Cross-dialogue Reasoning, and Synthesis Reasoning—use F1 and BLEU-1 for IDR/CDR and accuracy for SR, and the paper reports that even state-of-the-art LLMs struggle with the resulting long-term clinical reasoning tasks (Hu et al., 19 May 2026).

Speech evaluation has its own methodological disputes. MOSS-TTSD argues that metrics such as cpWER and cpSIM depend too heavily on diarization quality as speaker count rises, motivating TTSD-eval, which uses forced alignment and explicit speaker tags instead (Zhang et al., 20 Mar 2026). KPT, by contrast, emphasizes that automatic metrics do not fully capture quality in psychological dialogue and supplements BLEU, Distinct, and BERTScore with human ratings from 3 psychology professionals and 2 annotators familiar with psychological datasets, covering professionality, engagement, fluency, diversity, and correctness (Jiang et al., 2024). ProKG-Dial likewise combines automatic metrics such as BLEU-1 through BLEU-4 and ROUGE-L with expert ratings of Coherence, Diversity and Coverage, and Semantic Accuracy on a 1 to 5 scale (Liang et al., 3 Aug 2025).

Several limitations recur. Unified LLM-based dialogue–speech planning does not necessarily imply end-to-end audio generation: the joint model of (Zhou et al., 2023) explicitly does not include downstream acoustic models or vocoders and does not generate actual audio. Domain-specific synthetic dialogue depends strongly on the quality and completeness of external structure: ProKG-Dial acknowledges dependence on KG quality and the risk of over-filtering, while KPT is limited by incomplete knowledge-graph coverage and heuristic similarity thresholds (Liang et al., 3 Aug 2025, Jiang et al., 2024). Long-form speech synthesis remains constrained by speaker-count limits, annotation scarcity, or memory cost; MOSS-TTSD reports support up to 5 speakers, and DialoSpeech notes that scaling to minute-scale or longer dialogues is difficult because joint language-model and flow-matching training is memory intensive (Zhang et al., 20 Mar 2026, Xie et al., 9 Oct 2025). Full-duplex dialogue remains sensitive to latency and codec design; SyncLLM reports degradation at 240 ms and notes limitations from simple vocoding, lack of laughter modeling, and context-window limits (Veluri et al., 2024).

The literature therefore does not present a single settled paradigm. Instead, it presents a set of converging design principles: decompose the global problem, preserve local coherence, externalize missing structure when necessary, and evaluate with metrics aligned to the synthesized object. This suggests that “progressive dialogue synthesis” is best treated not as one architecture class but as a methodological orientation spanning staged generation, staged training, temporal synchronization, and co-adaptive refinement (Li et al., 2023, Hu et al., 19 May 2026).