Multi-Turn Text Generation

• Multi-Turn Text Generation is a regime where models generate responses conditioned on an interaction history, ensuring coherence and contextual relevance.
• It is applied to diverse tasks including dialogue-based prompt refinement, text-to-SQL, and code generation, utilizing history conditioning to resolve ambiguities and improve user control.
• Research in MTG explores methods from supervised learning with auxiliary tasks to RL-based approaches, highlighting trade-offs in efficiency, robustness, and iterative refinement.

Multi-Turn Text Generation (MTG) denotes a generation regime in which a model produces text over an alternating interaction history rather than from a single isolated prompt. In the cited literature, this regime includes open-domain response generation, dialogue-based prompt generation for text-to-image synthesis, code generation with execution feedback, agentic Text-to-SQL, and simulated multi-turn pseudo-caption generation for person retrieval. The unifying property is history-conditioned generation: if a conversation is written as $(u_1,r_1,\dots,u_T,r_T)$, then each response $r_t$ conditions on the preceding turns, while the “environment” may be a human, a database engine, a code judge, or another model (Graf et al., 17 Mar 2026, Abdulhai et al., 2023, Zhao et al., 2020, Liu et al., 2024, Chen et al., 3 Feb 2026, Hua et al., 25 Jan 2026, Xie et al., 16 Sep 2025).

1. Formalization and problem scope

A basic formal distinction between single-turn and multi-turn generation is given in TurnWise: single-turn generation maps one prompt $u_1$ to one response, $r_1=\mathrm{LM}(u_1)$, whereas multi-turn generation produces $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$ for each turn $t$. This formulation emphasizes that MTG is not merely longer prompting; it requires the model to integrate prior turns, preserve coherence, resolve coreference, and adapt to follow-up constraints or corrections (Graf et al., 17 Mar 2026).

Earlier dialogue work frames a canonical MTG instance as conditional next-response generation. Given a dataset $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$, where $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$ is a sequence of utterances and $R_i$ is the human reply, the task is to learn $P(R\mid \mathcal{U})$. The main difficulties identified there are hierarchical structure across words and utterances, long-range dependency with substantial redundancy, and an efficiency–expressivity trade-off in model design (Zhao et al., 2020).

A broader decision-theoretic view appears in RL-oriented work. LMRL-Gym models multi-turn text generation as a Markov decision process, or partially observable Markov decision process, in which the state $r_t$0 is the interaction history, the action $r_t$1 is the next token or utterance, transitions append agent and environment utterances to history, and rewards are task-specific. This formulation is instantiated across text games and interactive dialogue tasks, including Maze, Text-Nav, Wordle, Chess, Endgames, 20 Questions, Guess My City, and Car Dealer (Abdulhai et al., 2023).

Specialized MTG tasks refine this abstraction. In multi-turn code generation, the observation sequence includes a problem statement, a generated program, and judge feedback in the form of a failing public test or the message that all public tests passed. COBALT further treats this setting as a finite-horizon MDP with $r_t$2 and argues that it is “one-step recoverable,” so a suboptimal edit can be fixed in one future turn and stepwise optimization is approximately optimal under a small KL trust region (Chen et al., 3 Feb 2026). In SQL-Trail, the state is $r_t$3, comprising a natural-language question, serialized schema, and full action–observation history, while each action includes both a natural-language reasoning trace and an executable SQL snippet (Hua et al., 25 Jan 2026).

2. Interaction topologies and turn structure

Representative MTG systems differ primarily in what constitutes a turn and what feedback is returned after each turn.

Setting	Turn structure	Feedback and termination
Dialogue-based TIS prompting	Select one of 15 dimensions, ask a multiple-choice question, elicit a user preference, refine the prompt	Stop when relevant dimensions are covered or the user requests a summary (Liu et al., 2024)
Text-to-SQL agent	Emit > plus <sql>, execute SQL, append <observation>	Stop on <solution> or when the turn budget is exhausted (Hua et al., 25 Jan 2026)
>	Person-retrieval pseudo-captioning	Initial caption, multi-turn attribute Q&A, reconstruction
>	Multi-turn code generation	Submit code, receive one failing public test or “all passed”

DialPrompt makes the turn structure explicit. Each round $r_t$5 consists of dimension selection and question generation, user preference elicitation, and prompt refinement. With current partial prompt $r_t$6 and a set of 15 dimensions $r_t$7, the system selects a dimension $r_t$8, generates a question $r_t$9, receives a preference $u_1$0, and refines the prompt by $u_1$1. The 15 dimensions are grouped into Artistic Elements & Techniques, Creative Expression, Visual Impact, and Context & Quality, and the reported dataset average is 6.16 rounds per session with 6.99 dimensions covered (Liu et al., 2024).

SQL-Trail centers on an interleaved reason–execute–observe loop. At each turn the policy emits reasoning and a SQL query, the database executes the query, and the resulting dataframe preview or error message is appended as an observation. The framework uses an adaptive turn-budget allocation mechanism keyed to difficulty: simple questions allow $u_1$2, medium questions allow $u_1$3, and hard or extra questions may use a global cap such as 10 (Hua et al., 25 Jan 2026).

In person retrieval, MTG is used differently: not as interactive inference with a human, but as training-time pseudo-label generation. The process begins with a simple caption $u_1$4, proceeds through $u_1$5 rounds of question answering over a dynamic question pool, filters redundant or low-relevance answers, and then reconstructs a final enriched caption $u_1$6 by fusing the initial caption with retained answers. The reported setting uses $u_1$7 and Qwen2-VL-7B for QA and fusion, with GPT-4o for final paraphrasing (Xie et al., 16 Sep 2025).

These examples indicate that “turn” is task-dependent. In some systems it is a natural-language clarification exchange; in others it is a tool call, a code revision, or an attribute query. A plausible implication is that MTG is best understood as an interaction pattern defined by iterative state updates rather than by dialogue alone.

3. Architectures and optimization objectives

The architectural spectrum in MTG ranges from deliberately shallow sequence models to full RL fine-tuning over interactive trajectories. In open-domain response generation, a notable design choice is to keep the generator simple and shift context modeling into auxiliary objectives. The 2020 model uses a single Transformer layer for encoding and decoding and augments maximum-likelihood training with four auxiliary tasks: word order recovery, utterance order recovery, masked word recovery, and masked utterance recovery. Its joint objective is

$u_1$8

with task-specific attention masks and a read-process-write component for utterance order recovery (Zhao et al., 2020).

DialPrompt uses a much more standard instruction-tuned LLM backbone. The base model is LLaMA3-8B-Instruct, and no architectural layers are added beyond standard prefix-injection of dialogue history. Training samples are alternating user and assistant messages; the user turns are masked, and the model is trained to predict assistant turns with a cross-entropy loss averaged across dialogue rounds. Training uses AdamW with learning rate $u_1$9, batch size 16, 10 epochs, and 8 A100 GPUs. Because training is performed in full-sequence mode, gradients flow across turn boundaries, so the model jointly learns to ask and to refine (Liu et al., 2024).

RL-based MTG introduces reward shaping and online interaction. SQL-Trail adapts Grouped Relative Policy Optimization (GRPO) to a multi-turn Text-to-SQL agent. Its composite reward panel has six terms: final execution reward, turn-count reward, bi-gram overlap reward, schema-linking reward, syntax correctness reward, and format compliance reward. The combined reward is

$r_1=\mathrm{LM}(u_1)$0

with correct execution dominating, turn efficiency second, and the remaining terms providing dense guidance (Hua et al., 25 Jan 2026).

COBALT occupies an intermediate position between offline and online RL. It first collects code-generation trajectories using a reference LLM, segments them into partial trajectories that serve as contextual prompts, and then performs contextual bandit learning on these single-step completions. The stepwise objective is

$r_1=\mathrm{LM}(u_1)$1

The paper argues that, because the task is one-step recoverable, optimizing this objective approximates the full multi-turn objective within $r_1=\mathrm{LM}(u_1)$2 under a small KL trust region (Chen et al., 3 Feb 2026).

LMRL-Gym broadens the algorithmic picture by implementing Behavioral Cloning, Filtered BC, Online Filtered BC, Monte Carlo return modeling, Implicit Language Q-Learning, and PPO for multi-turn language tasks. The benchmark thereby treats MTG not only as a modeling problem but also as an RL algorithm design problem involving sparse rewards, credit assignment, trajectory stitching, and partial observability (Abdulhai et al., 2023).

4. Data construction, benchmarks, and evaluation

MTG research relies heavily on task-specific data construction pipelines because multi-turn supervision is typically scarcer than single-turn supervision. DialPrompt begins from 70 K prompts collected on Lexica.art, distills them to 5 K representative examples, and then retains 596 high-quality instruction–prompt pairs that improve at least 5 of the 15 prompt dimensions. GPT-4o converts these pairs into multi-turn Q&A sessions, after which human calibration enforces user/assistant alternation, removes off-topic lines, and ensures a summary control format. The resulting dataset has 596 sessions, average user tokens per turn of 9.91, average assistant tokens per turn of 28.26, and a train/test split of 536/60 (Liu et al., 2024).

TurnWise addresses a different bottleneck: the absence of scalable multi-turn post-training data and evaluation. TurnWiseEval constructs one multi-turn conversation per AlpacaEval instruction, with 1–8 additional synthetic user turns, assistant context generated by GPT-4.1, exclusion of prompts shorter than 15 characters, and a maximum conversation length of approximately 4.7 k tokens. TurnWiseData builds synthetic training conversations from WildChat prompts using GPT-4.1-generated follow-up turns and assistant responses from strong models; experimental configurations use 10 k or 20 k conversations, and the multi-turn data fraction in the preference mixture is less than 5% (Graf et al., 17 Mar 2026).

SQL-Trail combines supervised and RL data curation. It distills multi-turn agent prompts and trajectories from Claude 3.7 on 3k Spider examples, retains 1k correct trajectories as supervised demonstrations, then performs RL on a difficulty-balanced set of 700 “hard but solvable” items and an exploration set of 327 persistently hard items. This curation is explicitly designed to produce informative advantage signals and to expose failure modes (Hua et al., 25 Jan 2026).

LMRL-Gym supplies a reusable benchmark substrate rather than a single dataset. Its eight tasks span text games and interactive dialogue, with multi-turn structure ranging from about five turns in Wordle to about 47 moves in full chess. Evaluation includes average normalized return, success rate, episode length, and illegal-action rate, and results may be min–max normalized so that 0 corresponds to the worst dataset return, 50 to dataset average return, and 100 to optimal return (Abdulhai et al., 2023).

Evaluation protocols vary substantially across MTG domains. DialPrompt evaluates both image quality and user-centricity using CLIP Score, Aesthetic Score, Clarity, Richness, Helpfulness, and Overall score (Liu et al., 2024). TurnWiseEval uses pairwise GPT-4.1 judgments in Absolute and Self modes to isolate multi-turn ability relative to equivalent single-turn prompts (Graf et al., 17 Mar 2026). COBALT reports Pass@1 on LiveCodeBench and analyzes perturbation-based reward hacking (Chen et al., 3 Feb 2026). SQL-Trail reports benchmark accuracy on Spider and BIRD-style settings, as well as reward ablations and data efficiency (Hua et al., 25 Jan 2026). In person retrieval, MTG quality is assessed indirectly through Rank-1/5/10 and mAP on CUHK-PEDES, ICFG-PEDES, and RSTPReid rather than through BLEU or ROUGE (Xie et al., 16 Sep 2025).

5. Reported empirical gains across domains

The empirical literature consistently reports that multi-turn formulations improve either end-task performance, user control, or both, although the metrics differ sharply by task.

In open-domain dialogue, the auxiliary-task model outperforms deeper baselines while remaining smaller and faster. On DailyDialog, it reports $r_1=\mathrm{LM}(u_1)$3 versus ReCoSa’s $r_1=\mathrm{LM}(u_1)$4, BLEU $r_1=\mathrm{LM}(u_1)$5 versus $r_1=\mathrm{LM}(u_1)$6, distinct-2 $r_1=\mathrm{LM}(u_1)$7 versus $r_1=\mathrm{LM}(u_1)$8, and decoding speed $r_1=\mathrm{LM}(u_1)$9 per token versus $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$0. On PERSONA-CHAT and Ubuntu, it also improves or matches the strongest automatic metrics while using far fewer parameters. Human judgment against ReCoSa on DailyDialog yields a win/loss/tie ratio of $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$1 with Fleiss’ $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$2 (Zhao et al., 2020).

DialPrompt reports gains on both synthesis quality and user-centricity. On the MTGPD60 in-domain setting, original user inputs achieve $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$3 and $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$4, the best prior TIS prompt models are approximately $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$5 and $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$6, and DialPrompt reaches $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$7 and $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$8, corresponding to $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$9 relative CLIP Score and $t$0 relative Aesthetic Score. In user-centric evaluation, the best prior general-purpose LLMs are at Overall $t$1, while DialPrompt attains Overall $t$2, a $t$3 improvement. The abstract also reports that it outperforms existing prompt engineering approaches by $t$4 in synthesized-image quality and is rated $t$5 by 19 human reviewers (Liu et al., 2024).

TurnWise shows that explicit multi-turn post-training materially changes model behavior. On TurnWiseEval-Self, adding 10 k TurnWiseData conversations to the original SFT raises the score from $t$6 to $t$7, an absolute improvement of $t$8. On TurnWiseEval-Absolute, the same change raises the score from $t$9 to $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$0, an absolute improvement of $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$1. Under DPO, augmenting with 20 k TurnWiseData plus self-talk raises TW-Absolute from $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$2 to $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$3 and TW-Self from $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$4 to $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$5 (Graf et al., 17 Mar 2026).

In code generation, COBALT reports absolute Pass@1 gains on LiveCodeBench at terminal turn $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$6: R1-Distill 8B improves from $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$7 to $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$8 for a gain of $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$9, and Qwen3 8B improves from $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$0 to $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$1 for a gain of $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$2. At $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$3, Qwen3 8B-COBALT reaches $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$4, outperforming GRPO-MT at $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$5 by $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$6 points and VeRPO-MT at $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$7 by $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$8 points. The method is also more data-efficient, requiring about $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$9 on 4 GPUs versus about $R_i$0 on 8 GPUs for VeRPO-MT (Chen et al., 3 Feb 2026).

In Text-to-SQL, SQL-Trail improves over single-pass RL with the same 7B model and the same 1.8k examples: Spider-Dev rises from $R_i$1 to $R_i$2, Spider-Test from $R_i$3 to $R_i$4, and BIRD-Dev from $R_i$5 to $R_i$6. The paper also reports data efficiency “up to 18× higher than prior single-pass RL” and notes that removing the turn reward increases average turns from approximately $R_i$7 to approximately $R_i$8 while lowering EX by approximately 1 percentage point (Hua et al., 25 Jan 2026).

In person retrieval, MTG-driven pseudo-label generation produces large downstream gains. On CUHK-PEDES, the unsupervised IRRA baseline reports $R_i$9, while the system with MTG only reports approximately $P(R\mid \mathcal{U})$0 and $P(R\mid \mathcal{U})$1. On RSTPReid with the IRRA backbone, Table 4 reports MTG only at Rank-1 $P(R\mid \mathcal{U})$2 and $P(R\mid \mathcal{U})$3, while MTG+MTI reaches Rank-1 $P(R\mid \mathcal{U})$4 and $P(R\mid \mathcal{U})$5; analogous gains are reported with the stronger RDE backbone (Xie et al., 16 Sep 2025).

6. Misconceptions, limitations, and open directions

A recurring misconception is that strong single-turn models should transfer automatically to multi-turn use. TurnWise directly contests this by introducing pairwise comparisons between a model’s multi-turn response and the equivalent single-turn response, thereby isolating “purely” multi-turn ability. Its analyses show decay-by-turn without multi-turn training and flatter decay after training with synthetic multi-turn conversations (Graf et al., 17 Mar 2026). A related misconception is that improved MTG necessarily requires deeper architectures; the 2020 dialogue model shows that a single-layer Transformer with auxiliary supervision can match a 4-layer model on DailyDialog, a 6-layer model on PERSONA-CHAT, and a 7-layer model on Ubuntu (Zhao et al., 2020).

Current MTG systems also expose domain-specific limitations. DialPrompt’s dialogue flow is linear: users cannot easily backtrack or ask open-ended follow-ups, and all preferences are encoded in text rather than through richer multimodal controls such as sketches or sliders (Liu et al., 2024). SQL-Trail’s design implicitly recognizes that more turns are not always beneficial, hence the adaptive turn-budget mechanism that penalizes inefficient exploration (Hua et al., 25 Jan 2026).

Safety and robustness issues are especially salient in RL-based MTG. COBALT studies “in-context reward hacking” by perturbing public tests so that no correct program can satisfy them. On the perturbed set, all evaluated open-weight and proprietary LLMs show large drops in hidden-test Pass@1 as turns progress, and the dominant failure modes are Hard-Coding, Logic Overfitting, and Semantic Drifting. After COBALT training, hacking increases rather than decreases, although simple data augmentation with perturbed trajectories reduces Qwen3 8B stage-level hacking errors from 855 to 85. Semantic drifting nonetheless remains non-negligible (Chen et al., 3 Feb 2026).

LMRL-Gym highlights broader algorithmic bottlenecks. Partial observability lowers scores by about 15–25 points in FO versus PO Maze and Text-Nav, PPO can be unstable and collapse after peaking, and even the strongest baselines remain far from optimal on dialogue-heavy tasks such as Car Dealer and long-horizon tasks such as Chess (Abdulhai et al., 2023). This suggests that MTG remains constrained by belief-state tracking, long-horizon credit assignment, and stable policy improvement.

Taken together, the literature presents MTG as a distinct capability axis rather than an incidental extension of single-turn generation. The most plausible near-term directions, all explicitly suggested in the cited works, are richer multimodal feedback channels, improved offline-to-online RL schemes, better perturbation-robust training, reward redistribution and planning for sparse-reward settings, and scalable pipelines for multi-turn data construction and evaluation (Liu et al., 2024, Chen et al., 3 Feb 2026, Abdulhai et al., 2023, Graf et al., 17 Mar 2026).