---
title: Multi-Turn Text Generation
url: https://www.emergentmind.com/topics/multi-turn-text-generation-mtg
type: topic
---

# Multi-Turn Text Generation

Multi-Turn Text Generation (MTG) denotes a generation regime in which a model produces text over an alternating interaction history rather than from a single isolated prompt. In the cited literature, this regime includes open-domain response generation, dialogue-based prompt generation for text-to-image synthesis, code generation with execution feedback, agentic Text-to-SQL, and simulated multi-turn pseudo-caption generation for person retrieval. The unifying property is history-conditioned generation: if a conversation is written as $(u_1,r_1,\dots,u_T,r_T)$, then each response $r_t$ conditions on the preceding turns, while the “environment” may be a human, a database engine, a code judge, or another model [2603.16759], [2311.18232], [2004.01972], [2408.12910], [2602.03806], [2601.17699], [2509.12662].

## 1. Formalization and problem scope

A basic formal distinction between single-turn and multi-turn generation is given in TurnWise: single-turn generation maps one prompt $u_1$ to one response, $r_1=\mathrm{LM}(u_1)$, whereas multi-turn generation produces $r_t=\mathrm{LM}(u_1,r_1,\dots,u_t)$ for each turn $t$. This formulation emphasizes that MTG is not merely longer prompting; it requires the model to integrate prior turns, preserve coherence, resolve coreference, and adapt to follow-up constraints or corrections [2603.16759].

Earlier dialogue work frames a canonical MTG instance as conditional next-response generation. Given a dataset $\mathcal{D}=\{(\mathcal{U}_i,R_i)\}_{i=1}^N$, where $\mathcal{U}_i=(U_{i,1},\dots,U_{i,n})$ is a sequence of utterances and $R_i$ is the human reply, the task is to learn $P(R\mid \mathcal{U})$. The main difficulties identified there are hierarchical structure across words and utterances, long-range dependency with substantial redundancy, and an efficiency–expressivity trade-off in model design [2004.01972].

A broader decision-theoretic view appears in RL-oriented work. LMRL-Gym models multi-turn text generation as a Markov decision process, or partially observable Markov decision process, in which the state $s_t$ is the interaction history, the action $a_t$ is the next token or utterance, transitions append agent and environment utterances to history, and rewards are task-specific. This formulation is instantiated across text games and interactive dialogue tasks, including Maze, Text-Nav, Wordle, Chess, Endgames, 20 Questions, Guess My City, and Car Dealer [2311.18232].

Specialized MTG tasks refine this abstraction. In multi-turn code generation, the observation sequence includes a problem statement, a generated program, and judge feedback in the form of a failing public test or the message that all public tests passed. COBALT further treats this setting as a finite-horizon MDP with $T\le 8$ and argues that it is “one-step recoverable,” so a suboptimal edit can be fixed in one future turn and stepwise optimization is approximately optimal under a small KL trust region [2602.03806]. In SQL-Trail, the state is $(q,d,h_t)$, comprising a natural-language question, serialized schema, and full action–observation history, while each action includes both a natural-language reasoning trace and an executable SQL snippet [2601.17699].

## 2. Interaction topologies and turn structure

Representative MTG systems differ primarily in what constitutes a turn and what feedback is returned after each turn.

| Setting | Turn structure | Feedback and termination |
|---|---|---|
| Dialogue-based TIS prompting | Select one of 15 dimensions, ask a multiple-choice question, elicit a user preference, refine the prompt | Stop when relevant dimensions are covered or the user requests a summary [2408.12910] |
| Text-to-SQL agent | Emit `<think>` plus `<sql>`, execute SQL, append `<observation>` | Stop on `<solution>` or when the turn budget is exhausted [2601.17699] |
| Person-retrieval pseudo-captioning | Initial caption, multi-turn attribute Q&A, reconstruction | Redundancy filter removes repeated answers; final enriched caption is reconstructed [2509.12662] |
| Multi-turn code generation | Submit code, receive one failing public test or “all passed” | Test-time interaction continues up to $T_{\rm test}=8$ [2602.03806] |

DialPrompt makes the turn structure explicit. Each round $t$ consists of dimension selection and question generation, user preference elicitation, and prompt refinement. With current partial prompt $p^{(t)}$ and a set of 15 dimensions $K$, the system selects a dimension $k^{(t)}\in K$, generates a question $Q^{(t)}=\mathrm{AskDim}(p^{(t)},k^{(t)})$, receives a preference $u^{(t)}$, and refines the prompt by $p^{(t+1)}=f(p^{(t)},u^{(t)},k^{(t)};\theta)$. The 15 dimensions are grouped into Artistic Elements & Techniques, Creative Expression, Visual Impact, and Context & Quality, and the reported dataset average is 6.16 rounds per session with 6.99 dimensions covered [2408.12910].

SQL-Trail centers on an interleaved reason–execute–observe loop. At each turn the policy emits reasoning and a SQL query, the database executes the query, and the resulting dataframe preview or error message is appended as an observation. The framework uses an adaptive turn-budget allocation mechanism keyed to difficulty: simple questions allow $T_{\max}=2$, medium questions allow $T_{\max}=3$, and hard or extra questions may use a global cap such as 10 [2601.17699].

In person retrieval, MTG is used differently: not as interactive inference with a human, but as training-time pseudo-label generation. The process begins with a simple caption $T_s=\mathrm{MLLM}(I,P_{\mathrm{init}})$, proceeds through $N$ rounds of question answering over a dynamic question pool, filters redundant or low-relevance answers, and then reconstructs a final enriched caption $T_e$ by fusing the initial caption with retained answers. The reported setting uses $N=6$ and Qwen2-VL-7B for QA and fusion, with GPT-4o for final paraphrasing [2509.12662].

These examples indicate that “turn” is task-dependent. In some systems it is a natural-language clarification exchange; in others it is a tool call, a code revision, or an attribute query. A plausible implication is that MTG is best understood as an interaction pattern defined by iterative state updates rather than by dialogue alone.

## 3. Architectures and optimization objectives

The architectural spectrum in MTG ranges from deliberately shallow sequence models to full RL fine-tuning over interactive trajectories. In open-domain response generation, a notable design choice is to keep the generator simple and shift context modeling into auxiliary objectives. The 2020 model uses a single Transformer layer for encoding and decoding and augments maximum-likelihood training with four auxiliary tasks: word order recovery, utterance order recovery, masked word recovery, and masked utterance recovery. Its joint objective is
$$
\mathcal{L}_{\mathrm{full}}
=
\mathcal{L}_{\mathrm{MLE}}
+
\alpha\bigl(
\mathcal{L}_{\mathrm{wor}}+
\mathcal{L}_{\mathrm{uor}}+
\mathcal{L}_{\mathrm{mwr}}+
\mathcal{L}_{\mathrm{mur}}
\bigr),
$$
with task-specific attention masks and a read-process-write component for utterance order recovery [2004.01972].

DialPrompt uses a much more standard instruction-tuned LLM backbone. The base model is LLaMA3-8B-Instruct, and no architectural layers are added beyond standard prefix-injection of dialogue history. Training samples are alternating user and assistant messages; the user turns are masked, and the model is trained to predict assistant turns with a cross-entropy loss averaged across dialogue rounds. Training uses AdamW with learning rate $1\times 10^{-4}$, batch size 16, 10 epochs, and 8 A100 GPUs. Because training is performed in full-sequence mode, gradients flow across turn boundaries, so the model jointly learns to ask and to refine [2408.12910].

RL-based MTG introduces reward shaping and online interaction. SQL-Trail adapts Grouped Relative Policy Optimization (GRPO) to a multi-turn Text-to-SQL agent. Its composite reward panel has six terms: final execution reward, turn-count reward, bi-gram overlap reward, schema-linking reward, syntax correctness reward, and format compliance reward. The combined reward is
$$
R(\tau)=5\,r_{exec}+2\,r_{turns}+1\,r_{schema}+1\,r_{bigram}+1\,r_{syntax}+1\,r_{format},
$$
with correct execution dominating, turn efficiency second, and the remaining terms providing dense guidance [2601.17699].

COBALT occupies an intermediate position between offline and online RL. It first collects code-generation trajectories using a reference LLM, segments them into partial trajectories that serve as contextual prompts, and then performs contextual bandit learning on these single-step completions. The stepwise objective is
$$
J_{\rm step}(\pi)=\frac1T\sum_{t=1}^T
\mathbb{E}_{s_t\sim d_{\rm ref}}
\mathbb{E}_{a_t\sim\pi(\cdot\mid s_t)}[R(s_t,a_t)].
$$
The paper argues that, because the task is one-step recoverable, optimizing this objective approximates the full multi-turn objective within $O(T\epsilon)$ under a small KL trust region [2602.03806].

LMRL-Gym broadens the algorithmic picture by implementing Behavioral Cloning, Filtered BC, Online Filtered BC, Monte Carlo return modeling, Implicit Language Q-Learning, and PPO for multi-turn language tasks. The benchmark thereby treats MTG not only as a modeling problem but also as an RL algorithm design problem involving sparse rewards, credit assignment, trajectory stitching, and partial observability [2311.18232].

## 4. Data construction, benchmarks, and evaluation

MTG research relies heavily on task-specific data construction pipelines because multi-turn supervision is typically scarcer than single-turn supervision. DialPrompt begins from 70 K prompts collected on Lexica.art, distills them to 5 K representative examples, and then retains 596 high-quality instruction–prompt pairs that improve at least 5 of the 15 prompt dimensions. GPT-4o converts these pairs into multi-turn Q&A sessions, after which human calibration enforces user/assistant alternation, removes off-topic lines, and ensures a summary control format. The resulting dataset has 596 sessions, average user tokens per turn of 9.91, average assistant tokens per turn of 28.26, and a train/test split of 536/60 [2408.12910].

TurnWise addresses a different bottleneck: the absence of scalable multi-turn post-training data and evaluation. TurnWiseEval constructs one multi-turn conversation per AlpacaEval instruction, with 1–8 additional synthetic user turns, assistant context generated by GPT-4.1, exclusion of prompts shorter than 15 characters, and a maximum conversation length of approximately 4.7 k tokens. TurnWiseData builds synthetic training conversations from WildChat prompts using GPT-4.1-generated follow-up turns and assistant responses from strong models; experimental configurations use 10 k or 20 k conversations, and the multi-turn data fraction in the preference mixture is less than 5% [2603.16759].

SQL-Trail combines supervised and RL data curation. It distills multi-turn agent prompts and trajectories from Claude 3.7 on 3k Spider examples, retains 1k correct trajectories as supervised demonstrations, then performs RL on a difficulty-balanced set of 700 “hard but solvable” items and an exploration set of 327 persistently hard items. This curation is explicitly designed to produce informative advantage signals and to expose failure modes [2601.17699].

LMRL-Gym supplies a reusable benchmark substrate rather than a single dataset. Its eight tasks span text games and interactive dialogue, with multi-turn structure ranging from about five turns in Wordle to about 47 moves in full chess. Evaluation includes average normalized return, success rate, episode length, and illegal-action rate, and results may be min–max normalized so that 0 corresponds to the worst dataset return, 50 to dataset average return, and 100 to optimal return [2311.18232].

Evaluation protocols vary substantially across MTG domains. DialPrompt evaluates both image quality and user-centricity using CLIP Score, Aesthetic Score, Clarity, Richness, Helpfulness, and Overall score [2408.12910]. TurnWiseEval uses pairwise GPT-4.1 judgments in Absolute and Self modes to isolate multi-turn ability relative to equivalent single-turn prompts [2603.16759]. COBALT reports Pass@1 on LiveCodeBench and analyzes perturbation-based reward hacking [2602.03806]. SQL-Trail reports benchmark accuracy on Spider and BIRD-style settings, as well as reward ablations and data efficiency [2601.17699]. In person retrieval, MTG quality is assessed indirectly through Rank-1/5/10 and mAP on CUHK-PEDES, ICFG-PEDES, and RSTPReid rather than through BLEU or ROUGE [2509.12662].

## 5. Reported empirical gains across domains

The empirical literature consistently reports that multi-turn formulations improve either end-task performance, user control, or both, although the metrics differ sharply by task.

In open-domain dialogue, the auxiliary-task model outperforms deeper baselines while remaining smaller and faster. On DailyDialog, it reports $\mathrm{PPL}=38.60$ versus ReCoSa’s $42.34$, BLEU $1.66$ versus $1.12$, distinct-2 $14.95$ versus $10.18$, and decoding speed $12.2\text{ms}$ per token versus $40.9\text{ms}$. On PERSONA-CHAT and Ubuntu, it also improves or matches the strongest automatic metrics while using far fewer parameters. Human judgment against ReCoSa on DailyDialog yields a win/loss/tie ratio of $0.34/0.22/0.44$ with Fleiss’ $\kappa>0.6$ [2004.01972].

DialPrompt reports gains on both synthesis quality and user-centricity. On the MTGPD60 in-domain setting, original user inputs achieve $CS=0.264$ and $AS=5.913$, the best prior TIS prompt models are approximately $CS\approx 0.267$ and $AS\approx 5.932$, and DialPrompt reaches $CS=0.287$ and $AS=6.578$, corresponding to $+8.7\%$ relative CLIP Score and $+11.2\%$ relative Aesthetic Score. In user-centric evaluation, the best prior general-purpose LLMs are at Overall $\approx 5.25$, while DialPrompt attains Overall $=7.69$, a $+46.5\%$ improvement. The abstract also reports that it outperforms existing prompt engineering approaches by $5.7\%$ in synthesized-image quality and is rated $7.9/10$ by 19 human reviewers [2408.12910].

TurnWise shows that explicit multi-turn post-training materially changes model behavior. On TurnWiseEval-Self, adding 10 k TurnWiseData conversations to the original SFT raises the score from $30.1\%$ to $42.9\%$, an absolute improvement of $12.8\%$. On TurnWiseEval-Absolute, the same change raises the score from $10.0\%$ to $24.1\%$, an absolute improvement of $14.1\%$. Under DPO, augmenting with 20 k TurnWiseData plus self-talk raises TW-Absolute from $37.4\%$ to $43.4\%$ and TW-Self from $35.0\%$ to $44.2\%$ [2603.16759].

In code generation, COBALT reports absolute Pass@1 gains on LiveCodeBench at terminal turn $t=8$: R1-Distill 8B improves from $22.7$ to $31.7$ for a gain of $+9.0$, and Qwen3 8B improves from $32.3$ to $38.5$ for a gain of $+6.2$. At $t=3$, Qwen3 8B-COBALT reaches $34.9(\pm 0.3)$, outperforming GRPO-MT at $30.5$ by $+4.4$ points and VeRPO-MT at $33.1$ by $+1.8$ points. The method is also more data-efficient, requiring about $16.9\text{s/example}$ on 4 GPUs versus about $26.7\text{s}$ on 8 GPUs for VeRPO-MT [2602.03806].

In Text-to-SQL, SQL-Trail improves over single-pass RL with the same 7B model and the same 1.8k examples: Spider-Dev rises from $82.8\%$ to $84.5\%$, Spider-Test from $85.1\%$ to $86.1\%$, and BIRD-Dev from $56.3\%$ to $59.3\%$. The paper also reports data efficiency “up to 18× higher than prior single-pass RL” and notes that removing the turn reward increases average turns from approximately $2.3$ to approximately $3.2$ while lowering EX by approximately 1 percentage point [2601.17699].

In person retrieval, MTG-driven pseudo-label generation produces large downstream gains. On CUHK-PEDES, the unsupervised IRRA baseline reports $R1=32.94\%$, while the system with MTG only reports approximately $R1=63.53\%$ and $\mathrm{mAP}\approx 52.37\%$. On RSTPReid with the IRRA backbone, Table 4 reports MTG only at Rank-1 $=52.30\%$ and $\mathrm{mAP}=40.03\%$, while MTG+MTI reaches Rank-1 $=64.20\%$ and $\mathrm{mAP}=48.03\%$; analogous gains are reported with the stronger RDE backbone [2509.12662].

## 6. Misconceptions, limitations, and open directions

A recurring misconception is that strong single-turn models should transfer automatically to multi-turn use. TurnWise directly contests this by introducing pairwise comparisons between a model’s multi-turn response and the equivalent single-turn response, thereby isolating “purely” multi-turn ability. Its analyses show decay-by-turn without multi-turn training and flatter decay after training with synthetic multi-turn conversations [2603.16759]. A related misconception is that improved MTG necessarily requires deeper architectures; the 2020 dialogue model shows that a single-layer Transformer with auxiliary supervision can match a 4-layer model on DailyDialog, a 6-layer model on PERSONA-CHAT, and a 7-layer model on Ubuntu [2004.01972].

Current MTG systems also expose domain-specific limitations. DialPrompt’s dialogue flow is linear: users cannot easily backtrack or ask open-ended follow-ups, and all preferences are encoded in text rather than through richer multimodal controls such as sketches or sliders [2408.12910]. SQL-Trail’s design implicitly recognizes that more turns are not always beneficial, hence the adaptive turn-budget mechanism that penalizes inefficient exploration [2601.17699].

Safety and robustness issues are especially salient in RL-based MTG. COBALT studies “in-context reward hacking” by perturbing public tests so that no correct program can satisfy them. On the perturbed set, all evaluated open-weight and proprietary LLMs show large drops in hidden-test Pass@1 as turns progress, and the dominant failure modes are Hard-Coding, Logic Overfitting, and Semantic Drifting. After COBALT training, hacking increases rather than decreases, although simple data augmentation with perturbed trajectories reduces Qwen3 8B stage-level hacking errors from 855 to 85. Semantic drifting nonetheless remains non-negligible [2602.03806].

LMRL-Gym highlights broader algorithmic bottlenecks. Partial observability lowers scores by about 15–25 points in FO versus PO Maze and Text-Nav, PPO can be unstable and collapse after peaking, and even the strongest baselines remain far from optimal on dialogue-heavy tasks such as Car Dealer and long-horizon tasks such as Chess [2311.18232]. This suggests that MTG remains constrained by belief-state tracking, long-horizon credit assignment, and stable policy improvement.

Taken together, the literature presents MTG as a distinct capability axis rather than an incidental extension of single-turn generation. The most plausible near-term directions, all explicitly suggested in the cited works, are richer multimodal feedback channels, improved offline-to-online RL schemes, better perturbation-robust training, reward redistribution and planning for sparse-reward settings, and scalable pipelines for multi-turn data construction and evaluation [2408.12910], [2602.03806], [2311.18232], [2603.16759].

Source: https://www.emergentmind.com/topics/multi-turn-text-generation-mtg