---
title: State-Update Multi-turn Dialogue Strategy
url: https://www.emergentmind.com/topics/state-update-multi-turn-dialogue-strategy
type: topic
---

# State-Update Multi-turn Dialogue Strategy

State-update multi-turn dialogue strategy denotes a family of dialogue methods that treat conversation as an evolving process in which each turn revises a compact representation of what is currently relevant, instead of repeatedly consuming an ever-growing raw transcript. In the literature, this maintained representation may be a structured dialogue state of domain-slot-value triples, a rewritten self-contained utterance, a standalone reformulated query, a set of supporting sentences, an explicit recalled evidence set, a dependency-based thread structure, a latent dialogue-state vector, or a belief over hidden user state. The updated state is then used for downstream tracking, generation, planning, routing, safety control, or personalization [2009.10430][1906.07004][1903.05164][2509.17766][2605.15102][2503.00187][2605.24647].

## 1. Motivations and problem formulation

A recurring motivation is that naïve multi-turn processing by raw history accumulation is both semantically fragile and computationally inefficient. In long-horizon prompting, models exhibit a recency bias and a “forgetting phenomenon,” with performance degrading as the number of turns grows and with early evidence being used less effectively than late evidence. The same line of work also identifies linear history concatenation as a direct source of rising token cost and inference time [2509.17766].

Other strands identify a more structural problem: the current utterance often does not contain enough information on its own. In ordinary conversation, coreference and information omission are frequent, so a downstream model may fail simply because the utterance is underspecified. One study reports, in a preliminary study on 2,000 Chinese conversations, coreference in **33.5%**, omission in **52.4%**, and neither in **29.7%**, motivating explicit rewriting before downstream interpretation [1906.07004].

In retrieval-based response selection, the difficulty is framed differently but points to the same underlying issue. Binary history–response classification can learn semantic relatedness without learning “what should come next,” because the dialogue is collapsed into a single long document and temporal dependencies between utterances are weakly modeled. This motivates explicit training signals for utterance order, adjacency, and inconsistency detection [2009.04703].

Task-oriented dialogue state tracking (DST) exposes another limitation of uniform context handling. One line of work argues that using one fixed dialogue history for all slots is suboptimal because different slots require different dialogue contents at different turns; a fixed history can therefore be insufficient for some slots and redundant for others [2205.10059]. In multi-domain settings, contextual understanding is further complicated by heterogeneous schemas across domain-specific agents, quadratic growth in schema transitions, and the need to resolve referring expressions without modifying each downstream spoken language understanding subsystem [1903.05164].

The same state-update logic has also been extended beyond task completion. In safety research, multi-turn jailbreaking is described as contextual drift in an evolving conversation, which single-turn prompt filters do not address well. In personalization and counseling, the central claim is that dialogue should be treated as sequential decision-making under partial observability, because the relevant user condition is hidden, evolving, and action-conditioned [2503.00187][2605.24647].

## 2. Forms of dialogue state representation

There is no single canonical state representation. The surveyed literature instead uses several distinct but related forms, each chosen to support a particular downstream objective.

| Representation type | Characteristic form | Representative papers |
|---|---|---|
| Structured symbolic state | Linearized domain-slot-value sequence or slot-conditioned belief state | [2009.10430], [2205.10059] |
| Natural-language reformulation | Rewritten utterance or standalone query | [1906.07004], [1903.05164] |
| Explicit textual memory | Supporting sentences or recalled history turns | [2509.17766], [2605.15102] |
| Structural context decomposition | Dependency-based threads or self-contained sub-dialogues | [2010.01502] |
| Latent dynamical state | User-state embeddings, hidden dialogue state, or belief over hidden user state | [2210.04242], [2503.00187], [2605.24647] |

In task-oriented DST, dialogue state is typically defined as a compact representation of the user goal in dialogue history, often as a set of domain-slot-value triples such as `hotel(price=cheap)` or `taxi(destination=cambridge)`. One sequence-generation formulation linearizes the full state with markup tags, preserving inter-slot relations within a single structured output rather than predicting slots independently [2009.10430]. A related slot-wise model represents the previous state as a concatenation of slot and value markers, then conditions turn-level selection and generation on that explicit prior state [2205.10059].

Natural-language reformulation methods treat the rewritten text itself as the updated state. One rewriting framework learns \(p(R \mid H, U_n)\), where the current utterance is rewritten into a self-contained utterance \(R\) that recovers omitted or coreferred content [1906.07004]. Another approach formalizes contextual query reformulation as learning \(y_t = f_\theta(x_t)\), where \(x_t\) is a window of previous user and system turns and \(y_t\) is a standalone user query suitable for an existing downstream agent [1903.05164].

A third family makes state explicitly textual and cumulative. In prompt-based long-context reasoning, the state is the set of previously selected supporting sentences, carried forward turn by turn via XML tags and a “Previously selected” field [2509.17766]. In self-recall reasoning, the state is the recursively constructed recalled set \(H_k = H_{k-1} \cup \{o_k\}\), where the model emits `<HIS>` tags to copy relevant past utterances into its reasoning trace before producing the answer [2605.15102].

Latent state models move away from directly observable memory. Multi-turn emotional support models construct per-round user-state embeddings \(\mathbf{u}_i\) from system utterance, user utterance, emotion cause, and NRC VAD-based emotion embeddings, then stack them as \(\mathbf{U}_t=[\mathbf{u}_1;\mathbf{u}_2;\dots;\mathbf{u}_{t-1}]\) [2210.04242]. Safety steering models define a hidden dialogue state \(x_k\) in a learned state-space system [2503.00187]. PUMA represents personalization through an approximate belief \(q_\phi(s_t)\) over hidden user state, explicitly separating semantic memory from latent user state [2605.24647].

## 3. Mechanisms of state update across turns

In explicit DST, state update is typically incremental. One tracker compresses earlier information into the \((t-w)\)-th dialogue state \(b_{t-w}\) and combines it with the current \(w\) dialogue turns \(u_{t-w+1} \sim u_t\), so the model updates belief state from recent context plus carried-over prior state rather than rereading full history. Its architecture is coarse-to-fine: a context encoder, a state sketch decoder, a sketch encoder, and a sketch-guided state decoder [2009.10430].

Slot-wise update refines this idea by making history selection conditional on the slot. DiCoS-DST first applies a State Update Predictor to decide which slots should be updated and lets all other slots inherit their previous values. For slots selected for update, it scores each historical turn from three perspectives: explicit connection to the slot name (SN-DH), relevance to the current turn dialogue (CT-DH), and Implicit Mention Oriented Reasoning (IMOR). Only the top-\(k\) selected dialogue contents are then passed to the State Generator, explicitly minimizing distracting information before value prediction [2205.10059].

Text-to-text reformulation realizes state update through compression and rewriting. Utterance rewriting uses the pipeline \((H, U_n) \rightarrow R \rightarrow \text{downstream dialogue processing}\), so the dialogue history is compressed into a single enriched utterance that resolves references and fills in omitted content [1906.07004]. Contextual query reformulation uses a pointer-generator network to rewrite context-dependent input into a standalone query, allowing a production system to preserve multi-turn context without modifying downstream domain-specific agents [1903.05164].

Some methods update state by explicit evidence accumulation rather than full rewriting. State Reconstruction and History Reminder reconstruct each new turn from the current passage plus previously selected supporting sentences, and instruct the model that “Each response should contain ALL supporting sentences (previous + new ones).” The state is therefore the accumulated supporting evidence, not the full transcript [2509.17766]. Self-Recall Thinking performs a comparable update internally: it analyzes the query, recalls selected historical turns with `<HIS>` tags, integrates them into the reasoning chain, and answers from the recalled set \(H_N\) [2605.15102].

Structural decomposition offers a different update mechanism. A dialogue dependency parser can transform history into threads by predicting “reply-to” relations, removing low-confidence edges, and extracting root-to-leaf paths as self-contained sub-dialogues. Each new candidate response is then matched against thread-level representations rather than a flat context [2010.01502]. In response selection, Utterance Manipulation Strategies update the model’s implicit notion of dialogue state through three self-supervised tasks—Insertion, Deletion, and Search—which respectively teach utterance position, inconsistency detection, and temporal adjacency [2009.04703].

## 4. Learning paradigms for state-update behavior

The learning signal for state update varies widely across the literature. A prominent formulation is dual learning. In dual-DST, DST is cast as sequence generation and coupled with a dual utterance generator in an iterative framework. The primal tracker maps dialogue context to state sequence, the dual model maps turn state plus delexicalized system utterance to delexicalized user utterance, and both are updated through reconstruction errors and reward signals without labeled data. External-knowledge rewards from pretrained language models \(LM_u\) and \(LM_s\) are combined with BLEU-based reconstruction rewards as \(r^1 = \alpha r_k^1 + (1-\alpha) r_b^1\) and \(r^2 = \alpha r_k^2 + (1-\alpha) r_b^2\) [2009.10430].

Supervised reformulation methods often rely on copying-oriented objectives. Contextual query reformulation uses a pointer-generator network together with an Entity-Copy Auxiliary Objective, motivated by the observation that different gold rewrites for the same context often contain the same set of entities even when token order differs. The auxiliary classifier predicts which entity tokens should be copied, while inference remains based on the main rewriting model [1903.05164].

A different route is self-supervision. Utterance Manipulation Strategies require no additional annotation: the held-out utterance for insertion, the random utterance for deletion, and the shuffled predecessor search target are all generated automatically from dialogue data. The final training objective is the sum of response-selection loss and auxiliary UMS losses with equal weight [2009.04703].

Strategy induction and reuse make state-update behavior more explicit at the discourse level. IDEAS first induces original strategies from human-machine dialogues, abstracts them into a high-level strategy pool, and then, at each new turn, uses a ranker and user simulator to select a strategy and generate the next instruction conditioned on current dialogue history. A reflection module enforces correctness and coherence by rejecting instructions that contradict prior answers or are already answerable from existing answers [2404.11095].

Self-recall reasoning combines staged supervision with reinforcement learning. SRT first performs supervised fine-tuning so the model can emit recall-based reasoning chains with `<HIS>` tags, then applies GRPO using a composite reward \(R(Z_t, A_t) = R_{\text{format}}(Z_t) + R_{\text{recall}}(\hat{H}_t, H_t^*) + R_{\text{answer}}(A_t, A_t^*)\). The recall term is a scaled Jaccard score over predicted and gold recall sets, directly rewarding minimal sufficient evidence selection [2605.15102].

## 5. Planning, control, and action selection over updated state

Once state is maintained explicitly, many systems use it not only for understanding but for long-horizon decision-making. In knowledge-grounded conversation, one framework defines dialogue strategy as a distribution over knowledge items,
\[
p(\mathcal{Z}|c_t)=p(\mathcal{Z}|u_{t-1})*0.5+p(\mathcal{Z}|c_{t-1})*0.5,
\]
so knowledge selection depends jointly on the partner’s last utterance and prior context. The strategy is optimized with policy gradient under a compound reward that combines informativeness and coherence, while the decoder is kept fixed to preserve linguistic quality [1906.00549].

In emotional support dialogue, MultiESC explicitly scores a candidate strategy by
\[
F(s_t)=g(s_t)+\lambda \cdot h(s_t),
\]
where \(g(s_t)\) is history-based strategy prediction and \(h(s_t)\) is a lookahead heuristic estimating future user feedback over a bounded horizon \(L\) with beam search over top-\(k\) future strategy sequences. The same model maintains a user-state history \(\mathbf{U}_t\) and conditions both planning and generation on it [2210.04242].

State-aware policy learning in open-domain dialogue also uses future-state-dependent reward. WeaSuL models dialogue as trajectories of state-action pairs, weakly labels simulated \((s,a)\) pairs into VeryHigh, High, and Low reward buckets with three quality modules—Semantic Relevant, Semantic Coherence, and Consistent Flow—and further shapes reward with the next state through
\[
f_w(s_t, a_t, s_{t+1}) = g_w(s_t,a_t) + \gamma h(s_{t+1}) - h(s_t).
\]
This makes the learned policy depend not only on whether the current reply is locally good, but on whether it leads to a better next state [2108.01487].

Clinical dialogue makes the sequential-control interpretation explicit. DoctorAgent-RL formulates consultation as an MDP whose action space is \(\mathcal{A} = \{a_{query}, a_{diagnose}\}\). The doctor observes history \(H_t\), asks one question or terminates with a diagnosis, receives a patient response that updates the state, and is optimized by GRPO under rewards for diagnostic accuracy, information acquisition efficiency, and protocol compliance. A Dynamic Turn Budget Training Strategy randomly assigns a turn budget from 2 to 10 and reminds the model of the remaining turns after each interaction [2505.19630].

Routing and negotiation extend state update from content selection to model or policy selection. DialRouter treats multi-turn LLM routing as a sequential decision problem with state \(S_t=[\mathcal{T}_{t-1}, x_t]\), action \(a_t\) as the choice of LLM, offline MCTS for long-horizon trajectory discovery, and retrieval-based future state approximation during policy learning [2604.11095]. GameTalk optimizes full-conversation strategic utility rather than isolated turns, adapting GRPO, DPO, and STaR through branching rollouts from a shared root conversation so that reward depends on the whole interaction trajectory [2601.16276].

Two recent lines generalize state update into formal control or model-based planning. Safety steering models dialogue as a latent dynamical system,
\[
x_k = f_\theta(x_{k-1}, u_k),
\]
and uses a neural barrier function to define a safe set \(\mathcal{S}_k := \{x \in \mathbb{R}^m \mid \phi_k(x) < 0\}\), thereby enforcing invariant safety under contextual drift [2503.00187]. PUMA models personalization as partial observability, learns \(p_\theta(s_t \mid s_{t-1}, a_{t-1})\) and \(p_\theta(o_t \mid s_t)\), maintains a belief \(q_\phi(s_t)\), and selects actions by minimizing expected free energy \(G(a)\), explicitly balancing epistemic and pragmatic objectives [2605.24647]. MAPO, finally, addresses long-horizon subjective dialogue with dense process feedback, Monte Carlo returns, and a mixed advantage estimator that combines turn-level normalized return advantage with batch-level normalized immediate-reward advantage, remaining critic-free throughout [2603.06194].

## 6. Empirical findings across applications

Reported results indicate that state-update methods improve either long-horizon accuracy, efficiency, or robustness, and often all three.

| Area | Representative reported result | Paper |
|---|---|---|
| Task-oriented DST | Dual-DST reaches **49.88% joint accuracy** on MultiWOZ 2.1; its coarse-to-fine tracker reaches **48.79%**, versus **45.60%** for TRADE | [2009.10430] |
| Slot-wise history selection for DST | DiCoS-DST reports **61.02%** joint goal accuracy on MultiWOZ 2.1 and **61.13%** on MultiWOZ 2.2 | [2205.10059] |
| Utterance rewriting | Intent precision improves from **80.77%** to **89.91%**; chit-chat CPS improves from **6.3** to **7.7** | [1906.07004] |
| Query reformulation | InCar ResF1 All improves from **33.6** to **36.1**; on the hardest internal subset \(d \ge 3\), Entity F1 reaches **77.1** | [1903.05164] |
| Prompt-based state reconstruction | On HotpotQA, core information filtering improves by **32.6%**, QA score by **14.1%**, inference time drops by **73.1%**, and token consumption by **59.4%** | [2509.17766] |
| Self-recall reasoning | The abstract reports **4.7%** F1 improvement and **14.7%** end-to-end latency reduction | [2605.15102] |

The state-update idea also improves robustness and strategic accuracy in settings where local matching is insufficient. UMS improves retrieval-based response selection across Ubuntu, Douban, E-Commerce, and Kakao, and in adversarial evaluation the average drop in \(R_{10}@1\) is about **48%** for UMS versus about **58%** for baselines, indicating stronger temporal-consistency modeling [2009.04703].

In emotionally grounded and personalized dialogue, the gains are tied directly to state tracking quality. MultiESC reports **42.01** strategy accuracy and **34.01** weighted-F1, versus **31.61** accuracy for MISC, and is preferred by human evaluators on fluency, empathy, identification of the user’s situation, suggestion quality, and overall support [2210.04242]. On CAMI, PUMA reports **Curr-Acc = 0.689** and **Next-Acc = 0.717**, whereas the long-prompt baseline reaches **Curr-Acc = 0.667** and **Next-Acc = 0.231**, which is a particularly sharp difference on next-state prediction [2605.24647].

A common empirical pattern is that explicit state update is especially valuable when annotation is scarce or dependencies are long-range. Dual learning improves more than pseudo labeling when only **20% labeled data** is available in DST [2009.10430], while SRT is strongest on SRQA, the benchmark specifically designed for long-range dependency [2605.15102].

## 7. Limitations, distinctions, and open questions

A common misconception is to equate state update with symbolic belief tracking alone. The literature shows a much broader design space: state may be a symbolic sequence, a natural-language rewrite, a cumulative evidence set, an explicit recall chain, a latent dynamical vector, or a belief distribution over hidden user state [2009.10430][1903.05164][2509.17766][2503.00187]. What unifies these approaches is not the representational format but the commitment to turn-by-turn revision of a compact decision-relevant state.

The surveyed methods also expose recurring limitations. In utterance rewriting, information completion remains harder than coreference resolution, and positive samples are harder than negative ones [1906.07004]. Prompt-based state reconstruction is manually crafted, depends on correct `<info>` extraction, assumes that relevant evidence can be represented as supporting sentences, and is evaluated primarily on QA-style tasks [2509.17766]. Self-recall reasoning still deteriorates with dialogue length: as dialogue length increases from 8 to 32 turns, Missing Recall rises from **23.5%** to **35.9%**, Wrong Recall rises from **15.0%** to **19.2%**, and Over Recall remains around **5–6%** [2605.15102].

In emotionally sensitive dialogue, improved state-update planning does not eliminate broader safety and personalization concerns. MultiESC is reported to remain often general and repetitive compared with human supporters, may lack personalized, deep understanding or highly specific suggestions, and is explicitly positioned for peer support rather than professional psychotherapy; robust crisis detection is still needed [2210.04242]. PUMA’s main evaluation uses dataset-specific discrete state annotations and simulator-based dynamics, so its dynamic results should not be interpreted as direct clinical effectiveness [2605.24647].

Safety-oriented state update introduces its own trade-off. Neural barrier function steering yields stronger defenses against multi-turn jailbreaks, but the threshold \(\eta\) controls a clear safety-helpfulness frontier: smaller \(\eta\) gives less aggressive filtering and better helpfulness, while larger \(\eta\) yields stronger safety steering with more helpfulness degradation [2503.00187]. Structural decomposition methods also illustrate that not every state-update method is a full dialogue manager: Thread-Encoder is thread-aware and dependency-driven, but it remains fundamentally a response matching model rather than an explicit symbolic state tracker [2010.01502].

Taken together, these results suggest that state-update multi-turn dialogue strategy is best understood as a general architectural principle rather than a single algorithmic recipe. The central research questions now concern which state representation is appropriate for which task, how dense or delayed the supervisory signal should be, and how to balance compactness, interpretability, robustness, and long-horizon control in evolving dialogue.

Source: https://www.emergentmind.com/topics/state-update-multi-turn-dialogue-strategy