---
title: Multi-Turn Instruction Following (MTIF)
url: https://www.emergentmind.com/topics/multi-turn-instruction-following-mtif
type: topic
---

# Multi-Turn Instruction Following (MTIF)

Multi-Turn Instruction Following (MTIF) is the capability of a language model, agent, or vision-language system to satisfy instructions across a sequence of dialogue turns rather than within a single prompt. In MTIF, later user turns may be elliptical, anaphoric, accumulative, corrective, or conflicting with earlier ones, so correct behavior depends on preserving and updating dialogue state, retrieving relevant prior constraints, and applying the appropriate priority or structural relation at the current turn. Recent work has treated MTIF as a general problem spanning conversational web agents, multilingual dialogue, multimodal interaction, task-oriented dialogue, code generation, and long-horizon assistant behavior [2310.07301] [2402.15057] [2502.14494] [2503.13222] [2309.08637].

## 1. Problem formulation and conceptual scope

MTIF departs from ordinary single-turn instruction following because the current instruction is often not self-contained. Early work on Parrot characterizes the phenomenon through realistic follow-up language containing anaphora and ellipsis, such as context-dependent requests that cannot be resolved from the latest utterance alone [2310.07301]. MultiTurnInstruct sharpens this view by defining three levels of difficulty: retrieving information from instructions, tracking and reasoning across turns, and resolving conflicts among instructions [2503.13222]. In task-oriented dialogue, TOD-ProcBench formulates the problem as selecting the relevant segment of a natural-language policy, understanding the active condition path, and then producing the correct next action or response under nested, fine-grained constraints [2511.15976].

In agentic settings, the state dependence is explicit. MT-Mind2Web introduces **Conversational Web Navigation**, where the agent receives conversation history $C_t=\{q_1,A_1,\ldots,A_{t-1},q_t\}$ together with the current environment state $E_t$, and must predict the current turn’s action sequence $A_t$ [2402.15057]. This setting differs from ordinary conversational benchmarks because history includes not only user utterances but also prior actions, page states, and intermediate outcomes. The same broad pattern appears in visually grounded MTIF: CoLVLM Agent models each turn as a function of current visual observation, current instruction, and prior memory state, while CAMVR treats the problem as context-aware multi-turn visual-textual inference requiring persistent visual-textual memory and adaptive visual focus [2508.15164] [2509.05669].

Across these formulations, MTIF is not merely “longer prompting.” This suggests that MTIF is better understood as stateful instruction adherence under evolving dialogue conditions: the system must remember, retrieve, prioritize, and sometimes revise earlier commitments rather than simply answer the most recent query [2402.15057] [2503.13222] [2511.03508].

## 2. Inter-turn dependency structures

A major line of work argues that MTIF requires explicit modeling of how one turn depends on another. StructFlowBench makes this claim directly by treating **structural flow** as a second evaluation axis beyond constraint satisfaction, and defining six inter-turn relationships: **Follow-up**, **Refinement**, **Recall**, **Expansion**, **Summary**, and **Unrelatedness** [2502.14494]. The first five become structural constraints in the benchmark; **Unrelatedness** is treated as a natural conversational phenomenon but excluded from the structural constraint set because it does not encode a dependency to be preserved.

EvolIF generalizes this insight into a dynamic conversation generator with three layers: a **topic layer**, an **instruction layer**, and a **constraint layer** [2511.03508]. At each turn, the framework may continue a topic, introduce a new one, or backtrack to a previous topic; then it may add, remove, or modify constraints; finally it may perturb constraint parameters. Conversation length is not fixed in advance, because termination is controlled by simulated user patience rather than a predetermined number of turns. This design shifts MTIF evaluation from fixed-turn accuracy toward sustained interaction quality.

Several papers focus on accumulated or hierarchical constraint state. ImpRIF synthesizes both **system-instruction dialogues**, where system instructions have higher priority, and **user-accumulated dialogues**, where new constraints are added at each user turn and all historical constraints remain relevant [2602.21228]. CAPITU operationalizes accumulation more simply but very directly: in its three-turn conversations, each turn adds a new instruction while preserving all previous instructions, so by Turn 3 the model must satisfy three simultaneous constraints in a coherent literary discussion [2603.22576]. SEQUOR expands the same idea to long-horizon open-domain conversation, varying whether constraints are introduced once, accumulated, replaced, or mixed over fifty turns [2605.06353].

These dependency structures reveal an important misconception. MTIF is not exhausted by remembering facts from prior turns; it also requires recognizing whether the current turn is a refinement, a recall request, a summary over multiple prior turns, a return to an earlier topic, or a request that supersedes previous instructions. The literature repeatedly treats these relations as first-class objects rather than incidental prompt artifacts [2502.14494] [2511.03508] [2511.10051].

## 3. Benchmarks and evaluation methodology

The benchmark landscape reflects the breadth of the problem. Some datasets stress conversational context in web agents, some target accumulated constraints in text or multimodal dialogue, and others isolate structural dependence or long-horizon persistence.

| Benchmark | Setting | Key property |
|---|---|---|
| MT-Mind2Web | Conversational web navigation | 720 sessions; 3,525 instruction-action pairs; about 5 turns per conversation |
| MMMT-IF | Image-based multi-turn Q&A | 71 chats; 990 turns; 1–6 global instructions per chat |
| StructFlowBench | Structured multi-turn dialogue | 155 dialogues; 643 turns; 1,775 constraints |
| SEQUOR | Long realistic constraint following | 200 personas; 50 tasks each; conversations run to 50 turns |
| CAPITU | Brazilian Portuguese MTIF | 100 conversations; 3 turns; 300 turn-level instances |
| MultiTurnInstruct | Entangled instructions | ~1.1K conversations; nine capability categories |

Representative evaluation metrics vary with the setting. MT-Mind2Web uses **Element Accuracy**, **Operation F1**, **Step Success Rate**, and **Turn Success Rate**, with **Turn Success Rate** as the main metric [2402.15057]. StructFlowBench uses **CSR**, **ISR**, **DRFR**, and **WCSR**, where structural constraints receive higher weight than intra-turn constraints in **WCSR** [2502.14494]. MultiCodeIF evaluates fine-grained code compliance with **SSR**, **HSR**, and **IFRepair@k**, making the repair trajectory itself part of MTIF assessment [2507.00699]. Meeseeks introduces **Utility Rate** for all-requirements usability and **Meeseeks Score** for aggregated capability-tag accuracy under iterative feedback [2504.21625].

MMMT-IF is notable for making instruction adherence fully executable. It defines **Programmatic Instruction Following (PIF)** as
$$
\operatorname{PIF}(X,Y) = \frac{\# \text{ Instructions given in input context }X\text{ that are followed in response }Y}{\# \text{ Instructions given in input context }X},
$$
and extends this with **PIF-N-K**, which measures whether at least $K$ of $N$ sampled responses achieve perfect instruction following [2409.18216]. The paper reports that PIF correlates with human instruction-following ratings at **0.60** on **990 samples**, while avoiding the model-identity bias observed in LLM-based judges.

A separate line isolates the effect of simultaneous constraint count. ManyIFEval and StyleMBPP keep the underlying task fixed while varying the number of instructions, evaluating **Prompt-level Accuracy** and **Instruction-level Accuracy** and then fitting regression models to estimate unseen instruction combinations and unseen instruction counts [2509.21051]. This literature sits adjacent to MTIF proper, but it is directly relevant because many multi-turn settings implement increasing constraint load by accumulation rather than by replacing earlier instructions.

## 4. Methodological approaches

Research on MTIF has produced both training-time and inference-time methods. One approach is to build better data and better objectives for conversational dependence. Parrot introduces **Parrot-Ask**, a trainable user simulator for collecting human-like multi-turn instructions, and **Context-aware Preference Optimization (CaPO)**, which trains against three context errors: **Context Neglect**, **Context Hallucination**, and **Context Misunderstanding** [2310.07301]. M2Lingual constructs a fully synthetic multilingual multi-turn instruction-finetuning dataset by applying an **Evol** taxonomy to multilingual seeds and then generating follow-up turns through a 21-type multi-turn taxonomy covering follow-up, refinement, expansion, and recollection [2406.16783]. MathChat$_$ takes the same data-centric route for mathematics, generating **147,478** synthetic math dialogues to improve interaction and instruction following in error correction, error analysis, follow-up QA, and problem generation [2405.19444].

A second approach treats MTIF as selective memory management. In conversational web navigation, **Self-MAP** uses a three-stage pipeline of **memory**, **reflection**, and **planning** [2402.15057]. It stores prior interaction steps in a memory bank, retrieves the top-$K$ relevant snippets using multifaceted matching over the current instruction and ongoing trajectory, simplifies stored environment state with HTML element ranking, adds LLM-generated rationales, and then plans the next action by direct generation or MCQ-style selection. In multimodal dialogue, CoLVLM Agent applies an iterative **memory-perception-planning-execution** cycle, while CAMVR introduces a **Visual-Textual Context Memory Unit (VCMU)** and **Adaptive Visual Focus Guidance (AVFG)** so that relevant visual regions are selected using retrieved dialogue context [2508.15164] [2509.05669].

A third approach makes inter-turn structure explicit. GraphIF is a training-free framework that represents dialogue as a directed relation graph $\mathcal{G}=\{\mathcal{V},\mathcal{E},\mathcal{R}\}$, where nodes are dialogue turns and labeled edges encode relations such as **Identify_Global_Constraint**, **Identify_Context_Anchored**, **Identify_Modify**, **Identify_Summary**, and **New_Topic** [2511.10051]. An agent-based extractor builds the graph, a graph-prompt module verbalizes the structure, and a rewrite module refines the initial response. StructFlowBench performs a similar move at the benchmark level by assigning each turn a structural relation and evaluating whether the response respects that relation [2502.14494].

A fourth approach focuses on latent reasoning structure rather than explicit memory alone. ImpRIF formalizes complex instructions as **explicit reasoning graphs (ERGs)**, directed acyclic graphs whose nodes are **conditional**, **mathematical**, or **knowledge** nodes [2602.21228]. It synthesizes single-turn and multi-turn data from these graphs, supervises graph-driven chain-of-thought during SFT, and uses **GRPO**-style reinforcement learning with task reward, thinking-process reward, and partial reward. The central claim is that strengthening implicit reasoning over hidden constraint structure improves complex instruction following, including multi-turn dialogue settings.

Finally, several benchmarks explicitly operationalize self-correction. Meeseeks feeds back unmet requirements for up to three rounds, and MultiCodeIF runs up to four repair rounds using structured diagnostics for violated code constraints [2504.21625] [2507.00699]. These settings model a common deployment pattern in which the user does not abandon the conversation after the first failure but instead specifies what was missed.

## 5. Empirical regularities and recurring failure modes

The most stable empirical result in the MTIF literature is that performance declines as turns, active constraints, or both increase. MMMT-IF reports that average PIF across Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet drops from **0.81 at turn 1** to **0.64 at turn 20**, and that when each response is repeated four times, **PIF-4-4** is only **11%** for Gemini and GPT-4o [2409.18216]. ManyIFEval finds the same monotonic degradation under controlled instruction-count growth, with GPT-4o on ManyIFEval falling from **0.94 at $n=1$** to **0.21 at $n=10$** [2509.21051]. SEQUOR extends the pattern to long-horizon conversation: even in the **Single** regime, instruction-following accuracy decreases as the conversation grows longer; the abstract highlights drops exceeding **11%** for a single constraint, over **40%** when multiple constraints must be followed simultaneously, and more than **9%** when constraints are added or replaced at arbitrary points [2605.06353].

The second recurring result is that retrieval and relevance filtering matter as much as raw context length. In MMMT-IF, appending all instructions again at the end of the context improves PIF by **22.3 points on average**, indicating that part of the challenge is retrieving distributed instructions rather than merely executing them once found [2409.18216]. MT-Mind2Web reaches a closely related conclusion in web agents: relevance-based memory retrieval outperforms prepending full history, and prompt analysis shows that relevance-based ordering of memories works better than chronological ordering [2402.15057]. GraphIF likewise outperforms MemoryBank and MemoChat by explicitly modeling inter-turn relations rather than relying on coarse summaries or retrieval alone [2511.10051].

A third regularity is that recall is not enough. MultiTurnInstruct reports strong performance on **Memorization**, with GPT-4o reaching average **BLEU 0.821** and **61%** of examples at **1.00 BLEU**, yet models still struggle on **Privacy Protection**, **Personalization**, and **Prioritization** [2503.13222]. The paper’s attention analysis suggests that failures arise from weak integration and prioritization of earlier instructions rather than from simple information loss. This distinction is echoed elsewhere: SEQUOR notes that in the **Everything** regime, relevant constraints are introduced within at most the previous 15 turns, so degradation cannot be explained solely by context-window truncation [2605.06353].

A fourth regularity is that explicit feedback helps but does not eliminate the problem. MultiCodeIF reports that average hard satisfaction can rise from **63.0%** to **83.4%** over four repair rounds for Claude-3-7-Sonnet, yet multi-level constraints still reduce hard success from **54.5%** in single-level settings to **18.8%** in multi-level scenarios [2507.00699]. Meeseeks finds that most models improve by more than **50 percentage points** in **Utility Rate** from round 1 to round 3, but still shows persistent weaknesses in language constraints and word-count constraints [2504.21625].

Finally, several works identify specific hard subproblems. StructFlowBench reports that **Refinement** is the hardest structural relation, with even the best model only reaching about **0.8** on refinement [2502.14494]. CAPITU identifies exact word count, max word repeat, character-count range, and acrostic as especially difficult, and documents turn-level persistence degradation as constraints accumulate [2603.22576]. MT-Mind2Web shows that removing **memory simplification** causes the largest ablation drop, indicating that token-budget management and noisy HTML reduction are crucial for long conversational contexts [2402.15057].

## 6. Multimodality, multilinguality, and domain-specific extensions

MTIF is not restricted to text-only chat. TextBind addresses **multi-turn interleaved multimodal instruction-following in the wild**, where text and images can appear on either side of the dialogue and the assistant may need to understand, compare, refer back to, or generate images across multiple turns [2309.08637]. It synthesizes conversations from image-caption pairs with almost no human annotation and trains **MIM**, an architecture that integrates a vision encoder, a language model, and an image decoder. In human evaluation, TextBind-trained MIM exceeds LLaVA on instruction-following and multimodal context understanding, while its performance gap over baselines grows as the number of previous turns increases.

More recent visually grounded MTIF work emphasizes explicit state management. MMDR-Bench evaluates visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency, while CoLVLM Agent improves average human evaluation score to **4.03**, ahead of GPT-4o at **3.92** and Gemini 1.5 Pro at **3.85** [2508.15164]. CAMVR reports state-of-the-art performance on VisDial, adapted A-OKVQA, and a novel MTIF dataset by combining dynamic read-write memory and context-conditioned visual guidance [2509.05669]. These results support the claim that visually grounded MTIF is limited not only by perception quality but also by context retention and instruction chaining.

Multilingual MTIF exposes another axis of difficulty. An Arabic case study built from a translated MT-Bench reports that most open models perform better in English than Arabic, while GPT-4 remains strongest in both languages and Jais-13B-chat is the best open Arabic model with an average score of **5.08** [2310.14819]. M2Lingual addresses this problem by constructing **182K** multilingual instruction-response pairs covering **70 languages** and **17+ NLP tasks**, with strong gains on MT-Bench and on multilingual QA, summarization, math, and classification [2406.16783]. CAPITU adds a culturally grounded Brazilian Portuguese benchmark with automatically verifiable constraints and multi-turn accumulation, showing conversation-level accuracy from **60%** to **96%** across models [2603.22576].

Domain-specific MTIF benchmarks further show that the problem is not domain-agnostic prompt formatting. MathChat demonstrates that math-specialized models strong on single-turn QA underperform on multi-turn follow-up QA, error correction, error analysis, and problem generation, with round-to-round accuracy degradation and instruction misunderstanding in open-ended tasks [2405.19444]. TOD-ProcBench shows that even strong LLMs struggle to retrieve relevant policy fragments, detect subtle instruction-violating responses, and generate policy-compliant responses in multi-turn task-oriented dialogues [2511.15976]. MultiCodeIF reaches the same conclusion for programming, where models handle explicit constraints better than implicit or abstract ones and benefit from structured repair feedback without closing the gap on multi-level compositions [2507.00699].

## 7. Limitations, controversies, and open research directions

The MTIF literature is unusually explicit about evaluation caveats. MMMT-IF shows that LLM-based judges can be biased toward answers from the same model family, motivating code-executed scoring whenever constraints are objectively verifiable [2409.18216]. The Arabic MT-Bench study uses GPT-4 as a uniform evaluator for English and Arabic but notes that judged scores may not perfectly reflect human judgment and that Arabic evaluation may still be conservative [2310.14819]. StructFlowBench relies on GPT-4o as judge under a **Golden Context** strategy and reports a **Cohen-like Kappa of about 0.75** with expert judgments on sampled dialogues, which is supportive but not equivalent to direct human evaluation at scale [2502.14494].

Several benchmarks also delimit the scope of what is being measured. MMMT-IF states explicitly that PIF measures instruction compliance, not factual correctness of the answer to the image question [2409.18216]. CAPITU combines instruction compliance and coherence through
$$
\text{Final Score} = \frac{2 \cdot I + C}{3},
$$
thereby acknowledging that mechanically compliant responses can still be poor dialogue [2603.22576]. SEQUOR uses LLM-as-a-judge turn by turn and verifies judge quality on gold responses, but the benchmark remains a simulation of persona-driven open-domain interactions rather than a live human deployment setting [2605.06353].

Benchmark construction itself remains an open problem. MT-Mind2Web notes that its dataset and method are currently focused on **HTML-grounded** web agents and that evaluation is **offline**, leaving dynamic online interaction and multimodal pages as open challenges [2402.15057]. StructFlowBench assigns only one structural relation per turn even though real turns may simultaneously instantiate recall and follow-up or other overlapping relations [2502.14494]. EvolIF argues that fixed-turn benchmarks saturate quickly and proposes a patience-based evolving framework precisely because short, static benchmarks understate long-horizon breakdown [2511.03508].

The combined record suggests several convergent research directions. One is stronger explicit state representations for active, obsolete, and superseded constraints [2605.06353] [2511.03508]. Another is better integration of retrieval with reasoning, since several papers show that having the relevant history in context does not guarantee correct arbitration among competing instructions [2409.18216] [2503.13222]. A third is broader coverage of non-English and culturally grounded settings, where native benchmark design appears more informative than translation alone [2406.16783] [2603.22576]. A fourth is the extension of MTIF from text to persistent multimodal and agentic environments, where actions, visual state, and external tools become part of the instruction-following state itself [2402.15057] [2508.15164] [2509.05669].

Taken together, the literature portrays MTIF as a distinct systems-level competency rather than a small variant of ordinary instruction following. The central difficulty is not only remembering prior turns, but preserving the right constraints, retrieving the relevant state, resolving structure and priority across turns, and remaining robust as conversations become longer, more multimodal, and more procedurally entangled.

Source: https://www.emergentmind.com/topics/multi-turn-instruction-following-mtif