Multi-Turn Instruction Following

• Multi-turn instruction following is a process where models maintain cumulative context and adapt to evolving directives across multiple dialogue turns.
• Benchmarks use metrics like pass@k and constraint satisfaction rates to evaluate performance drops with increased dialogue complexity.
• Recent advances focus on explicit structural reasoning, iterative feedback loops, and memory augmentation to improve compliance and coherence.

Multi-turn instruction following is a core challenge in the development and evaluation of LLMs, vision-LLMs (LVLMs), and related agentic systems. This task requires not only initial comprehension and execution of a directive, but also the sustained, consistent application of evolving requirements, constraints, or user feedback across multiple dialogue turns or interaction rounds. Multi-turn scenarios stress semantic memory, context maintenance, dynamic plan adaptation, incremental constraint satisfaction, and fine-grained reasoning—often revealing gaps and pathologies missed by single-turn or isolated instruction-following evaluations.

1. Formal Definition and Core Properties

Multi-turn instruction following, as formalized across recent research, involves generating sequential responses that respect both the current user instruction and the cumulative set of constraints or directives introduced throughout the dialogue history. If the dialogue at turn $t$ comprises the sequence of instructions $$\mathcal{I}_1, \mathcal{I}_2, \ldots, \mathcal{I}_t$$ and model responses $\mathit{RES}_1, \ldots, \mathit{RES}_{t-1}$, then the model must compute

$$\mathit{RES}_t = \mathsf{Model}(\{\langle \mathcal{I}_k, \mathit{RES}_k \rangle\}_{k=1}^{t-1},\, \mathcal{I}_t)$$

subject to the satisfaction of all active constraints, which may interact, persist, conflict, or become obsolete as the session evolves (Wang et al., 5 Mar 2025, Li et al., 13 Nov 2025).

The quintessential properties distinguishing multi-turn from single-turn instruction following are:

• Cumulative context tracking: Retention and integration of all prior instructions, including those governing format, content, or style.
• Constraint aggregation and management: Correct information fusion and conflict resolution when new instructions modify, extend, or override earlier constraints.
• Incremental compliance: Ensuring each output satisfies a growing or shifting set of (possibly global) requirements, and that new modifications do not invalidate past accomplishments.
• Dialogue structure understanding: Sensitivity to intentional dialogue flows such as refinement, recall, expansion, and topic switching (Li et al., 20 Feb 2025).

2. Benchmarking Methodologies and Scenario Construction

State-of-the-art benchmarks adopt diverse methodologies to probe multi-turn instruction-following capabilities, spanning code generation (Wang et al., 5 Mar 2025, Duan et al., 1 Jul 2025), open-domain dialogue (Sun et al., 2023, Han, 17 Mar 2025), task-oriented dialogues (Ghazarian et al., 20 Nov 2025), web navigation (Deng et al., 2024), multimodal interaction (Epstein et al., 2024, Han et al., 21 Aug 2025, Li et al., 2023), and complex, dynamically evolving constraint frameworks (Jia et al., 5 Nov 2025, Li et al., 13 Nov 2025, Yang et al., 4 Feb 2026). Key design principles include:

a) Task Decomposition and Testable Constraints: Multi-turn scenarios are constructed by decomposing a base instruction into a sequence of verifiable sub-requirements (e.g., input/output invariants, style, API usage, format/output checks, or domain-specific policy conditions). For code, every instruction and response pair is objectively validated by associated test cases or static/dynamic analysis tools (Wang et al., 5 Mar 2025, Duan et al., 1 Jul 2025).

b) Structural Flow and Inter-turn Relations: Structural benchmarks, such as StructFlowBench, explicitly model the higher-order flow of dialogue, labeling turns with structured relations: Follow-up, Refinement, Recall, Expansion, Summary, or Unrelatedness. Dialogue plans can be parameterized by structural templates and task types, allowing controlled evaluation of recall, refinement, and expansion capacities (Li et al., 20 Feb 2025).

c) Edge Cases: Conflict and Entanglement: Specialized datasets (e.g., MultiTurnInstruct (Han, 17 Mar 2025)) probe the ability of models to handle conflicting, entangled, or privacy-protecting instructions, and to prioritize among conflicting directives using explicit or implicit precedence rules.

d) Feedback-Driven Repair and Iterative Correction: Some frameworks, such as Meeseeks and MultiCodeIF, simulate iterative user feedback, providing explicit correction prompts when outputs fail to satisfy constraints. The process continues until either convergence (all requirements met) or a fixed iteration limit is reached. This tests both the model's error localization and its capacity for self-correction under explicit guidance (Wang et al., 30 Apr 2025, Duan et al., 1 Jul 2025).

e) Multimodal and Multilingual Complexity: Recent extensions (e.g., MMMT-IF (Epstein et al., 2024), Multi-IF (He et al., 2024)) incorporate images, global formatting instructions, and non-English/interleaved scripts to systematically stress retrieval and compliance across rich, realistic conversational settings.

3. Evaluation Metrics and Quantitative Findings

Robust evaluation of multi-turn instruction following leverages a suite of task-sensitive, verifiable, and often automatically computable metrics. Salient examples include:

Metric Name	Mathematical Definition/Type	Usage Context
pass@k	$1 - \binom{n-c}{k}/\binom{n}{k}$	Code generation functional/constraint correctness
Programmatic IF (PIF)	$$\operatorname{PIF}(X,Y)=\frac{\# \text{ satisfied instructions}}{\# \text{given instructions}}$$	Multimodal answer compliance (Epstein et al., 2024)
Constraint Sat. Rate (CSR)	$$\frac{1}{M}\sum_{i=1}^{M}\frac{1}{n_i}\sum_{j=1}^{n_i}s_{i,j}$$	General, per-turn constraint satisfaction
Instruction Sat. Rate (ISR)	$$\frac{1}{M}\sum_{i=1}^{M} \mathbf{1}[\sum_j s_{i,j}=n_i]$$	Exact, all-constraints satisfaction rate
Weighted CSR (WCSR)	$(\sum_j w_j s_j)/(\sum_j w_j)$	Dual weighting of intra-turn and inter-turn constraints
Utility Rate/Meeseeks Score	Tag-wise coverage, mean per example	Fine-grained feedback and recovery metrics

Key empirical trends are consistent across domains:

• There is a steep, monotonic drop in instruction-following metrics as the number of dialogue turns, instruction complexity, or accumulated constraints increases (e.g., a 20–25pp pass@1 decrease from turn 1 to turn 5 in code, 0.877→0.707 on Multi-IF (He et al., 2024)).
• Complex, multi-level, or hierarchical instructions see marked performance collapse, with hard satisfaction rates plummeting in multi-constraint settings (e.g., 54.5%→18.8% as the instruction level increases in MultiCodeIF (Duan et al., 1 Jul 2025)).
• Explicit presentation of all active constraints (e.g., appending instructions at the prompt tail) substantially ameliorates the compliance drop, implicating instruction retrieval, rather than raw execution capacity, as a key failure point (Epstein et al., 2024).
• Error recovery rates—even in settings with automated, feedback-driven repair—remain modest (typically <30%) and full iteration is often required to achieve close to maximal compliance (Duan et al., 1 Jul 2025, Wang et al., 30 Apr 2025).
• Qualitative analyses reveal persistent failures in precise context reuse, formatting, fine-grained exception handling, privacy/conflict management, and chain-of-thought reasoning.

4. Modeling Advances for Multi-Turn Instruction Following

Recent research pursues several orthogonal strategies to advance multi-turn instruction following efficacy:

a) Explicit Structural and Graph Reasoning: Several works (GraphIF (Li et al., 13 Nov 2025), ImpRIF (Yang et al., 4 Feb 2026)) recast dialogue history and instruction dependencies as explicit graph structures (relation graphs, verifiable reasoning graphs). Modules extract inter-turn relations as labeled edges (e.g., global constraints, context anchoring) and render them as "graph prompts"—enabling models to globally align outputs with the accumulated set of requirements. Chain-of-thought or planning heads enforce topological orderings in reasoning.

b) Iterative Feedback and Self-Repair Loops: Benchmarks and frameworks such as Meeseeks (Wang et al., 30 Apr 2025) and MultiCodeIF (Duan et al., 1 Jul 2025) incorporate iterative feedback, requiring models to not only generate a compliant output but, on failure, refine their generation in light of fine-grained, automated error reports. This tests not just comprehension but repair and adaptation over multiple correction cycles.

c) Memory Augmentation and Retrieval Mechanisms: Planner agents such as Self-MAP (Deng et al., 2024) and CoLVLM (Han et al., 21 Aug 2025) employ explicit memory modules—short-term and long-term stores—to encode entities, sub-tasks, prior actions, and context. Retrieval-augmented planners fetch and summarize contextually relevant snippets, supporting coreference, anaphora, and cross-turn consistency.

d) Reinforcement Learning and Constraint Decomposition: Self-supervised RL methods (Ren et al., 16 Oct 2025) and curriculum strategies (incremental constraint synthesis) directly maximize dense, decomposed rewards over multiple constraints using programmatically derived or pseudo-labeled signals; this sidesteps the need for external supervision and addresses the sparse-reward problem inherent to multi-turn, multi-constraint MDPs.

e) Data-Driven Strategies and Preference Optimization: Human-in-the-loop datasets featuring anaphora, ellipsis, conflict, and priority manipulation (e.g., Parrot (Sun et al., 2023), MultiTurnInstruct (Han, 17 Mar 2025)) are used to expose models to realistic, diverse conversational phenomena. Context-aware preference optimization promotes subtle learning of context use and resolution.

5. Structural, Multimodal, and Multilingual Extensions

Recent benchmarks extend multi-turn instruction following beyond standard single-modal or English-only settings:

• Structural Flow Modeling: StructFlowBench (Li et al., 20 Feb 2025) formalizes six inter-turn relations and emphasizes the necessity of structural, cross-turn adherence alongside fine-grained per-turn correctness. Weighted metrics that privilege structural constraints improve alignment with human dialogic intent.
• Multimodal Complexity: MMMT-IF (Epstein et al., 2024) and TextBind (Li et al., 2023) stress multi-turn, multi-image, and multi-instruction compliance (including answer-formatting and context retrieval) in both input and output modalities, with programmatic (script-based) metrics and ablation studies distinguishing retrieval from compliance difficulty.
• Multilingual Robustness: Multi-IF (He et al., 2024) and Arabic MT-Bench (Boughorbel et al., 2023) show that error rates increase for non-Latin scripts and longer context, even for state-of-the-art models. Training on diverse multi-turn, multi-language data and employing prompt engineering to summarize active constraints can partially mitigate this gap.

6. Long-horizon Interaction, Limitations, and Future Directions

Extended, unbounded, or evolving dialogues, as modeled in EvolIF (Jia et al., 5 Nov 2025), stress the practical durability of LLMs' multi-turn instruction-following. Metrics such as Average Conversation Turns (ACT), Robustness (ROB), Longest Satisfaction Sequence (LSS), and Recovery (REC) reveal:

• Even state-of-the-art models sustain only 14–18 consecutive perfect turns before compliance fails.
• Robustness and recovery rates stagnate well below 75% (e.g., 70.31% for GPT-5 vs. 59.90% for Gemini-2.5-Pro).
• Failures are most prevalent for global, count-based, and entangled constraints, reflecting persistent deficits in memory, attention distribution, and planning.

Best practices and future research include:

• Architectures leveraging explicit memory and structural planning (Li et al., 13 Nov 2025, Deng et al., 2024).
• Incorporating chain-of-thought, self-reflective, and feedback-aware training (Yang et al., 4 Feb 2026, Wang et al., 30 Apr 2025).
• Systematic coverage of inter-turn and structural relations in pretraining and fine-tuning.
• Multimodal, multilingual, and conflict-heavy datasets for coverage-driven improvement.
• Iterative, verifiable, and automated evaluation harnesses to enable unbiased, scalable progress tracking.

A plausible implication is that scalable progress in multi-turn instruction following depends critically on (1) explicit modeling of latent structure within and across user instructions, (2) reward-aligned or feedback-driven adaptation mechanisms, and (3) sustained attention to context compression, retrieval, and reasoning over long conversational histories.

---

References

• (Wang et al., 5 Mar 2025) CodeIF-Bench: Evaluating Instruction-Following Capabilities of LLMs in Interactive Code Generation
• (Han et al., 21 Aug 2025) ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following
• (Epstein et al., 2024) MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark
• (Deng et al., 2024) On the Multi-turn Instruction Following for Conversational Web Agents
• (Li et al., 20 Feb 2025) StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
• (Sun et al., 2023) Parrot: Enhancing Multi-Turn Instruction Following for LLMs
• (Ghazarian et al., 20 Nov 2025) TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues
• (Duan et al., 1 Jul 2025) A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
• (He et al., 2024) Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
• (Jia et al., 5 Nov 2025) One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework
• (Wang et al., 30 Apr 2025) Ask, Fail, Repeat: Meeseeks, an Iterative Feedback Benchmark for LLMs' Multi-turn Instruction-Following Ability
• (Ren et al., 16 Oct 2025) Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
• (Li et al., 13 Nov 2025) GraphIF: Enhancing Multi-Turn Instruction Following for LLMs with Relation Graph Prompt
• (Han, 17 Mar 2025) Can LLMs Follow Multiple Turns of Entangled Instructions?
• (Yang et al., 4 Feb 2026) ImpRIF: Stronger Implicit Reasoning Leads to Better Complex Instruction Following