A Survey on Multi-Turn Interaction Capabilities of Large Language Models

Abstract: Multi-turn interaction in the dialogue system research refers to a system's ability to maintain context across multiple dialogue turns, enabling it to generate coherent and contextually relevant responses. Recent advancements in LLMs have significantly expanded the scope of multi-turn interaction, moving beyond chatbots to enable more dynamic agentic interactions with users or environments. In this paper, we provide a focused review of the multi-turn capabilities of LLMs, which are critical for a wide range of downstream applications, including conversational search and recommendation, consultation services, and interactive tutoring. This survey explores four key aspects: (1) the core model capabilities that contribute to effective multi-turn interaction, (2) how multi-turn interaction is evaluated in current practice, (3) the general algorithms used to enhance multi-turn interaction, and (4) potential future directions for research in this field.

• The paper presents a comprehensive survey of multi-turn interaction capabilities in LLMs, emphasizing benchmark frameworks like MT-Bench and 'LLM-as-a-Judge'.
• The methodology integrates analysis of context memory, planning, and reasoning, utilizing techniques such as recursive summarization and hierarchical reinforcement learning.
• The findings highlight future research directions focused on user satisfaction, calibrated evaluators, and leveraging real interaction data to improve dialogue systems.

Overview of Multi-Turn Interaction Capabilities of LLMs

Recent advancements in LLMs significantly extend the capabilities of dialogue systems to handle multi-turn interactions across various applications. These capabilities are crucial for enabling dynamic interactions between agents and users or environments, contributing to areas such as conversational search, consultation services, and interactive tutoring.

Evaluation Practices

Evaluation plays a pivotal role in advancing LLM research by providing benchmarks and metrics to assess multi-turn interaction capabilities. This paper reviews various frameworks for evaluating user-LLM interactions, focusing on aspects such as naturalness, task completion, and user satisfaction. Notable methods include user preference evaluations through platforms like MT-Bench and Chatbot Arena, where frameworks like "LLM-as-a-Judge" have become standard due to their high correlation with human assessments. There are benchmarking efforts like MT-Bench++ and MT-Bench-101 which extend the evaluation sets with follow-up questions and a comprehensive three-tier taxonomy, respectively.

Core LLM Capabilities

The paper highlights five primary capabilities pertinent to multi-turn interactions:

1. Multi-Turn Instruction Following: Large-scale instruction datasets implicitly facilitate this capability, although datasets explicitly capturing interaction patterns remain scarce. Enhancements include role-specific adapters and frameworks like IDEAS, which use strategic instruction-generation methods.
2. Context Memory: External memory systems enable efficient tracking and retrieval of dialogue history, while internal memory mechanisms incorporate contextual information directly within the model. Techniques such as recursive summarization and hash-based storage enhance memory capabilities.
3. Planning: Multi-turn planning involves organizing dialogue structure and managing complex interactions with tools and environments. Innovative methods include dialogue planning frameworks that balance long-term goals with immediate user engagement, and agent planning approaches that efficiently handle task decomposition and real-world tool use.
4. Multi-Turn Reasoning: Effective multi-turn reasoning combines strategic planning, commonsense knowledge, and self-correction. Approaches like reflexion facilitate self-correction behavior, while strategic reasoning benchmarks challenge advanced reasoning agents in games. In particular, multi-turn math and code reasoning tasks emphasize continuous improvement through iterative problem-solving and feedback integration.
5. General Conversation: Conversational capabilities have improved significantly through pre-training and instruction fine-tuning, approaching human-like proficiency. New research efforts focus on curating high-quality dialogue datasets to further enhance open-source LLMs.

Multi-Turn Interaction Algorithms

Algorithms proposed include hierarchical reinforcement learning strategies and preference optimization techniques to address single-turn learning limitations and policy-induced covariate shifts. Efforts like ArCHer and MTPO extend reinforcement learning methodologies to multi-turn dialogues, enabling long-term planning across complex conversational tasks.

Conclusion and Future Directions

The paper suggests future research directions focusing on more diverse training data from real user interactions, calibrated LLM-based evaluators, generated test data aligned with user satisfaction, self-evaluation frameworks, feedback integration, and exploring unexplored areas of multi-turn reasoning. These advancements aim to develop LLM-based agents capable of nuanced and contextually aware multi-turn interactions, improving user engagement and satisfaction in real-world settings.