LLMs Get Lost In Multi-Turn Conversation
Current large language models struggle dramatically in multi-turn, underspecified conversations compared to single-turn, fully-specified prompts. Across 15 models and six diverse generation tasks, performance drops by nearly 40% when information is revealed gradually across conversation turns. This degradation is driven not by declining capability but by catastrophic unreliability, as models make premature assumptions, generate bloated responses, and neglect information from middle turns. The paper introduces a novel simulation framework based on instruction "sharding" and reveals that standard fixes like temperature adjustment or simple information repetition fall far short of restoring single-turn performance.Script
A language model aces a task when you hand it a complete prompt. But split that same information across five conversation turns, and watch performance collapse by 40 percent. The model gets lost.
To study this, the researchers created a simulation environment that transforms complete instructions into atomic shards. A user simulator reveals one shard per turn, and the system tracks when the assistant attempts an answer and whether it succeeds.
They tested 15 models on six generation tasks, spanning code, database queries, math, and summarization. When all information arrives in a single turn, models average 95 percent of baseline. When sharded across turns, performance craters to 61 percent.
Decomposing the drop into two factors reveals the culprit: aptitude declines only 16 percent, but unreliability more than doubles. Models don't lose the ability to solve the task; they become wildly inconsistent.
Four failure modes emerge from conversation logs. Models attempt answers too early when information is sparse, bloat subsequent attempts with redundant text, ignore middle-turn details, and spiral into verbosity that derails reasoning.
Lowering temperature barely helps; agentic recap strategies fall short. The path forward demands models that are reliable by design across turns, not just capable in single-shot isolation. To learn more and create your own video summaries, visit emergent mind dot com.
