LLMs Get Lost in Evolving User Intent

Abstract: As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.

• The paper presents a novel framework to evaluate LLMs on evolving user intent using synthesized multi-turn conversations.
• The methodology highlights significant accuracy drops, with models like GPT 5.5 degrading by 18.7% across multiple intent transitions.
• The study underscores that simple memory mechanisms offer partial improvements, calling for architectures that better update and track dynamic user intent.

Robustness of LLMs to Evolving User Intent: Evaluation and Insights

Introduction

The paper "LLMs Get Lost in Evolving User Intent" (2607.20734) introduces a rigorous framework for evaluating LLMs not only in static, single-turn scenarios but, critically, in settings where user intent evolves through multi-turn interaction. The authors highlight a fundamental capability gap: while state-of-the-art LLMs achieve near-ceiling performance on fully specified, single-turn tasks, their robustness deteriorates when required to track, adapt to, and act upon user intent that is incrementally revealed, revised, or redirected.

Problem Formulation and Framework

The work formalizes user intent as a structured state comprising a target function (e.g., a task or API call), a set of arguments with corresponding values, and a record of which arguments are revealed to the agent. The evolution of intent is modeled via three transition types:

• Argument reveal: Gradually disclosing missing arguments;
• Argument revision: Changing previously stated values;
• Function switch: Pivoting to related but distinct tasks while retaining shared context.

To circumvent the infeasibility of manually authoring large-scale, verifiable multi-turn benchmarks, the authors propose a backward-synthesis approach: each multi-turn conversation is constructed to terminate at an "anchor" single-turn task, enabling evaluation via extant, automatic verifiers. Preceding conversational turns are retrospectively fabricated via LLM-powered extraction of counterfactual arguments and predecessor functions, producing feasible multi-turn trajectories with controlled intent transitions.

An illustration of this construction is provided in (Figure 1).

Figure 1: The framework extracts the source intent from single-turn data as an "anchor" and synthesizes a plausible multi-turn history culminating in the anchor turn; final evaluation leverages the existing dataset verifier.

Experimental Evaluation

The empirical study encompasses four tasks: GSM8K (math), BIRD-SQL (text-to-SQL), BrowseComp+ (deep research), and SWE-Bench Verified (software engineering). Each single-turn dataset is programmatically lifted into a multi-turn, intent-evolving scenario, enabling direct, apples-to-apples comparison of LLM performance in both settings.

Key findings:

• Substantial accuracy degradation is observed across all model families when transitioning from single-turn to evolving-intent settings. For example, GPT 5.5 accuracy on GSM8K drops by 18.7% (99.0% → 80.5%) after just six intent transitions. For tool-augmented domains (e.g., SWE-Bench), certain LLMs fail catastrophically.
• The performance drop is monotonic in the number and complexity of intent transitions. Argument reveals degrade accuracy less than revisions and function switches, with compound transitions generating more severe deterioration (Figure 2).

  Figure 2: Accuracy declines monotonically as the number of intent transitions increases, with function switches inducing steeper drops than reveals or revisions.

• The most pronounced degradations occur when conversations interleave different transition types, especially function switches. Ablation studies confirm that multi-type transitions represent a more significant challenge than repeated reveals or revisions alone.
• Context-discarding after function switches hampers performance: when additional transitions follow a function switch, model accuracy degrades further (Figure 3).

  Figure 3: Performance is higher when evaluated immediately after a function switch, but drops when further intent transitions intervene, indicating insufficient integration across conversational boundaries.

• Simple memory mechanisms—prompt or oracle recaps—improve but do not eliminate the performance gap relative to the single-turn upper bound (Figure 4).

  Figure 4: Prompt recap and oracle recap interventions partially recover accuracy under evolving intent but do not reach single-turn baselines.

• Multi-turn interaction accentuates difficulty, especially for hard source tasks or when models face longer conversational horizons with sustained intent evolution (Figure 5).

  Figure 5: The adverse effect of task difficulty is magnified in the evolving-intent setting compared to single-turn evaluation.

Additional Analytical Dimensions

Per-Turn Computational Overhead and Model Scaling

Analysis of open models indicates that average response length increases per turn under evolving intent, reflecting the need to process and reconcile a richer, dynamically shifting conversational state (Figure 6, left).

Simultaneously, model size correlates with robustness: within a model family, smaller models experience larger performance drops in evolving-intent settings despite parity in single-turn accuracy (Figure 6, right).

Figure 6: Left—average response length increases over conversation turns, indicating growth in context assimilation requirements. Right—smaller models are less robust to evolving-intent than larger counterparts.

Isolating Bottlenecks

The paper's analysis attributes degradation both to failures in accurately tracking current user intent and to the challenge of acting faithfully on tracked intent amid distractors. Notably, explicit oracle recaps (where the full contemporary user intent is restated to the model) close performance gaps for reveal and revision-type transitions but not for function switches or compound transitions.

Control Analyses

The authors show that increases in conversation length alone, absent substantive changes in user intent, do not affect agent accuracy. The critical factor is the dynamism of user intent, not multi-turn context size per se.

Implications and Future Directions

The results demonstrate that optimizing LLMs for single-turn benchmarks is insufficient for agentic, real-world dialogue, where user intent is inherently mutable. Current models lack mechanisms for maintaining a coherent, up-to-date belief state as user goals shift, particularly for function switches or when context must be selectively overruled or revised.

Practically, this impacts the reliability of LLM-based agents intended for interactive, collaborative usage scenarios—coding assistants, research copilots, or multi-step tool users. The inability to robustly incorporate evolving user requirements impedes deployment in any environment where user needs are emergent rather than pre-specified.

Theoretically, this exposes a structural shortcoming. Pre-trained transformer models, even at trillion-parameter scale, remain brittle to distributional shifts in conversational dynamics. Supervised fine-tuning and reinforcement learning from human feedback, when coupled with static benchmarks, do not imbue LLMs with the flexibility or memory hygiene necessary for long-horizon intent tracking.

The findings motivate several research directions:

• Architectures that maintain and update explicit user intent representations across turns;
• Training curricula and data augmentation using synthesized evolving-intent conversations, as demonstrated in initial RL experiments;
• Integration of planning and memory modules that can reason over and revise the agent's latent state as dialogue progresses;
• Benchmarks that more faithfully capture not just incremental information disclosure but also revision and task pivoting, to drive robust model development.

Conclusion

This work systematically demonstrates the limitations of current LLMs when deployed as agents tracking evolving user intent. The presented framework establishes an extensible testbed for future research on intent robustness, enabling scalable, verifiable multi-turn evaluation without bespoke human annotation. Until models can robustly accommodate dynamic user goals, real-world deployment in interactive applications will remain error-prone, especially as task and interaction complexity grow. Addressing these gaps will require innovations in both model architecture and data, informed by the comprehensive, controlled analyses enabled by this framework.