Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild

Abstract: Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about how individual users change their behavior over time. To address this gap, we analyze the conversational trajectories of $\sim$12,000 randomly sampled Microsoft Bing Copilot users and compare these with data from WildChat-4.8M. While the Copilot data contains significant population-level trends, we find that trends in individual user trajectories are much weaker; user habits prove to be overwhelmingly sticky. We also find stark differences between users of different activity levels: more active users have more successful conversations and use the LLM for more complex and professionally oriented tasks. Some user trends also appear in WildChat-4.8M, but we find evidence that this dataset is significantly skewed towards highly proficient "power" users. Ultimately, our results suggest that existing user behavior is difficult to change and demonstrate the extent of user heterogeneity. Our comparison between datasets highlights that WildChat does not represent typical user-AI interactions, an important caveat for downstream uses of the data.

• The paper reveals that aggregate behavioral shifts in LLM usage stem from changing user compositions rather than significant individual adaptation.
• It leverages stratified longitudinal analysis on ~12,000 Bing Copilot and over 1.8M WildChat users to assess trends in activity, linguistic complexity, and task completion.
• Findings highlight the need for explicit interventions to catalyze productive user adaptation and ensure representative benchmarking for model fine-tuning.

Longitudinal User Behavior in LLM Conversational Interfaces: Overview and Analysis

Introduction

"Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild" (2605.29018) presents a comprehensive longitudinal analysis of user interactions with LLM chatbots, specifically Microsoft Bing Copilot and WildChat-4.8M. The paper diverges from prior static analyses by investigating user adaptation, behavioral trajectories, and inter-user heterogeneity over time. It utilizes a stratified sampling methodology covering $\sim$12,000 Bing Copilot users and over 1.8M WildChat-4.8M users, categorizing activity by number of days active. Key behavioral axes include usage intensity, linguistic complexity, task completion, user intent, and conversational domain. Through both temporal and cross-platform comparative analysis, the paper interrogates whether users adapt substantially, how population-level changes manifest, and the consequences for downstream dataset usage and model fine-tuning.

Population-Level Behavioral Trends

Analysis reveals pronounced population-level escalation in user activity and linguistic complexity in Bing Copilot over the six-month study period, with average messages per conversation and syntactic complexity increasing approximately $1.5\times$. In contrast, WildChat exhibits slight decreases or noise in equivalent metrics, reflecting divergent user types and dataset collection protocols.

Figure 1: Population-level user activity and linguistic complexity rise over time in Bing Copilot, absent in WildChat.

Task completion rates also increase at population level in both datasets, though WildChat starts already at a high completion baseline and sees less pronounced gains. Intent analysis indicates a shift in Bing Copilot toward more complex tasks (information gathering, text generation), while simpler intents (information lookup, website navigation) become less prevalent over time. WildChat's trends are partially similar; however, information gathering falls, and text generation surges more dramatically, illustrating dataset-specific intent distributions.

Figure 2: Population-level task completion increases over time in both Bing Copilot and WildChat.

Figure 3: Open-ended intents rise in Bing Copilot, while WildChat shows distinct intent trajectory shifts.

User Heterogeneity as a Function of Activity Level

The analysis stratifies users by activity level (low: 1–10 days, middle: 11–25 days, high: 26+ days), revealing substantial behavioral heterogeneity. In Bing Copilot, more active users consistently demonstrate increased conversation frequency, message complexity, and task completion rates, and pursue more professionally-oriented or cognitively demanding domains. This stratification correlates inherently with successful LLM usage, indicating that power users do not simply "learn" these behaviors but likely exhibit them ab initio.

Figure 4: More active Bing Copilot users are consistently more engaged and produce higher linguistic complexity.

Completion rates stratified by intent show that high-activity Copilot users outperform low-activity users in complex intents such as summarization and analysis, while differences in simple intents are less pronounced. Domain analysis highlights that professional and creative content is more prevalent in high-activity groups, with casual and entertainment domains associated with lower activity.

Figure 5: Task completion rates positively correlate with Bing Copilot user activity; such stratification does not hold in WildChat.

WildChat, even at low activity levels, is dominated by power user behaviors—high prevalence of programming, creative writing, and API-like usage—suggesting dataset bias toward technical or repeat users and divergence from mainstream chatbot usage profiles.

Figure 6: WildChat's intent and domain distributions are more similar to high-activity Copilot users (lower J–S divergence).

Longitudinal Analysis of Individual User Trajectories

Contrary to hypotheses of continuous user adaptation, the paper demonstrates that individual user trajectories are marked by behavioral stickiness; most users change little over time. The magnitude of change over a user's chatbot lifetime (in terms of activity, linguistic complexity, completion, intent, and domain) is substantially less than that observed at the population level. Notably, even high-activity users—those presumed to have the greatest opportunity for learning—show only minor increases, and often early conversational behavior already resembles final profile.

Figure 7: User-level behavioral change over lifetimes is minor compared to population-level shifts.

Intent and domain exploration/exploitation dynamics exhibit only slight movement; Bing Copilot users shift minimally toward exploitation (reducing unique intent/domain pursuit), while WildChat low-activity users slightly reverse trend toward exploration, reinforcing that inter-user heterogeneity and user pool composition underlie observed aggregate changes.

Figure 8: Minimal shift from exploration to exploitation in Copilot; WildChat low-activity users trend inversely.

Dataset Bias, Generalization, and Downstream Implications

The comparison between Bing Copilot and WildChat-4.8M surfaces critical implications for downstream applications. WildChat's heavy power-user skew and substantial API-like content post-September 2024 render it unrepresentative of mainstream LLM user behavior. Artifacts and benchmarks derived from WildChat should not be presumed to generalize, especially for modeling or fine-tuning conversational agents targeting average users.

From a practical standpoint, the stickiness of individual user habits suggests limits in naturalistic user adaptation, and points to the necessity for explicit interventions or features to catalyze more productive LLM exploration. Theoretical implications include the distinction between adoption and adaptation: aggregate learning at the population level arises from shifting user pools (driven by new adopters with distinct profiles), not change within individuals.

Limitations and Future Directions

The study is constrained to January–September 2024 Bing Copilot and pre-September 2024 WildChat-4.8M data. User satisfaction and task sophistication are not exhaustively captured. WildChat user identification is limited to hashed IP mapping, with some conflation. Analysis focuses on group-level patterns; individual evolution may be noisier or multidirectional. Arbitrary dataset truncation may miss post-study behavioral shifts.

Future research should expand to longitudinal multi-platform analysis, incorporate richer behavioral and satisfaction metrics, and develop techniques to encourage productive user adaptation in LLM interfaces. Refining dataset construction protocols to better represent mainstream users is essential for responsible benchmarking and model fine-tuning.

Conclusion

"Adopt $\neq$ Adapt" rigorously demonstrates that population-level shifts in LLM conversational behavior are not reflective of individual adaptation but are instead driven by composition effects and user heterogeneity. High-activity users display elevated complexity, completion, and professional task engagement from their first conversations. These findings underscore the need for targeted interventions to facilitate user learning and the necessity for careful dataset selection when generalizing observed behaviors. The methodology and results invite future innovation in user modeling, benchmarking, and conversational agent design.