ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Abstract: Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 LLMs. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.

• The paper introduces a novel paradigm capturing user thoughts alongside dialog turns, exposing latent motivations for more effective AI alignment.
• It employs a robust dataset of 2,155 conversations with deep annotations, demonstrating improved next-message prediction and fine-tuning benefits.
• The study highlights that explicit thought annotations reveal semantic nuances that advanced LLMs struggle to infer, underscoring the value of cognitive data.

Authoritative Summary of "ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions" (2605.20087)

Introduction and Motivation

"ThoughtTrace" introduces a novel paradigm for human-AI interaction research by systematically capturing user thoughts—reasoning and reactions underlying each conversational turn—alongside multi-turn, real-world LLM usage. The central premise is that traditional conversation datasets only record overt linguistic exchanges, neglecting latent cognitive content (task motivations, constraints, expectations, evaluative reactions) which drives and frames dialogic behavior. By bridging this gap, ThoughtTrace enables rigorous modeling of user intent, satisfaction, and adaptive assistant alignment.

Figure 1: Representative ThoughtTrace example. User-AI dialog (top), with latent user thoughts—reasons for prompts and reactions to responses—collected at each turn (bottom).

Dataset Construction and Methodology

ThoughtTrace comprises 1,058 participants yielding 2,155 conversations (17,058 turns, 10,174 thought annotations) against 20 diverse LLMs, with demographic metadata and self-described tasks/objectives. The collection protocol features guided tutorials, enforced annotation of both user reasons and reactions per turn, and detailed post-task surveys to capture expectations and actual task completions. Users engage with LLMs blindly (model identity masked) on open-ended tasks, annotating each dialog instance with rich in-situ cognitive context.

Figure 2: Demographic and AI usage breakdown of participants—substantial coverage across age, education, occupation, and high AI utilization rates.

Conversation Properties

• Diversity and Depth: ThoughtTrace demonstrates significantly longer conversations (median 8 turns; token counts distributed 2k–5k+, compared to baselines heavily concentrated under 1k tokens) and broad topical coverage across seven domains and 36 subtopics (travel, learning, business, health, technology, etc.).

Figure 3: Turn distribution—ThoughtTrace peaks at 6–8 turns versus WildChat/LMSYS-Chat-1M's short exchanges.

• Task Extension Dominance: Multi-turn analysis reveals that 57% of user turns extend or elaborate previous tasks, as opposed to new requests or retries, underscoring the importance of modeling progressive intent evolution.

  Figure 4: Multi-turn relationship flow—extension/building on prior tasks dominates, increasingly so with dialog progression.

Thought Properties

1. Semantic Distinctiveness: UMAP analyses and semantic coverage metrics confirm that thought annotations are not simply verbatim paraphrases of utterances. Embedding distances between messages and annotated thoughts are substantially greater than consecutive message pairs; LLM-based coverage scores for message-to-thought are 3.22 (reasons) and 2.00 (reactions) out of 5, indicating only partial overlap.

   Figure 5: UMAP projections—semantic shifts between messages, reasons, and reactions are more pronounced than message-to-message.

2. LLM Inference Difficulty: Prompts to frontier models (GPT-5.4, Gemini-3.1 Pro Preview, Claude Opus-4.6) to infer user thoughts from dialog context yield low similarity scores (≤3/5), with systematic failures in capturing unspoken motivations or nuanced reactions, demonstrating the necessity of explicit thought annotation for high-fidelity modeling.
3. Taxonomic Diversity: ThoughtTrace spans seven reason categories (task motivation, continuation, reorientation, expectation, context, style, social) and five reaction categories (explicit affirmation, partial satisfaction, content relevance, presentation style, scope fit). Task motivation and continuation dominate reasons, while explicit affirmation and content relevance anchor reactions.

   Figure 6: Reason category distribution—task motivation (36.9%) and continuation (21.4%) are primary.

   

   Figure 7: Reaction category distribution—explicit affirmation dominates (72.2%), dissatisfaction most often from content relevance (11.9%).

4. Dynamical Dependencies: Thought types shift with conversation stage (initial turns: motivation; later: continuation/context), but remain relatively topic- and length-invariant, suggesting that underlying cognitive structure reflects dialogic progression more than subject matter.

Figure 8: Stage-wise distribution of reason types—motivation wanes as continuation/context reasons increase.

Utility of Thoughts for Modeling and Alignment

Next-Message Prediction

Experimental evaluation of LLMs predicting user behavior demonstrates a substantial performance boost when thought annotations are provided as context: semantic similarity improves from 21.6 to 30.6 (+41.7% relative gain) compared to history-only baseline, with maximal gains for models trained with richer thought context. This establishes that latent thought signals operationally enhance user simulation/modeling.

Figure 9: Utility experiments—thoughts improve (a) user behavior prediction (next-message) and (b) model alignment via supervision.

Alignment via Thought-Guided Supervision

Thought-guided rewrites for dissatisfied responses (using explicit reaction annotation as supervision) outperform message-guided rewrites by +4.5% win rate on Arena-Hard, and surpass the base model by +25.6%, indicating that thought annotations provide denser, more actionable feedback signals for RLHF/DPO fine-tuning. ThoughtTrace surfaces 2.2× more dissatisfaction instances than message-alone datasets, yielding higher-quality and more scalable alignment supervision.

Implications and Future Directions

The introduction of thought annotations as a data modality challenges the prevailing assumption that behaviorally observable dialog is sufficient for intent, satisfaction, and value modeling. ThoughtTrace enables:

• Systematic investigation of cognitive dynamics—how context shapes thoughts, and vice versa.
• Improved user simulation for training and evaluation, supporting models that predict and adapt to latent user goals.
• Advanced alignment benchmarking, replacing coarse proxies (e.g., follow-up message complaints) with ground-truth subjective signals.

Practically, datasets structured like ThoughtTrace can facilitate fine-grained personalization, context-aware assistant adaptation, and more robust evaluation beyond surface-level metrics. Theoretically, they motivate new lines of inquiry in interactive ToM, dialog reasoning, and mental state inference.

Figure 10: Token-level conversation lengths—ThoughtTrace contains more information-dense, extended exchanges compared to major baselines.

Figure 11: Message length stats—assistant responses majorly longer and more variable than user prompts throughout turns.

Figure 12: Thought length distribution—most thoughts are concise (4–20 tokens); longer in initial stages, stabilizing later.

Figure 13: Full topical breakdown—coverage across 36 subtopics, ensuring generalizability and broad utility.

Figure 14: Word clouds—task summaries and expectations reveal planning/problem-solving as primary activities, with structured, actionable output preferred.

Conclusion

ThoughtTrace empirically establishes user thoughts as a distinct and essential component of real-world LLM interaction modeling. Explicit documentation of motivations and reactions reveals latent information that is challenging for even advanced LLMs to infer, materially enhances user simulation and assistant alignment, and creates the foundation for next-generation assistant design targeting latent satisfaction, goal fulfillment, and adaptive dialog flow. Future work will address methodological limitations (reactivity, conscious-only cognition, recruitment bias) and extend thought-centered benchmarks, personalized agent training, and reward models, ultimately shifting conversational AI success measures from surface utterance quality to cognitive and affective fidelity.