TEC: A Collection of Human Trial-and-error Trajectories for Problem Solving

Abstract: Trial-and-error is a fundamental strategy for humans to solve complex problems and a necessary capability for AI systems operating in real-world environments. Although several trial-and-error AI techniques have recently been proposed, most of them rely on simple heuristics designed by researchers and achieve limited performance gains. The core issue is the absence of appropriate data: current models cannot learn from detailed records of how humans actually conduct trial-and-error in practice. To address this gap, we introduce a data annotation platform and a corresponding dataset, termed Trial-and-Error Collection (TEC). The platform records users' complete trajectories across multiple trials and collects their reflections after receiving error feedback. Using this platform, we record the problem-solving processes of 46 participants on 58 tasks, resulting in 5,370 trial trajectories along with error reflections across 41,229 webpages. With this dataset, we observe that humans achieve substantially higher accuracy compared to LLMs, which demonstrates that humans are more effective in trial-and-error than LLMs. We believe that the TEC platform and dataset provide a valuable foundation for understanding human trial-and-error behavior and for developing more capable AI systems. Platform and dataset are publicly available.

• The paper introduces TEC, a comprehensive dataset capturing 5,370 human trial-and-error trajectories with detailed error reflections and behavioral metrics.
• It employs a Chrome extension to log multi-trial interactions, enabling structured annotations that reveal adaptive diagnostic reasoning in problem solving.
• Comparative analysis shows humans outperform LLM baselines in error recovery and strategy shifts, highlighting gaps in current AI methodologies.

TEC: A Collection of Human Trial-and-error Trajectories for Problem Solving

Motivation and Context

The TEC platform and dataset address a persistent deficiency in AI and IR research: the lack of high-quality, structured data capturing human trial-and-error problem-solving behavior. Existing benchmarks, such as Mind2Web [deng2024mind2web], WebArena [zhou2024webarena], and session logs (TREC Session Track [carterette2016trec], TianGong-ST [10.1145/3357384.3358158]), either support only single trials without error reflection, or provide unsupervised, multi-query logs that lack per-trial correctness and diagnostic annotation. Consequently, research on trial-and-error capabilities in LLMs is restricted to simple heuristics and self-generated feedback, rather than learning from authentic human data [shinn2023reflexion, madaan2023selfrefine, gou2024criticlargelanguagemodels, ozer2025marmultiagentreflexionimprovesreasoning].

TEC bridges this gap with a platform capable of capturing complex, iterative problem-solving trajectories—including multi-trial web interaction data, per-trial correctness labels, evidence markers, and structured reflections on error. The resulting dataset enables rigorous comparative analyses between human and LLM strategies on information-seeking tasks and sets the stage for training and evaluating next-generation agent architectures that learn trial-and-error dynamics from human exemplars.

Platform Design and Data Schema

TEC is architected around two essential dimensions: iterative trial capture and structured error reflection. Its Chrome extension logs comprehensive behavioral traces—including DOM snapshots, user interactions, reading patterns, evidence marking, and webpage metadata—across any website, without per-site instrumentation. The backend orchestrates task assignment, data collection, annotation, evaluation, and privacy compliance. A replay-based annotation workflow is integrated, allowing participants not only to reflect immediately after each error with prioritized diagnostic tagging and corrective planning, but also to revisit and annotate their browsing history.

The dataset comprises 5,370 trials from 46 participants across 58 challenging open-domain QA tasks where baseline LLMs (and WebGPT) fail. The schema includes pre-task assessments, trial records with behavioral traces and correctness labels, structured reflections (error category, prioritized corrective plan), post-task success assessments, and cancellation annotations. Evidence markers and credibility ratings for submissions are present in every trial. All records are anonymized and released under MIT licensing.

Behavioral and Error Reflection Analysis

TEC data reveal that human trial-and-error problem-solving exhibits distinctive error diagnosis and recovery mechanisms. Of all task trajectories, 89% ultimately succeed, with a mean of 2.2 trials per task. Key behavioral metrics (pages per trial, dwell time, queries per session) exhibit log-normal and geometric distributions typical of human exploratory search.

Analysis of reflection annotations demonstrates that human corrective plans are tightly conditioned on the preceding error diagnosis, confirming diagnostic reasoning rather than heuristic repetition. For instance, 57% of “Format Error” diagnoses prompt a “Correct Format” plan, while “Unreliable Source” leads to “Deeper Processing” in 44% of instances. Query reformulation exhibits semantic drift: humans increasingly diverge from their original query in subsequent trials, both lexically and semantically, reflecting adaptive strategy shifts informed by prior failures.

Human vs. LLM Comparative Analysis

On the 58-task benchmark, TEC facilitates controlled evaluation of human performance versus four LLM baselines: Vanilla LLM, Retrieval-Augmented Generation (RAG), Vanilla Agent (ReAct), and Browser Agent, all executed with GPT-4o-mini and Qwen3-8B models. Key findings include:

• First-trial accuracy: Vanilla Agent with GPT-4o-mini achieves SR@1 of 58.6%, matching human SR@1 (56.6%).
• Recovery from error: Humans’ recovery rate after a failed first trial is 74.5%, substantially higher than all LLM baselines (≤ 50% for Vanilla Agent, dropping to ≈13% for Qwen3-8B Vanilla LLM).
• Average number of trials: Humans resolve tasks in fewer trials than most LLMs.
• Reflection and reformulation: Humans employ cross-trial semantic reformulations; LLMs primarily exhibit surface-level lexical changes, rarely shifting strategy in response to failure. Notably, access to richer toolsets (as in Browser Agent) does not translate to superior recovery: this agent fixates on parametric knowledge and URL navigation, underutilizing search and external evidence.
• Case study (e.g., "Who sang Smoke Gets in Your Eyes first?"): The human participant correctly diagnoses a misinterpretation and revises strategy for a successful second attempt, while LLM baselines oscillate between incorrect guesses or near-misses across all trials, unable to leverage retrieved snippets or error signals to close the gap.

These findings empirically substantiate a critical capability gap: LLMs, even when equipped with sophisticated reflection scaffolding, fail to mimic the diagnostic, adaptive search strategies of humans in iterative, error-driven problem solving.

Implications and Future Research Directions

Practically, TEC provides a rigorous foundation for developing and benchmarking agents with authentic trial-and-error learning behaviors. It lays the groundwork for:

• Supervised learning from human reflections: Training agent architectures to emulate error diagnosis and corrective planning at the level of granularity observed in TEC trajectories, directly addressing deficiencies in current heuristic- or self-reflection-based approaches [zhan2025evaluatingintelligencetrialerror, ma-etal-2025-s2r, yuan2025agentrtraininglanguagemodel].
• Multi-stage agent RL: Utilizing structured feedback signals for fine-grained reinforcement learning during multi-trial reasoning and web navigation [cheng2025agentr1trainingpowerfulllm, Guo_2025, kimiteam2025kimik15scalingreinforcement].
• User simulation and interface design: Building realistic simulators of search and error recovery behavior for interactive search systems, retrieval models, and web agents.

Theoretically, the dataset offers a granular lens for studying cognitive mechanisms underlying human problem solving in IR, including strategy drift, evidence aggregation, and real-time self-correction.

The platform and methodology can be extended to broader domains (e.g., decision-making, open-ended exploration) and scaled to larger cohorts. Future work involves systematic augmentation of the dataset, advanced agent training via reflection imitation, and controlled longitudinal studies to benchmark agent learning curves as they acquire human-like trial-and-error strategies.

Conclusion

TEC fills a critical gap in AI and IR research by providing a comprehensive platform and dataset capturing multi-trial, error-driven human problem solving for information-seeking tasks. Robust behavioral and annotation data demonstrate diagnostic reflection and adaptive query reformulation in humans, significantly outperforming LLM baselines in error recovery and strategy shift. These results highlight a fundamental capability gap and suggest directions for agent training and evaluation rooted in authentic human data. TEC thus constitutes a valuable resource for advancing research on trial-and-error learning and adaptive intelligent agents (2604.06734).