LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

Abstract: Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability to track evolving analytical context over long horizons untested. We introduce LongDS, a benchmark for long-horizon, multi-turn data analysis where agents must maintain, update, restore, and compose evolving analytical states. LongDS comprises 68 tasks constructed from real-world Kaggle notebooks, spanning 2,225 turns across six domains including Geoscience, Business, and Education. Tasks are designed around state-evolution patterns (e.g., counterfactual perturbation, rollback, multi-state composition), with an average dependency span of 11.3 turns. Evaluating five state-of-the-art models, we find that the best model reaches only 48.45% average accuracy, performance drops nearly 47 points from early to late turns, and long-horizon errors account for 52%--69% of failures. Further analysis shows that additional agent steps do not necessarily improve performance, suggesting that the key bottleneck is maintaining a correct analytical state rather than increasing interaction budget. We release LongDS to support research on reliable long-horizon agentic data analysis. Code and data will be released at https://github.com/zjunlp/DataMind.

• The paper introduces LongDS-Bench, a benchmark that exposes critical failures in maintaining persistent analytical state over multi-turn data analysis.
• It systematically evaluates agents across 2,225 turns in six domains, showing up to a 47% accuracy drop as tasks evolve.
• The study highlights the need for architectures with extended memory and explicit state recovery to advance reliable agentic data analysis.

LongDS-Bench: Diagnosing the Bottleneck of Long-Horizon Agentic Data Analysis

Motivation and Problem Formulation

Analytical workflows in real-world data science are inherently multi-turn and iterative, demanding persistent and context-sensitive state management across extended sessions. However, the majority of evaluation benchmarks for data-analysis agents ignore this reality, focusing instead on independent or short-horizon analytical tasks that regularly reset the analytical state at each turn. This omits the central challenge of agentic data science: evolving analytical state management, which entails maintaining, updating, restoring, and composing a trajectory of analytical objects, definitions, filters, and intermediate results across long interaction chains.

LongDS-Bench addresses this gap by introducing a benchmark dedicated to long-horizon, multi-turn data-analysis scenarios. Each task in LongDS unfolds as a persistent session featuring state-evolution patterns such as inheritance, update, counterfactual perturbation, rollback, and multi-state composition. This design isolates the critical limitation of current agentic AI systems—the inability to reliably maintain or reason over a long-range, dynamic analytical state.

Figure 1: Visualization of LongDS's persistent state management scenario, highlighting the necessity for agents to recover and select the appropriate analytical state across multiple turns.

Benchmark Design and Construction

LongDS is constructed from 68 real-world Kaggle notebooks, yielding a total of 2,225 user turns spanning six domains—Geoscience, Business, Education, Sports, Community, and Social Good. The benchmark curation pipeline consists of three stages: (1) source collection and filtering from high-quality notebooks with sufficient analytical depth, (2) manual and Codex-augmented construction and annotation of multi-turn, state-annotated tasks, and (3) rigorous human and model-based validation to guarantee dependency validity and answer reliability.

Five primary state-evolution patterns structure the benchmark, each requiring different forms of state-dependent reasoning:

• Initial: Establishing new analytical objects or baselines.
• Inheritance: Reusing the most recent valid state as default context.
• Update: Overwriting previous state to create a new default for subsequent analysis.
• Counterfactual: Introducing temporary local variations for a single turn.
• Rollback: Reverting to a prior state for comparative or diagnostic purposes.
• Composition: Combining multiple explicit state operations within and across turns.

Tasks have an average dependency span of 11.3 turns and a breadth of 2.9 direct dependencies per turn, ensuring that agents must handle complex and temporally remote state interactions.

Figure 2: Detailed example task from the Netflix market-opportunity domain, illustrating a 36-turn trajectory with multiple state-evolution operations.

The benchmark construction pipeline employs expert review and Codex-based skill transfer to automate large-scale high-quality task creation, as shown in the task curation diagram.

Figure 3: Pipeline for LongDS curation, encompassing filtering, state annotation, and multi-stage human/model validation.

Domain and benchmark coverage are visualized in task distribution plots.

Figure 4: Domain- and dataset-level distribution of tasks within LongDS.

Experimental Setup and Results

Evaluation is conducted using a ReAct-style agent architecture that supports iterative, reasoning-augmented code execution in persistent Jupyter Notebook environments. The judged output leverages DeepSeek-V4-Pro as an unrestricted LLM-as-judge, with further human audits confirming 93%+ agreement and Cohen's $\kappa$ above 0.86, yielding trustworthy automated scoring.

Five state-of-the-art models (GPT-5.4, Gemini-3.1-Pro, Claude-4.6-Sonnet, Kimi-K2.6, DeepSeek-V4-Pro) are systematically evaluated. The leading model, Gemini-3.1-Pro, achieves only 48.45% average accuracy despite significant resource allocation. GPT-5.4 and Claude-4.6-Sonnet follow closely, but none surpasses the 50% threshold. Performance varies substantially across domains, with generally better accuracy on Education and severe degradation in domains like Geoscience and Sports.

Long-horizon performance is profoundly limited:

• There is a nearly 47-point accuracy drop from early to late stages of each task.
• Accuracy declines sharply as dependency breadth and temporal span increase, and as state-evolution complexity rises.

  Figure 5: Model accuracy degrades along task progress, with growing dependency breadth, and as state-evolution patterns become more demanding.

Error Diagnosis and Bottleneck Analysis

In-depth error analysis, enabled by a robust annotation pipeline and validated with human audits, reveals that long-horizon analytical state failures dominate. Specifically:

• 52–69% of all errors are attributed to issues such as cascade (propagating upstream state errors), direct state-management failures (selecting or updating the wrong state), or context memory failures (forgetting global conventions or definitions).
• Cascade errors are the largest single contributor due to the irrecoverability of upstream mistakes.
• Increasing the allowed agent steps per turn or interaction tokens does not mitigate this degradation, indicating the core problem is not insufficient reasoning redundancy but the inability to maintain or recover the correct analytical state across an extended trajectory.

  Figure 6: Detailed breakdown of efficiency-performance trade-offs, domain-level efficiency, error type distributions (emphasizing long-horizon errors), and effects of persistent state resets.

Further analysis demonstrates that agent exploration (measured by the number of steps per turn) declines over time, reducing opportunities for state correction or recovery. Experiments involving resetting the code environment partway through a trajectory exhibit a trade-off: reset is beneficial when persistent state is already corrupted but harmful when the accumulated state is accurate, exposing the critical need for robust, state-aware recovery strategies.

Figure 7: Benchmark statistics including turn distribution, types of state-evolution transitions, and connection complexity.

Figure 8: Model accuracy diminishes as the dependency span to previous required turns lengthens, highlighting the challenge of tracking temporally distant dependencies.

Implications, Limitations, and Future Directions

LongDS-Bench exposes a fundamental bottleneck for agentic data analysis: reliable analytical state management remains unsolved in current LLM-based agents, even with substantial LLM advances. The results suggest that future progress will require:

• Architectures with extended, verifiable memory that combine symbolic state tracking with language-based reasoning components.
• Explicit state modeling, isolation, and rollback mechanisms to avoid irrecoverable cascade errors.
• Process-level supervision and reward modeling to penalize state drift and drifting context over long horizons (Qiu et al., 27 Apr 2026).

Practically, this limits the deployment of LLM agents as reliable long-term collaborators in scientific or production analytics. Theoretically, LongDS frames analytical state as a core challenge separating agentic reasoning from short-form code generation or conversational QA.

LongDS itself is subject to constraints in task diversity and may under-represent open-ended exploratory or visualization-heavy data analysis, focusing on quantitatively verifiable tasks. Addressing state-management in unconstrained, production-grade analytics remains an open objective.

Conclusion

LongDS-Bench proposes a realistic, diagnostic evaluation paradigm for the core challenge of long-horizon agentic data analysis: persistent analytical state management. Benchmarking across multiple domains and models, the study clearly demonstrates that current agentic AI systems are fundamentally limited by their inability to reliably track, update, and recover dynamic analytical states over extended workflows. LongDS thus defines both a rigorous challenge and a critical research agenda for the next generation of AI-powered analytical agents.

Citation: "LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis" (2605.30434)