LongDS Benchmark: Long-Horizon Data Analysis

• LongDS is a benchmark for long-horizon, multi-turn data analysis that evaluates persistent analytical state management across evolving tasks.
• It measures an agent's ability to construct, update, restore, and compose analytical states over 68 tasks with 2,225 turns based on real Kaggle workflows.
• Empirical results show a significant performance drop (nearly 47 percentage points) over long sessions, highlighting challenges in maintaining state fidelity.

LongDS, introduced as LongDS-Bench, is a benchmark for long-horizon, multi-turn data analysis in which an agent must maintain, update, restore, and compose an evolving analytical state across a persistent executable environment rather than solving isolated prompts. It was designed to evaluate a capability that prior data-analysis benchmarks largely miss: whether an LLM agent can behave like a real analyst over time by reusing prior definitions, revising filters and baselines, handling temporary perturbations, rolling back to earlier states, and combining multiple state versions as a session unfolds. The benchmark contains 68 tasks and 2,225 total turns constructed from real-world Kaggle notebooks, and the paper reports that even the best evaluated model reaches only 48.45 average score, with a nearly 47 percentage-point drop from early to late task progress (Xu et al., 28 May 2026).

1. Conceptual Scope and Motivation

LongDS was introduced to address two limitations in earlier benchmark design. First, many benchmarks use static, resettable tasks, so they do not test whether agents can preserve and reuse analytical state across turns. Second, even newer interactive benchmarks often rely on short or guided interactions in which the current turn already specifies the needed operation, thereby weakening the test of long-range dependency management. LongDS instead makes long-horizon data analysis explicitly about constructing, inheriting, updating, perturbing, restoring, and composing states over extended trajectories (Xu et al., 28 May 2026).

The benchmark is motivated by the observation that real-world data analysis is inherently iterative rather than one-shot. Analysts commonly define cohorts, metrics, filters, rules, and intermediate results; reuse them later without restating them; revise them when assumptions change; temporarily branch into counterfactual variants; and sometimes return to an earlier version of the analysis. LongDS is therefore organized around state evolution, not just task completion. A central implication is that success depends on selecting the correct analytical state among multiple versions and manipulations of that state, not merely on recalling prior text.

This framing also clarifies a common misconception. LongDS is not only a benchmark for long-context memory. The paper explicitly argues that the harder problem is whether the agent can keep the analysis correct as the session evolves, especially when dependencies stretch across many turns and when intermediate state corruption can propagate forward (Xu et al., 28 May 2026).

2. Formal Task Model and Persistent Environment

LongDS formalizes a multi-turn task as

$\mathcal{T} = (\mathcal{D}, E_0, U),$

where $\mathcal{D}$ is the set of data files, $E_0$ the initial executable environment, and $U=(u_1,\ldots,u_T)$ the sequence of user requests. At turn $t$, the agent receives the current request $u_t$, the interaction history $H_{<t}$, and the current environment state $E_{t-1}$, then produces a response $y_t$ and an updated environment state $E_t$ (Xu et al., 28 May 2026).

The distinguishing feature is that the environment is persistent rather than reset each turn. Previous code, variables, and intermediate results remain available across the trajectory. This makes the benchmark sensitive to errors in both semantic interpretation and environment evolution. An answer can be locally plausible while still being globally inconsistent with the correct analytical state. The benchmark therefore evaluates not only whether an operation can be performed, but whether it is performed under the right inherited, revised, or restored state.

This persistent-environment formulation also changes what counts as competence. In isolated tasks, each answer can be judged independently. In LongDS, a turn often depends on analytical decisions made many turns earlier. The benchmark reports an average dependency breadth of 2.85 dependencies per turn and an average dependency span of 11.29 turns, rounded in the abstract to 11.3 turns, indicating that the long-horizon structure is a design property rather than a side effect of task length (Xu et al., 28 May 2026).

3. Construction Pipeline and Benchmark Composition

LongDS was constructed from real Kaggle workflows through a three-stage pipeline. The source collection stage starts from 64 competitions and datasets, selects 4 notebooks per source, and yields 256 high-quality raw notebooks. After manual execution, repair, and filtering, the retained pool consists of 36 competitions/datasets and 77 executable filtered notebooks. The task construction stage begins with 3 seed tasks and uses Codex-assisted skill creation and conversion to transform notebooks into multi-turn tasks. The refinement stage applies expert review for dependency validity, difficulty, and answer reliability, followed by annotation-guided Codex validation and final consistency checks, producing 68 final tasks (Xu et al., 28 May 2026).

The resulting benchmark spans six domains and about 33 turns per task on average. The paper reports the following domain counts.

Domain	Tasks
Geoscience	19
Community	16
Business	12
Social Good	10
Education	8
Sports	3

Beyond raw size, the benchmark is structured around frequent state transitions. The paper reports the following average counts per task: Initial: 19.2, Update: 8.4, Counterfactual: 6.6, Rollback: 5.8, and Multi-state composition: 8.6. These figures show that LongDS is not merely a long dialogue corpus. It is an evaluation set in which repeated state manipulation is pervasive across the trajectory (Xu et al., 28 May 2026).

A further defining property is real-world provenance. Tasks are built from actual Kaggle notebooks and datasets rather than synthetic isolated prompts. Each turn is rewritten into a form with executable reference code and a structured reference answer, enabling automatic scoring while preserving the workflow character of applied data analysis.

4. State-Evolution Patterns

The conceptual center of LongDS is a taxonomy of six state-evolution patterns. These patterns specify how the analytical state changes, or does not change, across turns (Xu et al., 28 May 2026).

• Initial: establishes a reusable analytical object, such as a cohort, metric, rule, or intermediate result.
• Inheritance: reuses the most recent valid analytical state without restating it; the paper treats this as the default persistence mechanism rather than a separately counted category in the statistics.
• Update: revises a prior definition, formula, filter, aggregation rule, or baseline, and makes the revision the new default state.
• Counterfactual: introduces a temporary alternative assumption for the current turn only.
• Rollback: answers under an earlier anchored version of the analysis instead of the most recent state.
• Composition: combines two or more explicit state operations beyond default inheritance.

These categories distinguish LongDS from benchmarks that focus only on sequential instruction following. A counterfactual request, for example, should not overwrite the prevailing default state. A rollback request requires selecting an earlier anchored state rather than the most recent one. A composition request requires combining multiple state references rather than simply inheriting the latest context. This suggests that the benchmark operationalizes analytical competence as state selection and transformation under persistence, rather than as linear conversational continuity alone.

The performance progression reported in the paper reinforces this interpretation: accuracy declines from Initial to Update to Counterfactual to Rollback. Agents are therefore relatively better at constructing state than at revising, temporarily perturbing, or restoring it (Xu et al., 28 May 2026).

5. Evaluation Protocol and Empirical Results

Each LongDS turn has executable reference code and a structured reference answer. The judge model is DeepSeek-V4-Pro, which determines whether the agent’s answer is semantically and numerically consistent with the reference. The per-turn score is defined as

$\mathcal{D}$0

and the benchmark score is a macro-average of task-level accuracies:

$\mathcal{D}$1

The evaluation protocol is validated by a blind human audit reporting 93.11% agreement and Cohen’s $\mathcal{D}$2, which the paper presents as evidence that the LLM-as-judge procedure is reliable (Xu et al., 28 May 2026).

The benchmark evaluates five models: GPT-5.4, Claude-4.6-Sonnet, Gemini-3.1-Pro, DeepSeek-V4-Pro, and Kimi-K2.6. Their reported full-benchmark scores are as follows.

Model	Average score
Gemini-3.1-Pro	48.45
GPT-5.4	43.50
Claude-4.6-Sonnet	41.56
Kimi-K2.6	39.72
DeepSeek-V4-Pro	31.97

The headline result is that even the best model remains below 50% average accuracy. Performance also varies substantially by domain: models tend to do better in Education, struggle more in Geoscience, Business, and Sports, and no model is consistently strong across all domains. The appendix further reports a Codex evaluation on a sampled subset in which Codex reaches 65.55 average accuracy, but the paper explicitly treats this as a subset result rather than a replacement for the full-benchmark findings (Xu et al., 28 May 2026).

A central empirical trend is long-horizon degradation. The paper reports a nearly 47 percentage-point drop from early to late task progress. Accuracy falls as dependency breadth increases and also falls as dependency span increases. These results support the benchmark’s core claim that long-horizon analytical-state management remains a major failure mode for current agents.

6. Error Taxonomy, Interaction Budget, and Broader Significance

LongDS distinguishes ordinary mistakes from errors that are specific to long-horizon analysis. The paper identifies three main failure types: Context Memory Error, State Management Error, and Cascade Error. Long-horizon errors account for 52% of GPT-5.4’s failures and up to 69% for Kimi-K2.6, with Cascade Error reported as the largest category. State Management Error is also substantial, indicating that agents often choose the wrong state version or fail to restore the right one (Xu et al., 28 May 2026).

This error profile motivates one of the paper’s main conclusions: the key bottleneck is not merely remembering prior information, but maintaining the correct analytical state over time. Context memory issues exist, yet they are not the main bottleneck. A plausible implication is that improvements in retrieval or larger context windows alone are unlikely to solve the benchmark if the agent still cannot preserve state fidelity under update, rollback, and composition.

The paper also reports that increasing agent steps does not necessarily improve performance. Claude-4.6-Sonnet uses the most steps, GPT-5.4 uses fewer steps, and more steps do not translate into the best accuracy; GPT-5.4 is described as more cost-efficient, while Gemini-3.1-Pro is the most accurate overall. In addition, the average number of agent steps per turn decreases over the course of a trajectory by about 4.3 steps from early to late stages. The reset experiment reveals a trade-off: resetting the environment can help when persistent state has degraded, but it can hurt when persistent state is strong and useful. The paper interprets this as evidence that longer interaction budgets may create more chances for state drift, so additional interaction is not inherently beneficial (Xu et al., 28 May 2026).

In broader terms, LongDS shifts benchmark design from single-query correctness toward persistent analytical behavior. It asks whether a model can remember what has been defined, know when to update it, understand when a deviation is temporary, and restore earlier versions correctly. The results indicate that long-horizon agentic data analysis is not mainly limited by raw reasoning or coding ability alone. The central challenge exposed by LongDS is persistent analytical state management, making the benchmark a targeted instrument for research on more reliable data-analysis agents (Xu et al., 28 May 2026).