---
title: 'LongDS Benchmark: Long-Horizon Data Analysis'
url: https://www.emergentmind.com/topics/longds
type: topic
---

# LongDS Benchmark: Long-Horizon Data Analysis

LongDS, introduced as **LongDS-Bench**, is a benchmark for **long-horizon, multi-turn data analysis** in which an agent must maintain, update, restore, and compose an evolving **analytical state** across a persistent executable environment rather than solving isolated prompts. It was designed to evaluate a capability that prior data-analysis benchmarks largely miss: whether an LLM agent can behave like a real analyst over time by reusing prior definitions, revising filters and baselines, handling temporary perturbations, rolling back to earlier states, and combining multiple state versions as a session unfolds. The benchmark contains **68 tasks** and **2,225 total turns** constructed from real-world Kaggle notebooks, and the paper reports that even the best evaluated model reaches only **48.45** average score, with a nearly **47 percentage-point drop** from early to late task progress [2605.30434].

## 1. Conceptual Scope and Motivation

LongDS was introduced to address two limitations in earlier benchmark design. First, many benchmarks use **static, resettable tasks**, so they do not test whether agents can preserve and reuse analytical state across turns. Second, even newer interactive benchmarks often rely on **short or guided interactions** in which the current turn already specifies the needed operation, thereby weakening the test of long-range dependency management. LongDS instead makes long-horizon data analysis explicitly about **constructing, inheriting, updating, perturbing, restoring, and composing states** over extended trajectories [2605.30434].

The benchmark is motivated by the observation that real-world data analysis is inherently iterative rather than one-shot. Analysts commonly define cohorts, metrics, filters, rules, and intermediate results; reuse them later without restating them; revise them when assumptions change; temporarily branch into counterfactual variants; and sometimes return to an earlier version of the analysis. LongDS is therefore organized around **state evolution, not just task completion**. A central implication is that success depends on selecting the correct analytical state among multiple versions and manipulations of that state, not merely on recalling prior text.

This framing also clarifies a common misconception. LongDS is not only a benchmark for long-context memory. The paper explicitly argues that the harder problem is whether the agent can keep the analysis **correct as the session evolves**, especially when dependencies stretch across many turns and when intermediate state corruption can propagate forward [2605.30434].

## 2. Formal Task Model and Persistent Environment

LongDS formalizes a multi-turn task as
$$
\mathcal{T} = (\mathcal{D}, E_0, U),
$$
where $\mathcal{D}$ is the set of data files, $E_0$ the initial executable environment, and $U=(u_1,\ldots,u_T)$ the sequence of user requests. At turn $t$, the agent receives the current request $u_t$, the interaction history $H_{<t}$, and the current environment state $E_{t-1}$, then produces a response $y_t$ and an updated environment state $E_t$ [2605.30434].

The distinguishing feature is that the environment is **persistent** rather than reset each turn. Previous code, variables, and intermediate results remain available across the trajectory. This makes the benchmark sensitive to errors in both semantic interpretation and environment evolution. An answer can be locally plausible while still being globally inconsistent with the correct analytical state. The benchmark therefore evaluates not only whether an operation can be performed, but whether it is performed under the right inherited, revised, or restored state.

This persistent-environment formulation also changes what counts as competence. In isolated tasks, each answer can be judged independently. In LongDS, a turn often depends on analytical decisions made many turns earlier. The benchmark reports an average dependency breadth of **2.85** dependencies per turn and an average dependency span of **11.29 turns**, rounded in the abstract to **11.3 turns**, indicating that the long-horizon structure is a design property rather than a side effect of task length [2605.30434].

## 3. Construction Pipeline and Benchmark Composition

LongDS was constructed from real Kaggle workflows through a three-stage pipeline. The source collection stage starts from **64 competitions and datasets**, selects **4 notebooks per source**, and yields **256 high-quality raw notebooks**. After manual execution, repair, and filtering, the retained pool consists of **36 competitions/datasets** and **77 executable filtered notebooks**. The task construction stage begins with **3 seed tasks** and uses Codex-assisted skill creation and conversion to transform notebooks into multi-turn tasks. The refinement stage applies expert review for dependency validity, difficulty, and answer reliability, followed by annotation-guided Codex validation and final consistency checks, producing **68 final tasks** [2605.30434].

The resulting benchmark spans six domains and about **33 turns per task** on average. The paper reports the following domain counts.

| Domain | Tasks |
|---|---:|
| Geoscience | 19 |
| Community | 16 |
| Business | 12 |
| Social Good | 10 |
| Education | 8 |
| Sports | 3 |

Beyond raw size, the benchmark is structured around frequent state transitions. The paper reports the following average counts per task: **Initial: 19.2**, **Update: 8.4**, **Counterfactual: 6.6**, **Rollback: 5.8**, and **Multi-state composition: 8.6**. These figures show that LongDS is not merely a long dialogue corpus. It is an evaluation set in which repeated state manipulation is pervasive across the trajectory [2605.30434].

A further defining property is **real-world provenance**. Tasks are built from actual Kaggle notebooks and datasets rather than synthetic isolated prompts. Each turn is rewritten into a form with executable reference code and a structured reference answer, enabling automatic scoring while preserving the workflow character of applied data analysis.

## 4. State-Evolution Patterns

The conceptual center of LongDS is a taxonomy of six state-evolution patterns. These patterns specify how the analytical state changes, or does not change, across turns [2605.30434].

- **Initial**: establishes a reusable analytical object, such as a cohort, metric, rule, or intermediate result.
- **Inheritance**: reuses the most recent valid analytical state without restating it; the paper treats this as the default persistence mechanism rather than a separately counted category in the statistics.
- **Update**: revises a prior definition, formula, filter, aggregation rule, or baseline, and makes the revision the new default state.
- **Counterfactual**: introduces a temporary alternative assumption for the current turn only.
- **Rollback**: answers under an earlier anchored version of the analysis instead of the most recent state.
- **Composition**: combines two or more explicit state operations beyond default inheritance.

These categories distinguish LongDS from benchmarks that focus only on sequential instruction following. A counterfactual request, for example, should not overwrite the prevailing default state. A rollback request requires selecting an earlier anchored state rather than the most recent one. A composition request requires combining multiple state references rather than simply inheriting the latest context. This suggests that the benchmark operationalizes analytical competence as **state selection and transformation** under persistence, rather than as linear conversational continuity alone.

The performance progression reported in the paper reinforces this interpretation: accuracy declines from **Initial** to **Update** to **Counterfactual** to **Rollback**. Agents are therefore relatively better at constructing state than at revising, temporarily perturbing, or restoring it [2605.30434].

## 5. Evaluation Protocol and Empirical Results

Each LongDS turn has executable reference code and a structured reference answer. The judge model is **DeepSeek-V4-Pro**, which determines whether the agent’s answer is semantically and numerically consistent with the reference. The per-turn score is defined as
$$
s_{i,j} =
\begin{cases}
1, & \text{if consistent with the reference} \\
0, & \text{otherwise}
\end{cases}
$$
and the benchmark score is a **macro-average of task-level accuracies**:
$$
S_{\text{avg}} = \frac{1}{M} \sum_{i=1}^{M} \left( \frac{1}{N_i} \sum_{j=1}^{N_i} s_{i,j} \right).
$$
The evaluation protocol is validated by a blind human audit reporting **93.11% agreement** and **Cohen’s $\kappa = 0.8623$**, which the paper presents as evidence that the LLM-as-judge procedure is reliable [2605.30434].

The benchmark evaluates five models: **GPT-5.4**, **Claude-4.6-Sonnet**, **Gemini-3.1-Pro**, **DeepSeek-V4-Pro**, and **Kimi-K2.6**. Their reported full-benchmark scores are as follows.

| Model | Average score |
|---|---:|
| Gemini-3.1-Pro | 48.45 |
| GPT-5.4 | 43.50 |
| Claude-4.6-Sonnet | 41.56 |
| Kimi-K2.6 | 39.72 |
| DeepSeek-V4-Pro | 31.97 |

The headline result is that even the best model remains **below 50% average accuracy**. Performance also varies substantially by domain: models tend to do better in **Education**, struggle more in **Geoscience, Business, and Sports**, and no model is consistently strong across all domains. The appendix further reports a Codex evaluation on a sampled subset in which Codex reaches **65.55** average accuracy, but the paper explicitly treats this as a subset result rather than a replacement for the full-benchmark findings [2605.30434].

A central empirical trend is long-horizon degradation. The paper reports a **nearly 47 percentage-point drop** from early to late task progress. Accuracy falls as **dependency breadth** increases and also falls as **dependency span** increases. These results support the benchmark’s core claim that long-horizon analytical-state management remains a major failure mode for current agents.

## 6. Error Taxonomy, Interaction Budget, and Broader Significance

LongDS distinguishes ordinary mistakes from errors that are specific to long-horizon analysis. The paper identifies three main failure types: **Context Memory Error**, **State Management Error**, and **Cascade Error**. Long-horizon errors account for **52%** of GPT-5.4’s failures and up to **69%** for Kimi-K2.6, with **Cascade Error** reported as the largest category. **State Management Error** is also substantial, indicating that agents often choose the wrong state version or fail to restore the right one [2605.30434].

This error profile motivates one of the paper’s main conclusions: the key bottleneck is not merely remembering prior information, but **maintaining the correct analytical state** over time. Context memory issues exist, yet they are not the main bottleneck. A plausible implication is that improvements in retrieval or larger context windows alone are unlikely to solve the benchmark if the agent still cannot preserve state fidelity under update, rollback, and composition.

The paper also reports that **increasing agent steps does not necessarily improve performance**. **Claude-4.6-Sonnet** uses the **most steps**, **GPT-5.4** uses fewer steps, and more steps do **not** translate into the best accuracy; **GPT-5.4** is described as more cost-efficient, while **Gemini-3.1-Pro** is the most accurate overall. In addition, the average number of agent steps per turn **decreases over the course of a trajectory** by about **4.3 steps** from early to late stages. The reset experiment reveals a trade-off: resetting the environment can help when persistent state has degraded, but it can hurt when persistent state is strong and useful. The paper interprets this as evidence that longer interaction budgets may create more chances for **state drift**, so additional interaction is not inherently beneficial [2605.30434].

In broader terms, LongDS shifts benchmark design from single-query correctness toward persistent analytical behavior. It asks whether a model can remember what has been defined, know when to update it, understand when a deviation is temporary, and restore earlier versions correctly. The results indicate that long-horizon agentic data analysis is not mainly limited by raw reasoning or coding ability alone. The central challenge exposed by LongDS is **persistent analytical state management**, making the benchmark a targeted instrument for research on more reliable data-analysis agents [2605.30434].

Source: https://www.emergentmind.com/topics/longds