---
title: 'CausalDS: Benchmark for Causal Data Science'
url: https://www.emergentmind.com/topics/causalds
type: topic
---

# CausalDS: Benchmark for Causal Data Science

CausalDS is a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a *scene* consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. From each scene, the benchmark derives tasks spanning all three of Pearl’s rungs, with typical data-science prediction tasks appearing as Rung 1; most tasks include a data science coding component, and correctly abstaining when a question admits no warranted answer is treated as a first-class scored outcome [2607.08093].

## 1. Benchmark scope and motivation

CausalDS was introduced against a benchmark landscape that “largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure” [2607.08093]. Its design target is therefore not causal question answering in isolation, and not generic coding evaluation in isolation, but causal reasoning embedded in realistic data-science workflows.

The benchmark’s central object is the synthetic, SCM-grounded scene. This design combines a sampled causal graph, assigned mechanisms and disturbances, generated data, and a natural-language story. The generation process can be *empirically grounded* using distributions obtained from real-world datasets, “thus retaining empirical structure while reducing the ‘causal parrot’ risk through completely synthetic generation” [2607.08093]. In this formulation, novelty is not an incidental property of prompt wording; it is a property of the underlying sampled SCM and the synthetic scene built from it.

A common misconception is to treat CausalDS as merely a causal-knowledge benchmark with surface-form stories. The benchmark was instead constructed so that symbolic reasoning, statistical estimation, uncertainty quantification, abstention, and tool use/coding are jointly evaluated. This places it closer to an integrated causal data-science environment than to a curated bank of causal vignettes [2607.08093].

## 2. Structural causal model generation

Scene construction begins with SCM generation. Graph sampling starts from canonical motifs, including “chain, fork, collider, confounding, IV, frontdoor, mediation,” and for larger graphs uses an “anchor-based grafting” mechanism that combines motifs via shared nodes [2607.08093]. This permits the generation of complex causal structures while preserving recognizable local design patterns.

The underlying SCM is defined by structural assignments of the form
\[
V_i = f_i(\mathrm{Pa}_i, U_i),
\]
where \(V=(V_1,\ldots,V_n)\) and \(G\) is the DAG [2607.08093]. Nodes are assigned mechanism families appropriate for their type, “mixing both empirically extracted mechanism types (e.g., linear, interaction, Boolean logic, neural, mixture-noise, sharp-threshold) and synthetic ones,” while noise families include “Gaussian, Laplace, etc.” [2607.08093]. Some motifs include latent variables, making certain effects non-identifiable and enabling confounding and adjustment scenarios.

The benchmark also controls identifiability regimes during sampling. This matters because many downstream tasks are *identification-gated*: an agent must decide whether an estimand is identifiable before estimating it. That design is consistent with standard causal methodology in which a causal model is specified, a target parameter is defined, identifiability is assessed, and only then is estimation attempted [1809.02408].

The resulting SCM supplies observational, interventional, and counterfactual ground truth. This explicit use of all three causal levels matches broader formulations that distinguish association, intervention, and counterfactual reasoning as separate but connected task families [2509.20846].

## 3. Observation layer, story synthesis, and scene packaging

A defining feature of CausalDS is the separation between the conceptual SCM and the publicly observed data. After fixing the DAG and mechanisms, conceptual variables may be observed only through noisy proxies, and in some cases “only a *bundle* of noisy measurements is released; sometimes the core variable itself is missing” [2607.08093]. For continuous measurements, the observation model can take the form
\[
W_{jr} = h_{jr}(Z_j) + \epsilon_{jr},
\]
and for binary measurements
\[
W_{jr} \sim \mathrm{Bernoulli}(p_{jr}(Z_j)).
\]
These observation models are designed to challenge statistical estimation while leaving the underlying SCM semantics unchanged [2607.08093].

Multiple released variants of the same scene, such as “clean, proxy, proxy_hard,” systematically modify measurement noise, variable availability, or calibration-set size [2607.08093]. A crucial benchmark principle is that “the observation layer never changes causal identifiability—relevant only to estimation difficulty, not SCM semantics” [2607.08093]. This addresses a second common misconception: more difficult observation does not imply a different causal question.

Narrative rendering is handled by a mapper–auditor process. Nodes are mapped to realistic domain variables, stories are generated without graph jargon, and auditors check “full coverage and directionality,” as well as the absence of unsupported links [2607.08093]. This makes the natural-language layer graph-faithful rather than merely plausible.

The scene is then packaged into public and private artifacts.

| Artifact type | Contents |
|---|---|
| Public artifacts | `story.md`, `data.parquet`, calibration set, `schema.json`, task list |
| Private artifacts | SCM, true graph, effect sizes, identifiability, evaluation labels, observation diagnostics |

This packaging design supports deterministic grading while preventing leakage of the latent causal structure [2607.08093].

## 4. Task families across Pearl’s rungs

From each scene, CausalDS derives tasks spanning “Association (Rung 1), Intervention (Rung 2), and Counterfactuals (Rung 3)” [2607.08093]. The benchmark does not reduce these rungs to abstract labels; it instantiates them as concrete data-science and causal-inference problems.

| Rung | Benchmark examples |
|---|---|
| Rung 1 — Associational | predicting outcomes, estimating associations, explaining-away via conditioning on colliders |
| Rung 2 — Interventional | graph reconstruction, identifiability, adjustment sets, effect estimation, bias diagnostics |
| Rung 3 — Counterfactual | ETT, natural direct/indirect effects, counterfactual identifiability queries |

The interventional layer includes tasks such as deciding whether \(E[Y \mid do(X)]\) is identifiable, finding adjustment sets, and estimating a population average treatment effect when appropriate [2607.08093]. The counterfactual layer includes the effect of treatment on the treated and mediation-style estimands. Representative formulas appearing in the benchmark specification include
\[
\mathrm{ATE} = \mathbb{E}[Y \mid \mathrm{do}(X{=}x_1)] - \mathbb{E}[Y \mid \mathrm{do}(X{=}x_0)]
\]
and
\[
\mathrm{ETT} = \mathbb{E}[Y_{x_1} - Y_{x_0} \mid X{=}x_1].
\]
These are scored against SCM-derived ground truth rather than against heuristic textual judgments [2607.08093].

Because scene generation explicitly permits latent confounding and non-identifiability, the task family includes situations where no warranted answer exists from the observed data. This creates a natural connection to algorithmic identification frameworks that either derive a do-answer or certify non-identifiability; in the Shpitser–Pearl tradition, non-identifiability can be witnessed by a hedge [1806.07161].

## 5. Evaluation protocol, scoring, and abstention

CausalDS evaluates agents in a “file-backed, code-executing sandbox,” with “each task a fresh agent conversation” [2607.08093]. Agents read files, generate code, run analyses, and produce answer files. The benchmark therefore measures not only whether an answer is correct, but whether an agent can execute the sequence of operations needed to reach it under realistic observational imperfections.

Evaluation is organized along five axes: “(i) symbolic causal reasoning, (ii) data science/statistical execution (via code), (iii) uncertainty quantification, (iv) abstention/epistemic humility, and (v) tool/coding proficiency” [2607.08093]. This decomposition makes it possible to distinguish, for example, a failure to recognize a collider from a failure to fit an estimator, or a failure to abstain from a failure to generate code.

Scoring is deterministic and task-dependent. The benchmark uses exact match, \(F_1\), absolute or relative error, ROC-AUC, Brier or log-loss, and proper interval scores [2607.08093]. For interval-valued outputs it adopts the Gneiting–Raftery interval score,
\[
\mathrm{IS}_\alpha(\hat\ell,\hat u;\tau)
= (\hat u-\hat\ell)
+ \tfrac{2}{\alpha}(\hat\ell-\tau)\mathbf{1}[\tau < \hat\ell]
+ \tfrac{2}{\alpha}(\tau-\hat u)\mathbf{1}[\tau > \hat u].
\]
Abstention is not treated as a silent failure mode; it is routed to a separate scored pool and positively rewarded when the query is non-identifiable [2607.08093].

This abstention-aware design is methodologically significant. In standard causal workflow, identifiability is not optional metadata but a prerequisite for valid estimation [1809.02408]. CausalDS operationalizes that principle as an explicit benchmark objective.

## 6. Position within causal data science research

CausalDS occupies a distinctive position relative to adjacent causal research. It is not itself a discovery algorithm, an identification package, or a causal estimator. Instead, it is a benchmark that composes many of the tasks those methods address: graph reasoning, identifiability, effect estimation, counterfactual reasoning, and coding-based statistical execution [2607.08093].

Its task design is aligned with long-standing causal methodology. The benchmark’s emphasis on specifying an SCM, deriving target causal quantities, checking identifiability, and only then estimating from observed data mirrors the “Causal Roadmap” structure [1809.02408]. Its explicit handling of identifiable and non-identifiable intervention questions aligns with algorithmic identification traditions, including systems that either return a closed-form expression for \(P_x(y)\) or fail with a proof of non-identifiability [1806.07161]. Its use of all three of Pearl’s rungs also situates it alongside recent work that treats association, intervention, and counterfactual generation or inference as distinct operational levels rather than as a single undifferentiated task family [2509.20846].

Several design choices address limitations of earlier benchmark paradigms. CausalDS is “integrated, not isolated”; it does not separate symbolic causal reasoning from realistic data analysis. It emphasizes de novo generation rather than a small set of curated examples, thereby reducing contamination and the “causal parrot” risk. It also elevates non-identifiability and abstention to benchmarkable phenomena rather than treating every prompt as answerable [2607.08093].

A plausible implication is that CausalDS can be used not only for ranking agents, but also for diagnosing which component of a causal data-science workflow has failed: causal structure recognition, identification logic, estimation under noisy observation, uncertainty quantification, or tool use. That diagnostic granularity is a direct consequence of the benchmark’s scene construction, observation layer, multi-rung task derivation, and multi-axis scoring regime [2607.08093].

Source: https://www.emergentmind.com/topics/causalds