CausalDS: Benchmark for Causal Data Science

• CausalDS is a benchmark that evaluates causal reasoning within realistic data-science workflows using synthetic SCM-derived scenes.
• It integrates symbolic reasoning, statistical estimation, uncertainty quantification, and coding tasks across Pearl’s three rungs.
• The framework enables detailed diagnosis of failures in identification, estimation, and uncertainty by testing both identifiable and non-identifiable scenarios.

CausalDS is a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. From each scene, the benchmark derives tasks spanning all three of Pearl’s rungs, with typical data-science prediction tasks appearing as Rung 1; most tasks include a data science coding component, and correctly abstaining when a question admits no warranted answer is treated as a first-class scored outcome (Leban et al., 9 Jul 2026).

1. Benchmark scope and motivation

CausalDS was introduced against a benchmark landscape that “largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure” (Leban et al., 9 Jul 2026). Its design target is therefore not causal question answering in isolation, and not generic coding evaluation in isolation, but causal reasoning embedded in realistic data-science workflows.

The benchmark’s central object is the synthetic, SCM-grounded scene. This design combines a sampled causal graph, assigned mechanisms and disturbances, generated data, and a natural-language story. The generation process can be empirically grounded using distributions obtained from real-world datasets, “thus retaining empirical structure while reducing the ‘causal parrot’ risk through completely synthetic generation” (Leban et al., 9 Jul 2026). In this formulation, novelty is not an incidental property of prompt wording; it is a property of the underlying sampled SCM and the synthetic scene built from it.

A common misconception is to treat CausalDS as merely a causal-knowledge benchmark with surface-form stories. The benchmark was instead constructed so that symbolic reasoning, statistical estimation, uncertainty quantification, abstention, and tool use/coding are jointly evaluated. This places it closer to an integrated causal data-science environment than to a curated bank of causal vignettes (Leban et al., 9 Jul 2026).

2. Structural causal model generation

Scene construction begins with SCM generation. Graph sampling starts from canonical motifs, including “chain, fork, collider, confounding, IV, frontdoor, mediation,” and for larger graphs uses an “anchor-based grafting” mechanism that combines motifs via shared nodes (Leban et al., 9 Jul 2026). This permits the generation of complex causal structures while preserving recognizable local design patterns.

The underlying SCM is defined by structural assignments of the form

$V_i = f_i(\mathrm{Pa}_i, U_i),$

where $V=(V_1,\ldots,V_n)$ and $G$ is the DAG (Leban et al., 9 Jul 2026). Nodes are assigned mechanism families appropriate for their type, “mixing both empirically extracted mechanism types (e.g., linear, interaction, Boolean logic, neural, mixture-noise, sharp-threshold) and synthetic ones,” while noise families include “Gaussian, Laplace, etc.” (Leban et al., 9 Jul 2026). Some motifs include latent variables, making certain effects non-identifiable and enabling confounding and adjustment scenarios.

The benchmark also controls identifiability regimes during sampling. This matters because many downstream tasks are identification-gated: an agent must decide whether an estimand is identifiable before estimating it. That design is consistent with standard causal methodology in which a causal model is specified, a target parameter is defined, identifiability is assessed, and only then is estimation attempted (Saddiki et al., 2018).

The resulting SCM supplies observational, interventional, and counterfactual ground truth. This explicit use of all three causal levels matches broader formulations that distinguish association, intervention, and counterfactual reasoning as separate but connected task families (Xia et al., 25 Sep 2025).

3. Observation layer, story synthesis, and scene packaging

A defining feature of CausalDS is the separation between the conceptual SCM and the publicly observed data. After fixing the DAG and mechanisms, conceptual variables may be observed only through noisy proxies, and in some cases “only a bundle of noisy measurements is released; sometimes the core variable itself is missing” (Leban et al., 9 Jul 2026). For continuous measurements, the observation model can take the form

$W_{jr} = h_{jr}(Z_j) + \epsilon_{jr},$

and for binary measurements

$W_{jr} \sim \mathrm{Bernoulli}(p_{jr}(Z_j)).$

These observation models are designed to challenge statistical estimation while leaving the underlying SCM semantics unchanged (Leban et al., 9 Jul 2026).

Multiple released variants of the same scene, such as “clean, proxy, proxy_hard,” systematically modify measurement noise, variable availability, or calibration-set size (Leban et al., 9 Jul 2026). A crucial benchmark principle is that “the observation layer never changes causal identifiability—relevant only to estimation difficulty, not SCM semantics” (Leban et al., 9 Jul 2026). This addresses a second common misconception: more difficult observation does not imply a different causal question.

Narrative rendering is handled by a mapper–auditor process. Nodes are mapped to realistic domain variables, stories are generated without graph jargon, and auditors check “full coverage and directionality,” as well as the absence of unsupported links (Leban et al., 9 Jul 2026). This makes the natural-language layer graph-faithful rather than merely plausible.

The scene is then packaged into public and private artifacts.

Artifact type	Contents
Public artifacts	story.md, data.parquet, calibration set, schema.json, task list
Private artifacts	SCM, true graph, effect sizes, identifiability, evaluation labels, observation diagnostics

This packaging design supports deterministic grading while preventing leakage of the latent causal structure (Leban et al., 9 Jul 2026).

4. Task families across Pearl’s rungs

From each scene, CausalDS derives tasks spanning “Association (Rung 1), Intervention (Rung 2), and Counterfactuals (Rung 3)” (Leban et al., 9 Jul 2026). The benchmark does not reduce these rungs to abstract labels; it instantiates them as concrete data-science and causal-inference problems.

Rung	Benchmark examples
Rung 1 — Associational	predicting outcomes, estimating associations, explaining-away via conditioning on colliders
Rung 2 — Interventional	graph reconstruction, identifiability, adjustment sets, effect estimation, bias diagnostics
Rung 3 — Counterfactual	ETT, natural direct/indirect effects, counterfactual identifiability queries

The interventional layer includes tasks such as deciding whether $E[Y \mid do(X)]$ is identifiable, finding adjustment sets, and estimating a population average treatment effect when appropriate (Leban et al., 9 Jul 2026). The counterfactual layer includes the effect of treatment on the treated and mediation-style estimands. Representative formulas appearing in the benchmark specification include

$$\mathrm{ATE} = \mathbb{E}[Y \mid \mathrm{do}(X{=}x_1)] - \mathbb{E}[Y \mid \mathrm{do}(X{=}x_0)]$$

and

$$\mathrm{ETT} = \mathbb{E}[Y_{x_1} - Y_{x_0} \mid X{=}x_1].$$

These are scored against SCM-derived ground truth rather than against heuristic textual judgments (Leban et al., 9 Jul 2026).

Because scene generation explicitly permits latent confounding and non-identifiability, the task family includes situations where no warranted answer exists from the observed data. This creates a natural connection to algorithmic identification frameworks that either derive a do-answer or certify non-identifiability; in the Shpitser–Pearl tradition, non-identifiability can be witnessed by a hedge (Tikka et al., 2018).

5. Evaluation protocol, scoring, and abstention

CausalDS evaluates agents in a “file-backed, code-executing sandbox,” with “each task a fresh agent conversation” (Leban et al., 9 Jul 2026). Agents read files, generate code, run analyses, and produce answer files. The benchmark therefore measures not only whether an answer is correct, but whether an agent can execute the sequence of operations needed to reach it under realistic observational imperfections.

Evaluation is organized along five axes: “(i) symbolic causal reasoning, (ii) data science/statistical execution (via code), (iii) uncertainty quantification, (iv) abstention/epistemic humility, and (v) tool/coding proficiency” (Leban et al., 9 Jul 2026). This decomposition makes it possible to distinguish, for example, a failure to recognize a collider from a failure to fit an estimator, or a failure to abstain from a failure to generate code.

Scoring is deterministic and task-dependent. The benchmark uses exact match, $F_1$, absolute or relative error, ROC-AUC, Brier or log-loss, and proper interval scores (Leban et al., 9 Jul 2026). For interval-valued outputs it adopts the Gneiting–Raftery interval score,

$$\mathrm{IS}_\alpha(\hat\ell,\hat u;\tau) = (\hat u-\hat\ell) + \tfrac{2}{\alpha}(\hat\ell-\tau)\mathbf{1}[\tau < \hat\ell] + \tfrac{2}{\alpha}(\tau-\hat u)\mathbf{1}[\tau > \hat u].$$

Abstention is not treated as a silent failure mode; it is routed to a separate scored pool and positively rewarded when the query is non-identifiable (Leban et al., 9 Jul 2026).

This abstention-aware design is methodologically significant. In standard causal workflow, identifiability is not optional metadata but a prerequisite for valid estimation (Saddiki et al., 2018). CausalDS operationalizes that principle as an explicit benchmark objective.

6. Position within causal data science research

CausalDS occupies a distinctive position relative to adjacent causal research. It is not itself a discovery algorithm, an identification package, or a causal estimator. Instead, it is a benchmark that composes many of the tasks those methods address: graph reasoning, identifiability, effect estimation, counterfactual reasoning, and coding-based statistical execution (Leban et al., 9 Jul 2026).

Its task design is aligned with long-standing causal methodology. The benchmark’s emphasis on specifying an SCM, deriving target causal quantities, checking identifiability, and only then estimating from observed data mirrors the “Causal Roadmap” structure (Saddiki et al., 2018). Its explicit handling of identifiable and non-identifiable intervention questions aligns with algorithmic identification traditions, including systems that either return a closed-form expression for $V=(V_1,\ldots,V_n)$0 or fail with a proof of non-identifiability (Tikka et al., 2018). Its use of all three of Pearl’s rungs also situates it alongside recent work that treats association, intervention, and counterfactual generation or inference as distinct operational levels rather than as a single undifferentiated task family (Xia et al., 25 Sep 2025).

Several design choices address limitations of earlier benchmark paradigms. CausalDS is “integrated, not isolated”; it does not separate symbolic causal reasoning from realistic data analysis. It emphasizes de novo generation rather than a small set of curated examples, thereby reducing contamination and the “causal parrot” risk. It also elevates non-identifiability and abstention to benchmarkable phenomena rather than treating every prompt as answerable (Leban et al., 9 Jul 2026).

A plausible implication is that CausalDS can be used not only for ranking agents, but also for diagnosing which component of a causal data-science workflow has failed: causal structure recognition, identification logic, estimation under noisy observation, uncertainty quantification, or tool use. That diagnostic granularity is a direct consequence of the benchmark’s scene construction, observation layer, multi-rung task derivation, and multi-axis scoring regime (Leban et al., 9 Jul 2026).