CausaLab: Causal AI Benchmark and DSL

• CausaLab is a collection of integrated frameworks and DSLs designed to rigorously evaluate causal reasoning and counterfactual inference in AI systems.
• It provides synthetic lab environments where LLM agents use structured hypothesis tracking and targeted interventions for causal discovery.
• The platform emphasizes balanced observational and interventional strategies to achieve accurate mechanism recovery and consistent reasoning.

CausaLab refers to several interrelated frameworks, environments, and languages designed for the rigorous study, benchmarking, and development of causal reasoning, especially as it pertains to artificial intelligence systems, LLMs, and vision-LLMs. Across its primary incarnations—namely as a scalable synthetic laboratory environment for interactive causal discovery by LLM agents (Yang et al., 25 May 2026), as the foundation of fine-grained causal reasoning benchmarks for LLMs and vision-LLMs (Chen et al., 2024, Chen et al., 2024), and as an embedded Haskell DSL for causal modeling via neuron diagrams (Walkingshaw et al., 2011)—CausaLab centers on operationalizing and evaluating causal inference, mechanism recovery, and counterfactual reasoning in both computational and philosophical contexts.

1. CausaLab as a Synthetic Benchmark Environment for LLM Agents

CausaLab (Yang et al., 25 May 2026) provides a controlled, scalable, episodic environment where LLM-based agents are evaluated on their ability to perform interactive causal discovery absent prior knowledge. In each episode, an agent is placed in a synthetic laboratory composed of a “manipulator” crystal and a held-out “reactor” crystal, both governed by a hidden, randomly instantiated structural causal model (SCM). The agent’s task is to infer both the underlying causal graph and structural equations by leveraging a combination of passive observation and active intervention.

Episodes unfold through these stages:

1. Hidden SCM instantiation: A random DAG $G=(V,E)$ over endogenous variables $V = O \cup \{Y\}$ (where $O$ are properties like temperature or pressure, and $Y$ is resonance frequency) with assigned structural equations and exogenous terms.
2. Data streams: The environment generates prior measurement records (observational), enables interventions on the manipulator crystal, and reveals properties (but not outcome) for the reactor crystal governed by the same SCM.
3. Interactive loop: The agent alternates between hypothesis emission—using a domain-specific language (DSL) to record its evolving SCM hypothesis—and selecting interventions on $X \in C \subset O$ with a shift-style intervention operator.
4. Final submission: The agent predicts $\hat Y_{\rm reactor}$ and emits a final DSL-encoded SCM hypothesis.

Scoring is dual: (a) Task accuracy requires the agent to predict $Y_{\rm reactor}$ within a specified tolerance, whereas (b) Mechanism-recovery quantifies how precisely the agent’s hypothesized graph and equations match the ground-truth SCM (all-edge $F_1$).

Notably, the environment distinguishes between predictive success and mechanism fidelity: agents may predict accurately under purely observational regimes (92% accuracy at $F_1=0.471$ for GPT-5.2-high in the 6-node setting), yet fail to recover the generative mechanism. Mixed observation-intervention strategies yield better structural fidelity ($F_1=0.80$) but at slightly reduced task accuracy (80%). Pure intervention regimes perform poorly across both metrics. The environment’s architecture enables the systematic study of experimental design, premature commitment, and consistency-aware reasoning in LLMs.

2. Domain-Specific Language for Trajectory-Auditable Hypothesis Tracking

Each agent in CausaLab encodes its causal hypothesis at every interaction step using a structured DSL. The DSL supports serialization of directed edges, explicit frequency equations, and numeric coefficients:

$V = O \cup \{Y\}$0

Incrementality is ensured: only newly discovered elements are added; past beliefs are preserved. A deterministic parser maps the DSL hypothesis into a candidate graph and function set, enabling stepwise auditing and comparison of agent trajectories to the ground truth mechanism (Yang et al., 25 May 2026). This structure is central to experimental transparency and systematic evaluation.

3. Evaluation Metrics and Experimental Regimes

Two principal metrics are central:

• Task Accuracy: $V = O \cup \{Y\}$1 for a given tolerance $V = O \cup \{Y\}$2.
• All-edge $V = O \cup \{Y\}$3 Score: Precision and recall computed on the set of predicted versus true graph edges.

CausaLab compares multiple regimes:

• Purely observational: Agent receives $V = O \cup \{Y\}$4 observations, no interventions; maximizes predictive accuracy but achieves poor mechanism recovery.
• Mixed observation–intervention: A balanced allocation of $V = O \cup \{Y\}$5 observations and $V = O \cup \{Y\}$6 interventions provides the best trade-off.
• Pure intervention: No observations, only interventions; both accuracy and $V = O \cup \{Y\}$7 are low.
• Golden (offline, optimal): Injection of an optimal low-MEC intervention chain in a non-interactive setting.

The table below summarizes results for GPT-5.2-high (6-node setting):

Regime	Task Accuracy (%)	All-edge $V = O \cup \{Y\}$8
Pure observation	92	0.471
Mixed (2 obs + 8 int)	80	0.80
Pure intervention	< 50	< 0.4
Golden (offline)	44	0.574

Mixed strategies demonstrate that integrating observational and interventional data is critical to mechanism learning (Yang et al., 25 May 2026).

4. Agent Behavior, Failure Modes, and Consistency Verification

Empirically, agent strategies exhibit a canonical failure: premature stopping. Agents fit provisional SCMs from observation, execute minimal interventions, and commit to hypotheses before exhausting the intervention budget. This leads to (a) underused informative interventions and (b) hypotheses inconsistent with prior observed data.

A major mitigation is consistency verification: before finalizing a hypothesis, the agent evaluates whether its SCM fits all previously collected data within a set tolerance. If not, experimentation continues until consistency is achieved or the budget is exhausted. This approach increased GPT-5-mini's 4-node task accuracy from 48% to 60%, highlighting the benefit of self-monitoring and trajectory auditing (Yang et al., 25 May 2026).

5. CausaLab in Causal Reasoning Benchmarks and Datasets

CausaLab also serves as the conceptual and infrastructural basis for CELLO (Causal Evaluation of Large Vision-LLMs) (Chen et al., 2024) and is a central component in the CaLM (Causal Evaluation of LLMs) benchmark (Chen et al., 2024). In these contexts, CausaLab enables or inspires rigorous evaluations of model causal reasoning across:

• The full ladder of causation: discovery, association, intervention, counterfactual.
• Multimodal tasks: from language-only to vision-language modalities.
• Structured prompting: e.g., CELLO-CoT, which leverages multi-stage chain-of-thought prompting to guide models through entity identification, causal graph construction, task determination, and application of causal inference rules.

These platforms illuminate persistent failure modes, e.g., superficial pattern-matching, hallucinated answers, and a lack of robustness to adversarial perturbation, while providing large-scale, trajectory-auditable, and scenario-diverse evaluation datasets.

6. CausaLab as a DSL for Philosophical and Computational Causation

CausaLab also denotes a Haskell-embedded domain-specific language for studying and explaining causation via neuron diagrams (Walkingshaw et al., 2011). This implementation supports:

• Expression of causal models as graph-based Haskell code.
• Automated inference of causes using counterfactual propagation.
• Visual explanation of causal relationships and enumeration of alternative “what-if” scenarios.
• Extension to non-Boolean values and the invention of new causal node primitives.

This incarnation foregrounds philosophical precision, extensibility, and the ability to systematically test and visualize causal explanations.

7. Impact, Open Challenges, and Future Directions

CausaLab, across its instantiations, exposes current LLMs’ and LVLMs’ weaknesses as scientific reasoners. Key takeaways include:

• Predictive accuracy does not equate to mechanistic causal understanding.
• Balanced mixed strategies, self-auditing mechanisms, and trajectory-auditable reasoning are instrumental in closing the mechanism recovery gap.
• Current models underexplore experimental action space, prematurely commit, and inadequately leverage interventional data.
• Recommendations for future work include richer hypothesis languages, integration of explicit consistency verification, the development of active-experiment planners, scaling to richer SCM families, and benchmarks that probe dynamic, multimodal, and time-series causal reasoning (Yang et al., 25 May 2026, Chen et al., 2024, Chen et al., 2024).

CausaLab frameworks lay the groundwork for the next generation of causal AI benchmarking, hybrid symbolic-neural causal inference, and philosophical analysis of causation in both artificial and natural agents.