Causal Judgment Tasks Overview

• Causal Judgment Tasks are formal experiments and computational models used to assess how both humans and AI infer, endorse, or generalize causal relationships.
• They employ diverse paradigms—graphical entailment, probabilistic judgments, and counterfactual scenarios—to robustly benchmark causal reasoning abilities.
• These tasks aid in advancing cognitive science and AI alignment by elucidating model biases, performance metrics, and underlying inference mechanisms.

Causal judgment tasks are formal experimental and computational procedures designed to elicit, evaluate, and model the ways in which human and artificial agents infer, endorse, or generalize about causal relationships between variables, events, interventions, or agents. These tasks serve both as cognitive probes into underlying reasoning mechanisms and as benchmarks for evaluating the causal capabilities of machine learning systems, notably LLMs. Causal judgment tasks can take diverse forms, from structured queries in graphical and probabilistic settings to open-ended vignettes assessing norms, counterfactuals, or attributions. As a research program, causal judgment tasks bridge cognitive science, AI alignment, natural language processing, and formal causal inference.

1. Fundamental Task Designs and Formal Definitions

Causal judgment tasks are operationalized in multiple paradigms, each tailored to probe a target aspect of causal reasoning.

Graphical-Structure Entailment Tasks. The Corr2Cause format (Sun et al., 23 May 2025) exemplifies a deductive-causal-judgment paradigm. For random variables $X = \{X_1, ..., X_n\}$, each instance is defined by:

• A premise set $S$ of observed statistical relations (e.g., "X₁ is correlated with X₂", "X₃ is independent of X₁ given X₂"),
• A causal hypothesis $h$ (e.g., "$X_a$ causes $X_b$").

The label $y \in \{\text{Yes},\text{No}\}$ is determined by checking whether $h$ is logically entailed for all directed acyclic graphs (DAGs) consistent with $S$, under the rules of $d$-separation. The output is:

$$y = \text{Yes} \iff \forall G \in \mathcal{D} :\; X_a \leadsto X_b \text{ and no back-door confounding}$$

The Corr2Cause dataset uses English premise lists, binary hypothesis statements, and is heavily class-imbalanced toward "No" (≈80:20).

Probabilistic Judgment and Contingency Tasks. In classic contingency judgment (Carro et al., 15 Oct 2025), agents are shown trial tables varying the presence/absence of a putative cause $C$ and an effect $E$. The key statistic, $\Delta P = P(E|C) - P(E|\neg C)$, operationalizes normative causal strength. Causal illusions—judging $C$ to be causal when $\Delta P = 0$—are a primary target of assessment. In adapted LLM protocols, models are prompted to assign effectiveness ratings to null-contingency tables.

Causal Attribution Networks. In network-exploration tasks (Berenberg et al., 2018), causal judgments are decomposed into microtasks—link validation, chain extension, and pathway refinement. Workers or models propose, edit, and validate directed links, thereby constructing large causal attribution graphs. Accuracy is defined via precision, recall, and F1 relative to a gold or crowd-majority network.

Counterfactual and Counter-normative Scenarios. MoCa (Nie et al., 2023) and T³ (Chang, 13 Jan 2026) benchmarks adopt human-inspired vignettes, with participants or LLMs asked direct counterfactual questions ("If X had not occurred, would Y have happened?"), norm-violation attributions, or ambiguity-resolution queries. Performance is measured by agreement with human-majority or expert ground truth, with explicit handling of "ambiguous" cases where a wise refusal is warranted.

Collider Reasoning Tasks. In recent collider-graph protocols (Dettki, 10 Dec 2025, Dettki et al., 3 Feb 2026), agents judge the probability of cause or effect nodes given partial information in a $C_1 \to E \leftarrow C_2$ DAG, often without explicit parameters. These responses are then modeled by fitting noisy-OR causal Bayes nets, extracting interpretable causal-strength and leakage parameters.

2. Evaluation Metrics and Benchmarking Principles

Causal judgment tasks require specialized, fine-grained metrics:

Metric	Definition / Formula	Use Case
F₁ (positive)	$$F_1 = 2 \frac{\text{precision} \times \text{recall}}{\text{precision} + \text{recall}}$$	Skewed binary tasks (Sun et al., 23 May 2025)
Utility	$\displaystyle \frac{TP}{TP+FN}$	Sensitivity to true positives (Chang, 13 Jan 2026)
Safety	$\displaystyle \frac{TN}{TN+FP}$	Specificity to true negatives (Chang, 13 Jan 2026)
WRR	$$\displaystyle \frac{\#\{\hat{y}=\text{AMBIGUOUS}\wedge y=\text{AMBIGUOUS}\}}{\#\{y = \text{AMBIGUOUS}\}}$$	Wise refusal on underdetermined
Spearman ρ	Rank correlation between human and agent judgments	Human–machine alignment
MAE, R²	Mean absolute error and leave-one-out $R^2$ on fitted causal Bayes nets	Causal-consistency diagnostics (Dettki, 10 Dec 2025, Dettki et al., 3 Feb 2026)

In multi-class or multi-extraction settings, metrics include macro-averaged F1, exact match, and token-level span alignment for extraction-based pipelines (Yang et al., 2022). For crowd tasks, additional metrics include edit-efficiency (time per contributed link), convergence statistics ($\varphi_{nodes}$, $\varphi_{edges}$), and motif z-scores.

3. Empirical Findings, Human–LLM Comparisons, and Pathologies

Broad experimental results from cognitive science and recent machine reasoning studies indicate several recurring patterns:

Surface-level pitfalls in LLMs. Direct-prompted LLMs often exploit spurious cues such as explicit wording ("causes"/"correlates") and overfit to paraphrasing or lexical pattern-matching, resulting in marginal generalization ability over random baselines (e.g., GPT-4 F₁=29.08 vs. Random F₁=20.38 on Corr2Cause). Structured intermediate representations (knowledge graphs) substantially alleviate these limitations, yielding up to 47.5% relative improvement in F₁ (Sun et al., 23 May 2025).

Cognitive bias replication and divergence. In null-contingency paradigms (Carro et al., 15 Oct 2025), LLMs exhibit illusions of causality paralleling, but sometimes exceeding, those observed in humans (GPT-4o-Mini median effectiveness rating 75 on normatively zero-effect scenarios). However, in collider tasks (Dettki, 10 Dec 2025, Dettki et al., 3 Feb 2026), LLMs deviate from the typical "open-world" human heuristics (high background leakage, weak explaining away) and instead display tight, "closed-world" rule reasoning (high causal strength, low leak, strong explaining away, and near-zero Markov violation). Humans, by contrast, tend to assume unmentioned causes and reveal weak Bayesian signatures.

Ambiguity, wise refusal, and over-cautiousness. T³ benchmarking shows that safety-aligned models can fall into "skepticism traps," over-refusing valid causal links for specificity (up to 60% false negatives), while larger models may "paralyze" under L3 (counterfactual) ambiguity, defaulting to excessive hedging rather than committing to a direction (Chang, 13 Jan 2026).

Crowdsourcing and structured-decomposition gains. Microtask decomposition (Iterative Pathway Refinement) yields statistically efficient and accurate causal-attribution networks in crowd experiments (Berenberg et al., 2018), with clear efficiency gains over single-link tasks and motif structures closely resembling real-world causal systems.

Generalization and resource constraints in humans. Behavioral studies reveal that human causal judgments are sensitive to evidence order (generalization-order effects), category focus (agent/recipient asymmetry (Zhao et al., 2021)), and resource rationality, rarely recomputing global posteriors but committing to early category formations that predict systematic transfer patterns.

4. Methods: Structured Reasoning, Graph Induction, and Protocol Design

Structured Thinking Pipelines. Inducing intermediate explicit representations—knowledge graphs or adjacency structures—enables decomposition of mapping (text $\rightarrow$ structure) from graph-based logical inference, improving robustness to natural language variation and avoiding spurious overfitting (Sun et al., 23 May 2025). Prompts guiding graph construction to explicit JSON schemas, followed by structured graph evaluation, have proven experimentally superior to direct end-to-end labeling, nearly doubling recall rates.

Out-of-distribution robustness. Chain-of-thought (CoT) prompting consistently elevates LLM self-consistency and human-alignment, especially under abstraction or prompt overload (Dettki, 10 Dec 2025, Dettki et al., 3 Feb 2026). CoT regularizes probabilistic and causal judgments, increases explaining-away effect strength, and suppresses Markov violations relative to direct (one-shot) response protocols.

Ambiguity- and refusal-sensitive evaluation. Benchmarks must explicitly support underdetermined cases, scoring not only label accuracy but also "wise refusal" when a scenario is logically ambiguous. High-resolution axes such as Utility (sensitivity), Safety (specificity), and WRR (wise refusal rate) are necessary to diagnose and mitigate asymmetric failure modes—over-skepticism (false rejection) and over-endorsement (false acceptance) (Chang, 13 Jan 2026).

Microtask and crowdsourcing decomposition. Efficient causal network mapping benefits from propose/rank/vote protocols and iterative pathway-edit interfaces. Task efficiency and coverage rates are rigorously evaluated via pathway-length-normalized metrics and edge-motif statistics (Berenberg et al., 2018).

Extraction and fine-grained relation modeling. In event causality extraction and causal QA (Yang et al., 2022), models must not only identify cause/effect spans but also categorize relation types (Cause, Cause_By, Enable, Prevent, etc.) and answer span-based causal questions. Span-extraction plus classification and multi-hop, cross-sentence reasoning are central requirements for success.

5. Cognitive and Theoretical Underpinnings

Probabilistic causal-graph frameworks. Bayesian network inference and d-separation rules provide normative reference algorithms for causal judgment. Empirical results show that, given explicit subjective-probability elicitation, adults can approximate Bayesian updates in causal discounting tasks (Morris et al., 2013), and Piagetian developmental trajectories correspond to increasing sophistication in detecting independence patterns and inferring mediation or confounding.

Subjective causality and preference-based models. Formally, observed agent preferences over interventions, when satisfying axioms such as cancellation, definiteness, and recursivity, uniquely identify a subjective recursive structural-equation model, context distribution, and utility, enabling rationalization of any set of causal preferences in expected-utility terms (Halpern et al., 2024).

Bayesian causal induction. Probability tree models encode agent beliefs over causal hypotheses, with interventions modeled as "clamped" node resolutions, and constraints enforced by restricting hypothesis priors (Ortega, 2011). These frameworks reproduce both child and adult learning patterns in classical tasks such as the blicket detector.

Fine-grained semantic distinctions. Datasets like FCR (Yang et al., 2022) introduce fine-grained causal categories (Enable, Prevent, etc.), multi-relation and multi-sentence reasoning, and counterfactual QA formats, highlighting challenges inherent in true scientific or policy causal judgment tasks.

6. Practical Guidelines and Implications

• Explicit induction of structured, machine-readable representations (knowledge graphs, adjacency lists) offers dramatic gains in robustness and alignment in both LLM and crowd pipelines (Sun et al., 23 May 2025, Berenberg et al., 2018).
• Chain-of-thought reasoning and staged justification protocols increase task-level consistency, suppress LLM failure modes, and enable better diagnostic alignment to normative, graph-based causal reasoning (Dettki, 10 Dec 2025, Dettki et al., 3 Feb 2026).
• Ambiguity-resolving protocols that allow or mandate wise refusal must be integrated to prevent superficial accuracy masking failures in uncertainty-handling (Chang, 13 Jan 2026).
• For scientific or policy applications, benchmarks should encode real downstream causal questions, avoid proxy metrics such as pure classification accuracy, and incorporate annotation and evaluation splits that respect randomization and unconfoundedness (Cadei et al., 2024).
• Adopting propose-vote-refine microtask workflows supports scalable causal network exploration, with broad applicability across domains (Berenberg et al., 2018).
• Behavioral task design should leverage order effects, asymmetry manipulations, and resource-rational approximations to probe causal program generalization and revision in human and machine learners (Zhao et al., 2021).
• Extraction-based causal NLP tasks must expand beyond single-cause labels, supporting multi-hop inference and counterfactual decision-making (Yang et al., 2022).

Causal judgment tasks, codified across a spectrum of empirical, computational, and applied settings, are central to the study and evaluation of both natural and artificial causal reasoning. Advances in structured reasoning protocols, metric development, and task decomposition have begun to close key gaps in reliability, generalization, and interpretability of both human and machine causal judgments.