Causal Foundation Models

• Causal foundation models are large-scale ML systems that integrate structural causal models and potential outcomes for explicit causal reasoning.
• They utilize transformer-based architectures like ADAG, PFN, and BCAT to recover causal graphs and compute interventional effects from diverse data.
• These models drive applications in causal discovery, counterfactual forecasting, fairness, and prescriptive decision-making across multiple fields.

A causal foundation model is a large-scale machine learning system, typically based on transformer or related architectures, designed to learn, represent, and transfer causal structure and reasoning across many domains or tasks. These models unify principles from structural causal modeling, attention-based neural network design, and in-context learning to achieve robust, zero-shot inference of causal quantities, causal structure, or invariants, and to support downstream applications in scientific discovery, fairness, robustness, and prescriptive decision making.

1. Conceptual Foundations and Definitions

Causal foundation models incorporate and operationalize the formal language of causality—directed acyclic graphs (DAGs), structural equation models (SEMs), and do-interventions—into transferable neural architectures. The central object is a mapping from observed high-dimensional data (tabular, time series, images, or text) to representations of causal structure or causal effect estimands (e.g., interventional distributions, treatment effects).

Two major paradigms support this:

• Structural Causal Models (SCMs): Each variable $X_i$ is generated by a structural equation $X_i = f_i(\mathrm{PA}_i, \varepsilon_i)$, with $\mathrm{PA}_i$ the parent set in the causal graph and $\varepsilon_i$ exogenous noise variables. Interventions replace $f_i$ by a fixed value, producing interventional distributions $P(Y \mid do(X=x))$ (Yin et al., 23 Jun 2025, Ma et al., 12 Jun 2025).
• Potential Outcomes Framework: Each unit possesses potential outcomes $Y(1)$, $Y(0)$ under treatment and control, with average treatment effect (ATE) $E[Y(1)] - E[Y(0)]$ central to causal inference (Ma et al., 12 Jun 2025, Zhang et al., 2023).

A causal foundation model is realized as a foundation model—trained on diverse, task-rich, or synthetic data—augmented with the capacity for explicitly causal reasoning, often implemented as neural networks (transformers, attention blocks, PFNs) pre-trained on generated (SCM-driven) or multi-environment data and able to answer causal queries or recover graphs in new settings without further retraining.

2. Model Architectures and Training Regimes

Architectures for causal foundation models are informed by the requirements of both high expressivity and causal invariance. Notable instantiations include:

• Attention-DAG (ADAG): Learns a nonlinear attention-based map from observational data to DAG adjacency matrices under a linear SEM encoding acyclicity via a smooth trace constraint. Stacks $L$ linear-attention layers, with the output after $L$ layers representing the predicted adjacency (Yin et al., 23 Jun 2025).
• Prior-Data Fitted Networks (PFNs): Permutation-equivariant transformers pre-trained on synthetic data generated from a broad prior over SCMs. CausalFM exemplifies this by embedding a causal prior in the training distribution so the model can perform Bayesian causal inference entirely in-context at test time (Ma et al., 12 Jun 2025). Constraints for well-specified priors enforce identifiability and consistency (priors must guarantee the target estimand is identified from observed data and that the model prioritizes the correct SCM class as data increases).
• Block Causal Transformers (BCAT): Use block-diagonal attention masks to enforce spatial and temporal causality in autoregressive PDE-based prediction tasks, supporting efficient, parallel next-frame prediction and preserving inductive biases for physical dynamics (Liu et al., 31 Jan 2025).
• Self-Attention as Dual Covariate Balancing: CInA (Causal Inference with Attention) demonstrates theoretical equivalence between softmax attention weights and dual optimal balancing weights in SVM-based causal effect estimation, enabling self-supervised multi-task training of zero-shot causal inference transformers (Zhang et al., 2023).
• Neural Aggregators for Graph Recovery: SEA models aggregate marginal estimates from classical causal discovery algorithms (run on many small variable subsets), together with global summary statistics, into global adjacency matrices using layered axial attention blocks, enabling scalable, robust causal discovery (Wu et al., 2024).
• Dual-Encoder Disentanglement: For structured time series, dual encoders with contrastive tasks disentangle physical signal from instrument artifacts, creating separate latent spaces for each causal factor to support robust few-shot prediction and adaptation (Audenaert et al., 7 Jul 2025).

Training is typically performed on large, diverse synthetic or real datasets, often with explicit domain or environment variation and leveraging multi-task or meta-learning objectives. Causal augmentation further includes generating counterfactual or interventional samples (e.g., via SCMs), enforcing invariance or fair representations, or structured masking and contrastive learning (Zhang et al., 18 Dec 2025, Bühler et al., 7 Jan 2026).

3. Causal Inference, Discovery, and Zero-shot Transfer

A principal advantage of causal foundation models is their ability to perform causal inference and/or discovery in new domains without retraining—"zero-shot" generalization. This is achieved via:

• Implicit Low-Dimensional Priors: By jointly pre-training across many domains, models like ADAG learn a shared low-dimensional causal prior, enabling better DAG recovery on novel small-sample tasks (Yin et al., 23 Jun 2025).
• Amortized Bayesian Inference: PFNs trained on SCM-generated data can compute conditional interventional distributions or treatment effects for new input datasets directly via forward passes, outperforming or matching classical per-dataset estimators with improved computational efficiency (Ma et al., 12 Jun 2025, Saretzky et al., 30 Nov 2025).
• Direct Causal Structure Prediction: Large neural networks can recover global causal graphs from local or marginal estimates and global summary statistics (SEA), and through supervised "causal pretraining" directly predict causal graphs from time series with scaling of generalization as model/data size increases (Wu et al., 2024, Stein et al., 2024).
• Domain Knowledge Integration: Recent advances condition causal foundation models on full or partial domain-knowledge graphs via attention biasing or GCN-based encoding, allowing flexible, plug-in use of expert knowledge to enhance causal effect estimation (Reuter et al., 16 Feb 2026).

4. Applications: Discovery, Robustness, Fairness

The breadth of causal foundation models encompasses several key scientific and operational applications:

• Causal Discovery: Rapid, robust discovery of causal structure in biological (e.g., single-cell) or engineered systems, supporting downstream modeling and interventional planning (Wu et al., 2024, Yin et al., 23 Jun 2025).
• Counterfactual and Interventional Forecasting: Time-series foundation models can simulate rare events (e.g., market crashes) via causal interventions in hidden states ("activation transplantation"), revealing steerable, compositional latent concept subspaces and supporting "what-if" stress testing (Sanyal et al., 6 Sep 2025).
• Domain Generalization and Robustness: Disentangling causal from non-causal or spurious features (e.g., via frequency-domain methods) in vision models enables robust performance under distribution shift and adverse conditions (Zhang et al., 18 Dec 2025).
• Prescriptive Decision-Making: In manufacturing, causal foundation models (PFN-based) enable simulation of intervention effects on system-level KPIs like OEE for operational optimization through formal ranking of candidate interventions via their causal effects (Saretzky et al., 30 Nov 2025).
• Fairness: PFN-based causal foundation models (FairPFN) can mitigate the direct and indirect effects of protected attributes in predictions—without access to the ground-truth causal graph—by learning from millions of synthetic fairness-focused SCMs (Robertson et al., 8 Jun 2025).

5. Integrating Causality with Foundation Model Practice

Integrating causality into foundation models shapes both architectural and algorithmic design:

• Causal Pretraining and Data Augmentation: Models are trained on large volumes of SCM-driven, counterfactual, or interventional data to enforce invariant or fair representations and causal feature learning (Binkyte et al., 28 Feb 2025, Bühler et al., 7 Jan 2026).
• Causal Representation Learning: Regularizers or constraints are implemented to promote disentanglement and invariance of latent representations, aligning latent features with causal parents or separating physical and nuisance factors (Zhang et al., 18 Dec 2025, Audenaert et al., 7 Jul 2025, Rajendran et al., 2024).
• Post-training Causal Alignment and Auditing: At fine-tuning or deployment, causal penalties (e.g., counterfactual fairness loss) or direct interventions in model hidden states (e.g., steering vectors in LLMs) operationalize causal guarantees in output behavior (Rajendran et al., 2024, Binkyte et al., 28 Feb 2025).
• Causal Evaluation: Auditing models with formal do-calculus or counterfactual tests ensures that interventions (e.g., flipping a protected attribute) yield outputs satisfying causal invariance or privacy constraints (Binkyte et al., 28 Feb 2025, Robertson et al., 8 Jun 2025).
• Balancing Competing Objectives: Causality enables principled trade-offs among accuracy, fairness, privacy, robustness, and explainability by isolating path-specific effects and permitting targeted interventions or obfuscations in dependence with model goals (Binkyte et al., 28 Feb 2025).

6. Limitations, Challenges, and Future Directions

Causal foundation models face several active challenges:

• Causal Knowledge Specification: For high-dimensional data (vision, language), discovering, encoding, or even validating appropriate causal graphs is nontrivial, and discovery algorithms remain sample- and computation-intensive at scale (Binkyte et al., 28 Feb 2025, Reuter et al., 16 Feb 2026).
• Identifiability in Non-Synthetic Domains: Identifiability guarantees may break down outside the training distribution, or when the causal prior mismatches real data-generating mechanisms, especially in presence of hidden confounders or cycles (Wu et al., 2024, Bongers et al., 2016, Bühler et al., 7 Jan 2026).
• Concept Superposition and Mapping: The entanglement of multiple semantic concepts in shared neural features complicates mapping of causal graph nodes to model representations and thus the controlled intervention or fair prediction (Binkyte et al., 28 Feb 2025, Rajendran et al., 2024).
• Scalability: Computational cost is a barrier for models operating on high-dimensional SCM priors or requiring large numbers of interventional samples. Efficient approximations and parameter-efficient updates (e.g., LoRA, modular adaptation) are areas of active research (Binkyte et al., 28 Feb 2025, Yin et al., 23 Jun 2025).
• Transportability and Generalization: Ensuring that learned causal relations and invariants transfer across modalities, languages, or domains remains an open research problem (Binkyte et al., 28 Feb 2025, Saretzky et al., 30 Nov 2025).

Research directions include scalable causal discovery for LLM and vision backbones, dynamic causal alignment during continual fine-tuning, data-efficient representation learning for causal concepts, and unified benchmarks that evaluate multi-objective, interventional, and counterfactual performance in diversified domains.

7. Summary Table: Core Causal Foundation Model Instantiations

Model/Approach	Domain	Core Objective	Distinguishing Features	Reference
ADAG	Tabular/SEM	Causal graph discovery	Attention-based, multi-task prior, zero-shot inference	(Yin et al., 23 Jun 2025)
PFN/CausalFM	Tabular	Causal inference	SCM-driven synthetic pretraining, in-context deep Bayesian learning	(Ma et al., 12 Jun 2025)
SEA	Biological/causal discovery	Global graph estimation	Aggregates classical outputs from subgraphs, robust to misspecification	(Wu et al., 2024)
FairPFN	Tabular/fairness	Causal fairness	SCM-generated bias/fair datasets, no graph input required	(Robertson et al., 8 Jun 2025)
BCAT	Spatiotemporal/PDE	Fluid dynamics prediction	Block causal masks encode spatio-temporal structure	(Liu et al., 31 Jan 2025)
PriMa-Causa	Manufacturing	Prescriptive maintenance	PFN for "what-if" intervention analysis on production KPIs	(Saretzky et al., 30 Nov 2025)
time2time	Time-series	Rare event simulation	Direct causal interventions in hidden state statistics	(Sanyal et al., 6 Sep 2025)
Causal Pretraining	Time-series	End-to-end graph recovery	Scaling with data/model size, zero-shot causal discovery	(Stein et al., 2024)
Causal Foundation Models with Partial Graphs	Tabular	CID estimation with partial domain knowledge	Attention biasing by partial ancestors, GCN encoding	(Reuter et al., 16 Feb 2026)

Causal foundation models represent the convergence of scalable neural model design and the rigorous demands of causal reasoning, forming a new paradigm in which generalizable, interpretable, and interventional semantics are embedded at the architecture, data, and objective level. Their deployment changes both the practical workflow and foundational limits of causal inference, discovery, and robust decision making in scientific, engineering, and societal domains.