Pre-trained Causal Foundation Models

• Pre-trained causal foundation models are neural architectures that use synthetic data from SCMs combined with transformer designs for robust, zero-shot causal reasoning.
• They support diverse applications such as treatment effect estimation, counterfactual generation, and causal discovery across tabular, time-series, image, and language domains.
• Methodologies include synthetic causal pre-training, amortized causal inference, and explicit structural priors that ensure scalable and interpretable insights in complex systems.

Pre-trained Causal Foundation Models are neural network architectures trained to generalize causal reasoning and inference across tasks, data domains, and modalities via large-scale pre-training—often via synthetic or hybrid datasets explicitly generated from known or hypothesized structural causal models (SCMs). They leverage either “foundation model” architectures such as Transformers, encode explicit causal principles into their structure or training, and support zero-shot or in-context inference for causal queries such as graph discovery, effect estimation, counterfactual generation, and conditional policy optimization. This paradigm encompasses developments in tabular, time-series, computer vision, and natural language domains, unifying advances in amortized Bayesian causal inference, causal graph induction, and counterfactual data generation.

1. Foundational Principles and Design Paradigm

Pre-trained causal foundation models are characterized by three core principles:

1. Synthetic Causal Pre-training: Pre-training is conducted on massive synthetic datasets generated from diverse classes of SCMs. This ensures the model is exposed to numerous causal structures, mechanisms, regimes, and interventions—enabling meta-learning across families of causal queries and statistical settings (Swelam et al., 10 Nov 2025, Liu et al., 25 Jan 2026, Thumm et al., 11 Mar 2026).
2. Amortized Causal Inference: Inference is performed in one or a few forward passes, amortizing (i.e., learning to solve) canonical causal tasks such as treatment effect estimation, causal discovery, or counterfactual prediction, without per-dataset optimization (Balazadeh et al., 9 Jun 2025, Komanduri et al., 22 May 2026, Ma et al., 12 Jun 2025).
3. Foundation Model Architecture: Transformer-style and related backbones are used to maximize model capacity, permutation-invariance, and support complex attention or masking strategies necessary for encoding high-dimensional and structured causal queries (Balazadeh et al., 9 Jun 2025, Swelam et al., 10 Nov 2025, Kougioulis et al., 20 Feb 2026).

The causal foundation model paradigm thus extends the general-purpose adaptability of large pre-trained neural networks to high-level questions of cause and effect, accommodating both observational and interventional data.

2. Formalization and Model Classes

Foundation models for causality instantiate diverse but unifying architectures across domains:

Domain	Foundation Model Example	Causal Target	Training Principle
Tabular	TabPFN, CausalPFN, CausalFM	ATE, ITE, CATE	SCM synthetic sampling, PFN loss
Time-series	LCM, CausalTimePrior, TimesFM	Graph, effect, counterfactual	TSCMs, in-context learning
Images	FM-CGM	Counterfactual image	VLMs+DPMs+causal SCM extraction
Language	CausalBERT, Willig et al.	Causal Q/A	Textual causal pairs, regularized pre-training

Editor's term: For clarity, “PFN-based” models refer to the Prior-data Fitted Network meta-learning framework undergirding many tabular and time-series instantiations (Ma et al., 12 Jun 2025, Swelam et al., 10 Nov 2025, Liu et al., 25 Jan 2026). “Zero-shot” refers to the lack of any downstream adaptation required for new causal domains.

Structural Causal Model (SCM) Priors

Practically all causal foundation models ground their data generation and loss in SCM priors. These are typically sampled as follows (Swelam et al., 10 Nov 2025, Liu et al., 25 Jan 2026, Thumm et al., 11 Mar 2026, Stith et al., 14 May 2026):

• Sample a random DAG (Erdős–Rényi, scale-free, SBM, etc.) over d variables.
• Assign mechanism families per node: linear, MLPs, random Fourier features, autoregressive, etc.
• Draw noise scales, possibly heteroscedastic or non-Gaussian.
• Generate observational data and—by replacing mechanism(s) as per do-calculus—interventional data.

This exposure to both observational and “do-” data is essential for generalization across regimes and causal queries (Thumm et al., 11 Mar 2026, Swelam et al., 10 Nov 2025, Liu et al., 25 Jan 2026).

3. Key Architectures and Methodological Innovations

3.1 Tabular and Treatment Effect Models

• TabPFN and CausalPFN consist of multi-layer, permutation-equivariant transformers fed with tokens representing (contextual) data table rows—covariates, treatments, outcomes—plus query rows for which potential outcomes are inferred (Balazadeh et al., 9 Jun 2025, Swelam et al., 10 Nov 2025, Azqueta-Gavaldon et al., 5 Jun 2026). Models are trained via cross-entropy losses against posterior-predictive targets of SCM-sampled (conditional) distributions.
• CausalFM generalizes this to arbitrary adjustment regimes (back-door, front-door, IV), using cluster-DAG sampling and constrained Bayesian neural network mechanisms in the prior; in-context attention layers amortize posterior inference for CATEs, CAPOs, etc. (Ma et al., 12 Jun 2025).
• CCPFN specifically targets continuous-treatment settings, pre-training transformers to reconstruct individual treatment-response curves using tri-encoder token streams for (covariates, treatments, outcomes), and using histogram-based cross-entropy losses over the support of $\mu_t(x)$ (Stith et al., 14 May 2026).

3.2 Time-Series and Causal Discovery Models

• CausalTimePrior introduces explicit TSCM-based priors, supporting hard/soft/time-varying interventions, nonlinear autoregression, and regime-switching; proof-of-concept PFNs use in-context GRUs/transformers to estimate interventional effects with no per-dataset re-fitting (Thumm et al., 11 Mar 2026).
• Large Causal Models (LCM) pre-train transformer-based encoders on large mixtures of synthetic and realistic sequence/graph pairs to directly infer full lagged adjacency tensors. Input embedding uses Conv1D, correlation injection, and deep transformer stacks; output is a soft causal tensor $\hat{\mathbb{A}}$ (Kougioulis et al., 20 Feb 2026).
• SEA (Sample-Estimate-Aggregate) models use an axial-attention network to combine outputs of classical algorithms run over subsets/marginals and pairwise statistics, reconstructing global causal graphs with theoretical performance guarantees (Wu et al., 2024).

3.3 Foundation Models for Visual Causality

• FM-CGM combines a foundation VLM (Qwen3-VL) for concept and causal graph extraction, a concept manipulator module that applies symbolic do-interventions (reasoning over extracted DAGs), and a diffusion generator (Stable Diffusion XL) with Causal Semantic Guidance (CSG). CSG masks and guides cross-attention at inference to enforce minimal intervention edits on descendant concepts only, preserving invariants (Komanduri et al., 22 May 2026).

3.4 Foundation Models for Causal Structure in Language

• CausalBERT injects explicit cause–effect knowledge into BERT-family encoders by self-supervised classification/ranking over large textual cause–effect pairs, regularized to prevent catastrophic forgetting. This approach enables the model to support causal-pair classification, question-answering, and COPA-style inference tasks (Li et al., 2021).
• Text-prompted causal discovery (Willig et al.): LLMs, when prompted pairwise about variables, reveal “some” causal structure by recalling common-sense and factual causal links, but lack systematic counterfactual or intervention-level reasoning (Willig et al., 2022).

4. Empirical Performance and Applications

Causal foundation models achieve strong empirical accuracy and robustness across diverse causal tasks:

• Treatment Effect Estimation:
  • TabPFN, CausalPFN: Outperform S/T/X-learners, TARNet, CFRNet, and BART in ITE and ATE estimation (e.g., PEHE ≈0.74 vs. TARNet 0.87, ACIC ATE bias ≈0.012) (Balazadeh et al., 9 Jun 2025, Ma et al., 12 Jun 2025).
  • CCPFN establishes state-of-the-art curve reconstruction for continuous-treatments (MISE ≈2.22–3.75) and optimal policy error, outperforming DRNet, VCNet, GPS, CausalForestDML (Stith et al., 14 May 2026).
• Causal Discovery:
  • In tabular domain, adapter frameworks on frozen TabPFN encoder layers extract adjacency matrices with ROC-AUC up to 0.87 (f=5), outperforming GIES, IGSP, DCDI (Swelam et al., 10 Nov 2025).
  • For temporal data, LCMs achieve AUC ≈0.99 (in-distribution), ≈0.93–0.98 (OOD) (Kougioulis et al., 20 Feb 2026); CausalTimePrior-trained PFNs distinguish causal from spurious signals (effect direction accuracy ≈70.4%) (Thumm et al., 11 Mar 2026).
• Visual Counterfactuals:
  • FM-CGM with CSG achieves highest VLM-Eff (0.808) and lowest LPIPS (0.1865) for minimal, causally-consistent image edits, outperforming prompt-guided baselines (Komanduri et al., 22 May 2026).
• Causal Fairness:
  • FairPFN reliably reduces ATE of protected attribute to near zero (0.00±0.06 on synthetic, 0.01±0.03 on Law School admissions), often lying on the Pareto frontier for fairness-accuracy (Robertson et al., 8 Jun 2025).
• Textual Causality:
  • CausalBERT outperforms BERT and RoBERTa on SemEval, NATO-SFA, and COPA causal reasoning tasks (F1=84.2%, COPA accuracy = 93.5%) (Li et al., 2021).

5. Theoretical Guarantees, Generalization, and Limitations

Theoretical Guarantees

• SEA is theoretically capable of reconstructing faithful global causal graphs using only local marginal estimates on graph subsets, leveraging identifiability results for summary statistics and subset patterns (Wu et al., 2024).
• Primal-dual relationships between covariate balancing and attention yield provable optimality for self-attention–based estimators in the design of CInA for treatment effects (Zhang et al., 2023).
• Well-specified SCM priors and cluster-DAG sampling ensure that PFN-based models consistently recover true interventional effects, under identifiability and positivity assumptions (Ma et al., 12 Jun 2025, Stith et al., 14 May 2026).

Generalization and Scalability

• FMs scale with data/model size, exhibiting improved performance log-linearly with both (observed for CPNNs, LCMs, TabPFN) (Swelam et al., 10 Nov 2025, Kougioulis et al., 20 Feb 2026, Stein et al., 2024).
• Mixture pre-training on both synthetic and real data is essential to avoid domain-gap failures and to transfer OOD (Kougioulis et al., 20 Feb 2026).
• Zero-shot inference and amortization yield orders-of-magnitude speed-up over classical or per-dataset optimization methods (LCM, SEA, PFN-based) (Wu et al., 2024, Kougioulis et al., 20 Feb 2026).

Limitations

• In tabular, vision, and time-series settings, causal generalization remains bounded by model- and prior-misspecification, lack of support for hidden confounders, and unmodeled interventions (Swelam et al., 10 Nov 2025, Kougioulis et al., 20 Feb 2026, Thumm et al., 11 Mar 2026).
• Group fairness gains are inconsistent for MNAR covariate shifts; selection bias correction is not solved solely by causal pre-training (Liu et al., 25 Jan 2026).
• Visual causality mediation by VLMs in FM-CGM lacks guarantees of SCM fidelity at pixel-level; only semantic plausibility is enforced (Komanduri et al., 22 May 2026).
• Attention/adapter-based causal token probes (TabPFN) are sensitive to layer selection and may lose signal in over-specialized deep layers (Swelam et al., 10 Nov 2025).
• Scalability to very high-dimensional input (e.g. thousands of features) and cycles/time-variance in graphs requires additional architecture and data design (Stein et al., 2024, Swelam et al., 10 Nov 2025).

6. Impact, Current Use-Cases, and Future Directions

Pre-trained causal foundation models have shifted causal inference towards data-driven, modular, and general workflows.

• Automated inference: TabPFN, CausalPFN, CausalFM, and others enable researchers to obtain calibrated causal estimates and counterfactual predictions without model selection, per-data optimization, or hand-crafted feature engineering (Balazadeh et al., 9 Jun 2025, Ma et al., 12 Jun 2025).
• Interdisciplinary applications: Models have demonstrated utility in genomics, clinical medicine, econometrics, neuroscience (e.g., TimesFM in fMRI causal analysis), climate, and scientific imaging (Crimi et al., 15 Sep 2025, Ma et al., 12 Jun 2025, Komanduri et al., 22 May 2026).
• Interpretable scientific discovery: Foundation models facilitate transparent, post-hoc extraction of causal graphs from internal representations, as shown by causal token adapters and attention-based extractors (Swelam et al., 10 Nov 2025).

Planned and ongoing developments include:

• Expanding to non-backdoor and instrumental variable settings (Ma et al., 12 Jun 2025, Stith et al., 14 May 2026);
• Scaling to high-dimensional, multi-modal, or longitudinal settings via graph-neural, permutation-equivariant, or multi-stream architectures (Kougioulis et al., 20 Feb 2026, Swelam et al., 10 Nov 2025);
• Extending contrastive, disentangled, and structured objectives to disentangle latent generative and observational factors (e.g. physics vs. instrument) (Audenaert et al., 7 Jul 2025);
• Developing robust, fairness-preserving models by integrating selection bias correction and dynamic fairness regularization (Liu et al., 25 Jan 2026, Robertson et al., 8 Jun 2025).

Pre-trained causal foundation models are evolving towards turnkey, data- and task-flexible engines for general-purpose causal reasoning, with increasing accuracy, interpretability, and domain coverage as their foundational principles and architectures are scaled and diversified.