Pre-trained Causal Foundation Model

Updated 7 December 2025

Pre-trained causal foundation models are transformer architectures pre-trained on synthetic SCM datasets to encode causal relationships and extract interventions without retraining.
They utilize SCM sampling, dual-attention encoding, and adapter probes to capture both statistical and causal structures, optimizing metrics like ROC-AUC and AP.
These models support zero-shot causal discovery, effect estimation, and prescriptive analytics, effectively bridging prediction and causation in diverse applications.

A pre-trained causal foundation model is a parameterized machine learning model—typically based on transformer architecture—that is pre-trained on large libraries of synthetic or real data generated under explicit structural causal model (SCM) priors so as to capture, encode, and (often) enable direct extraction of causal relationships, interventions, and effects without retraining or specialized adjustment for each new task. Such models aim to bridge prediction and causation, supporting applications ranging from causal discovery and effect estimation to what-if simulation and fairness. Both the representational and practical capabilities of causal foundation models depend critically on the breadth of their synthetic SCM priors, pre-training protocols, probing/intervention mechanisms, and the learned geometric topology of their internal layers.

1. Pre-training Regimes: SCM Sampling and Data Construction

Pre-trained causal foundation models are defined by their synthetic data generation—and therefore by their induced "causal prior." Most recent architectures draw random directed acyclic graphs (DAGs) $G$ from a family of topology primitives (e.g., Erdős–Rényi, scale-free, Watts–Strogatz, stochastic-block, geometric) with various expected edge counts and graph-specific parameters. The nodes in these DAGs represent observable variables; edges encode direct causal influence as per the semantics of structural equation models.

For each node $j$ , values $x_j$ are simulated according to additive-noise SCM mechanisms:

$x_j = f_j(x_{\mathrm{pa}(j)}) + h_j(x_{\mathrm{pa}(j)}) \cdot \epsilon_j, \quad \epsilon_j \sim \left\{ \begin{array}{l} N(0, \sigma^2) \ \text{Laplace}(0, \sigma^2) \ \text{Cauchy}(0, \sigma^2) \end{array} \right.$

with $\sigma \sim \mathrm{Unif}(0.2, 2)$ . The $f_j$ can be linear (random weights and biases) or nonlinear (random Fourier features, etc.). Synthetic datasets include mixtures of observational samples and explicit interventional samples—where 50% of the time a random variable is set independently, simulating $\mathrm{do}$ operations.

During pre-training, each dataset spans a range of feature set sizes (typically $f \in [4, 20]$ ), and samples are split between purely observational and mixed observational/interventional blocks. This protocol is designed to enable the model to internalize generic causal dependencies, mechanisms, and the semantics of interventions (Swelam et al., 10 Nov 2025).

2. Model Architecture: Embeddings and Attention Mechanisms

Most causal foundation models leverage transformer architectures to process and encode synthetic SCM datasets. For instance, TabPFN uses an embedding pipeline where each datum $X \in \mathbb{R}^{n \times f}$ , together with cell-level intervention bits, is embedded into $H_0 \in \mathbb{R}^{n \times f \times d}$ via learned projections. Dual-attention encoding alternates between row-wise (sample) and column-wise (feature) multi-head self-attention, typically over 12 layers (Swelam et al., 10 Nov 2025).

Causal dependencies are captured in the model's internal representations. Specifically, attention heads enable each feature-value token to globally attend across samples and features, allowing the network to encode both statistical and causal structure as permitted by the synthetic SCM prior. For probe-based extraction of causal graphs, the internal representations from intermediate layers (often $L = 4, 5, 6$ ) are found to be optimal, suggesting a "mid-layer sweet spot."

Causal foundation models can also include architectural innovations for disentanglement (e.g., dual-encoder designs for separating physical from instrumental factors) or specialized mechanisms for fairness (e.g., protected-attribute encoding) (Audenaert et al., 7 Jul 2025, Robertson et al., 8 Jun 2025).

3. Probing Causal Knowledge: Adapters, Decoding, and Effect Extraction

To extract causal information stored within frozen foundation models, adapter frameworks are employed. For TabPFN, a set of $t=30$ "causal tokens" per feature (initialized as learnable queries) are cross-attended against frozen encoder representations. After a series of decoder layers, token statistics (max, min, mean, std) are aggregated, and summary vectors corresponding to each variable are split into parent and child embeddings. Causal edge probabilities are then predicted via a sigmoid over dot products:

$\hat{s}_{ij} = \sigma(V_i^\top U_j), \quad \hat{A} = [\hat{s}_{ij}] \in [0,1]^{f \times f}$

Binary cross-entropy loss is optimized over all ordered pairs, and acyclicity is enforced via an augmented Lagrangian penalty on the spectral radius (Swelam et al., 10 Nov 2025).

For treatment effect estimation and prescriptive analytics (e.g., PriMa-Causa (Saretzky et al., 30 Nov 2025)), a query encoding $\mathrm{do}(T = t')$ is appended to a context window, and multiple forward passes under different interventions compute conditional average treatment effects (CATE):

$\tau(x) = \mathbb{E}[Y \mid \mathrm{do}(T=1), X = x] - \mathbb{E}[Y \mid \mathrm{do}(T=0), X = x]$

This process enables "what-if" simulation and action ranking without the need to explicitly fit a new SCM on each dataset.

4. Evaluation, Benchmarking, and Empirical Performance

Causal foundation models are evaluated against a spectrum of baselines: direct gradient-based causal discovery architectures (e.g., AVICI), classical constraint- or score-based algorithms (e.g., GIES, IGSP), and differentiable causal inference pipelines (e.g., DCDI). Core metrics include ROC-AUC and average precision (AP) for edge detection, structural Hamming distance (SHD), mean absolute error (MAE) for effect estimation, and precision-in-estimation-of-heterogeneous-effects (PEHE) (Swelam et al., 10 Nov 2025, Saretzky et al., 30 Nov 2025, Balazadeh et al., 9 Jun 2025).

Key findings include:

TabPFN+adapter achieves ROC-AUC of 0.88–0.90, on par with AVICI (0.90) and significantly outperforming GIES/IGSP (0.75–0.80). AP degrades as variable count increases but always outpaces classical methods.
Layerwise probing reveals causal information is most accessible in intermediate layers, with earlier layers lacking abstraction and later layers over-specialized for predictive tasks.
For treatment effect estimation (PriMa-Causa), mean PEHE is 0.696 vs 0.852 for the best S-learner baseline, demonstrating superior alignment of predicted and true conditional effects (Saretzky et al., 30 Nov 2025).
In evaluation on real-world and synthetic data, these models match or surpass state-of-the-art dedicated causal discovery and inference methods across heterogeneous tasks, with little or no fine-tuning required (Balazadeh et al., 9 Jun 2025, Saretzky et al., 30 Nov 2025).

5. Generalization, Adaptability, and Limitations

The key power of pre-trained causal foundation models lies in zero-shot generalization and adaptation without retraining. By pre-training on large and diverse priors over SCMs—including interventions and multiple topology types—these models learn causal representations and inference procedures that are robust to moderate shifts in distribution, noise, and even limited sample sizes. This enables their deployment in tabular, time-series, and even interventional policy contexts.

Foundational models can be "probed" via lightweight adapters for causal structure recovery and applied directly to observational (possibly confounded) data for effect estimation. However, several limitations persist:

The ability to discover causal structure is bounded by the expressivity of the SCM prior; mechanisms beyond those seen in pre-training can degrade performance.
Identifiability is not guaranteed when confronted with unobserved confounding, multiple-valued or endogenous protected attributes, or nonstationary mechanisms (Robertson et al., 8 Jun 2025, Swelam et al., 10 Nov 2025).
Effect estimation is typically limited to binary treatments/interventions or simple CATE queries; nuanced path-specific or joint continuous interventions require methodological extensions.

Best practices include pre-training on highly diverse SCM families, careful balance of observational and interventional data, and, where appropriate, architectural choices that support modular probing and adaptation (Swelam et al., 10 Nov 2025, Robertson et al., 8 Jun 2025).

6. Broader Implications and Future Directions

Recent advances demonstrate that pre-trained causal foundation models can provide a universal, amortized, and highly efficient engine for causal discovery, effect estimation, and interpretability across domains. The emergence of adapter-style probing, the identification of mid-layer causal representations, and strong empirical performance open avenues for:

General-purpose, zero-shot deployable causal discovery tools across tabular, time-series, and structured domains (Swelam et al., 10 Nov 2025, Yin et al., 23 Jun 2025).
High-fidelity, explainable root-cause identification and intervention ranking in operational settings (e.g., prescriptive maintenance for OEE in production lines) (Saretzky et al., 30 Nov 2025).
Modular architectures that facilitate fairness, latent structure disentanglement, or domain adaptation with minimal supervision or retraining (Robertson et al., 8 Jun 2025, Audenaert et al., 7 Jul 2025).
Theoretical progress toward unifying predictive and causal representation in deep learning, with attention to the identifiability, sample efficiency, and robustness properties afforded by massive synthetic SCM pre-training (Yin et al., 23 Jun 2025, Swelam et al., 10 Nov 2025).

Limitations such as reliance on graph exogeneity, challenges in cross-domain generalization, and scalability to complex interventions persist. Proposed research directions include learning richer priors encompassing multiple, continuous, or path-specific interventions; developing targeted adapters for fairness and counterfactual recourse; and scaling architectures to hundreds of variables and tasks.

References:

"Does TabPFN Understand Causal Structures?" (Swelam et al., 10 Nov 2025)
"Integrating Causal Foundation Model in Prescriptive Maintenance Framework for Optimizing Production Line OEE" (Saretzky et al., 30 Nov 2025)
"FairPFN: A Tabular Foundation Model for Causal Fairness" (Robertson et al., 8 Jun 2025)
"Learning Causal Graphs at Scale: A Foundation Model Approach" (Yin et al., 23 Jun 2025)