CausalFM: Causal Foundation Models

• CausalFM is a framework that integrates structural causal models with foundation-model training to enable interventional prediction via synthetic SCM priors.
• The PFN formulation allows pretrained transformers to approximate posterior predictive interventional distributions, supporting back-door, front-door, and IV causal queries.
• Applications of CausalFM span static tabular data, temporal dynamics, and multimodal generative pipelines, enhancing treatment effect estimation and counterfactual generation.

CausalFM denotes a line of research that integrates structural causal modeling with foundation-model training or adaptation. In the narrow and formal sense introduced for tabular causal inference, CausalFM is a prior-data fitted network (PFN) framework in which a transformer is pretrained on synthetic datasets generated from Bayesian priors over structural causal models (SCMs), and then used in-context to approximate posterior predictive interventional distributions and causal queries such as conditional average treatment effects (CATEs) without retraining on the target dataset (Ma et al., 12 Jun 2025). In later work, the term is also used more broadly for causality-aware foundation-model systems, including SCM-based data augmentation for tabular foundation models, temporal PFN extensions, and modular multimodal causal generative pipelines (Bühler et al., 7 Jan 2026, Thumm et al., 11 Mar 2026, Komanduri et al., 22 May 2026, Talupula et al., 18 Jun 2026).

1. Terminology, scope, and historical positioning

In the PFN literature, CausalFM is positioned as a causal analogue of tabular foundation models such as TabPFN: a single pretrained model is intended to amortize inference across many downstream causal tasks by learning from a synthetic prior over SCMs rather than from one empirical dataset at a time (Ma et al., 12 Jun 2025). The framework is explicitly formulated for identifiable causal settings, and its initial empirical instantiation targets back-door CATE estimation, while the formal construction also covers front-door and instrumental-variable settings (Ma et al., 12 Jun 2025).

A later temporal foundation-model paper characterizes CausalFM as the static, cross-sectional ancestor of a temporal extension. In that characterization, CausalFM operates on i.i.d. units with covariates $X$, treatment $T$, and a single outcome $Y$, and its primary query is

$\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$

That paper further states that CausalFM extends earlier causal PFNs to front-door adjustment and instrumental-variable settings via structured SCM priors, but remains a treatment-effect estimator that outputs a point estimate $\hat{\tau}(x)$ rather than explicit reliability signals (Talupula et al., 18 Jun 2026).

The surrounding literature has broadened the label. In low-data tabular adaptation, “CausalFM” is used conceptually for approaches that inject causal structure into the fine-tuning process through SCM-generated synthetic data rather than through a causal foundation-model architecture itself (Bühler et al., 7 Jan 2026). In multimodal generation, it is used even more expansively for modular systems in which pretrained vision-language and diffusion models are assembled into a causal pipeline for concept discovery, intervention, and counterfactual generation (Komanduri et al., 22 May 2026). This suggests that CausalFM has become an umbrella term for methods that make foundation models causally structured, causally supervised, or causally constrained.

2. PFN formulation and Bayesian interventional inference

The formal CausalFM framework starts from a causal inference setting

$\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$

where $Z$ is the observed variable set, $(P_{\mathrm{obs}},P_{\mathrm{int}})$ is an observational/interventional pair compatible with the assumed causal regime, and $Q(P_{\mathrm{int}})$ is an identifiable causal query (Ma et al., 12 Jun 2025). Instead of placing a prior directly on observational distributions, CausalFM defines a prior $\Pi(\mathcal S)$ over SCMs $T$0. Each SCM induces an observational distribution $T$1 and an interventional distribution $T$2, allowing synthetic generation of both context datasets and causal labels (Ma et al., 12 Jun 2025).

The PFN is trained to approximate a conditional posterior predictive interventional distribution,

$T$3

with observational data $T$4 appearing only in the input context and not through any per-dataset optimization (Ma et al., 12 Jun 2025). In the standard PFN abstraction, one samples $T$5, draws a dataset $T$6, and trains a transformer $T$7 to minimize expected negative log-likelihood so that

$T$8

CausalFM retains this amortized Bayesian structure, but replaces purely predictive targets with interventional ones (Ma et al., 12 Jun 2025).

For temporal causal foundation models, the same PFN logic is extended from static SCMs to temporal structural causal models (TSCMs). CausalTimePrior defines a prior $T$9 over TSCMs and generates paired observational and interventional time series so that a PFN can answer intervention queries on held-out dynamic systems by in-context learning rather than by per-dataset fitting (Thumm et al., 11 Mar 2026). A small proof-of-concept GRU PFN trained on 100K TSCMs achieved RMSE $Y$0, MAE $Y$1, and effect size correlation $Y$2, competitive with per-dataset VAR-OLS and markedly better than mean prediction, which had RMSE $Y$3 and effect size correlation $Y$4 (Thumm et al., 11 Mar 2026).

3. SCM priors, identification regimes, and structural assumptions

A central claim of CausalFM is that causal foundation models should be trained from SCM priors that are compatible with the target identification regime and well-specified for the query of interest (Ma et al., 12 Jun 2025). A prior is well-specified if evaluating the causal query on the posterior predictive interventional distribution yields a consistent Bayesian estimator as sample size grows. The framework therefore rejects indiscriminate mixing of identifiable and non-identifiable regimes within a single prior when the target query requires concentration of the posterior over a well-defined causal parameter (Ma et al., 12 Jun 2025).

To construct such priors at scale, CausalFM uses causality-inspired Bayesian neural networks (BNNs) organized through cluster-level DAGs, or C-DAGs. At the cluster level, parent–child relations specify which variables or latent groups feed into each structural component; at the function level, BNNs sample nonlinear mechanisms for covariates, treatments, mediators, instruments, and outcomes (Ma et al., 12 Jun 2025). This allows generation of high-dimensional mixed-type tabular data while preserving the causal semantics of back-door, front-door, or IV adjustment.

The initial CausalFM formulation makes the identification regimes explicit. In the back-door case, with $Y$5 and binary treatment $Y$6, the estimand is

$Y$7

In the front-door case, with $Y$8, the framework targets a mediated CATE under the front-door criterion. In the IV case, with $Y$9, the target is a conditional average potential outcome $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$0 identified through an instrumental-variable structure and completeness assumptions (Ma et al., 12 Jun 2025). A later temporal paper reuses these regimes almost verbatim as “CausalFM-style identification priors,” comprising back-door, front-door, and instrumental-variable settings, inside a richer temporal prior over dynamic regimes (Talupula et al., 18 Jun 2026).

The wider structural-causal background is older than PFNs. Identifiable functional model classes show that, under appropriate functional-form and independent-noise assumptions, one can identify the complete causal DAG from the joint distribution rather than only a Markov equivalence class (Peters et al., 2012). Although CausalFM does not rely on that theorem directly, it shares the broader premise that structural assumptions are not auxiliary: they are the source of identifiability.

4. Tabular adaptation and data-level causal regularization

A data-centric interpretation of CausalFM appears in SCM-based fine-tuning of tabular foundation models under data scarcity. In this setting, the base model is a tabular foundation model such as Mitra, and the causal component is introduced through synthetic data augmentation rather than through PFN-style pretraining on SCM priors (Bühler et al., 7 Jan 2026). The method CausalMixFT fits an SCM on the target dataset, samples structurally consistent synthetic rows, mixes them with the scarce real data during fine-tuning, and validates only on real data (Bühler et al., 7 Jan 2026).

The downstream dataset is written as

$\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$1

and the synthetic dataset as

$\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$2

Fine-tuning uses the mixed objective

$\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$3

with default $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$4, balanced mini-batches, cross-entropy loss, validation on real data only, and patience $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$5 for early stopping (Bühler et al., 7 Jan 2026).

The SCM itself is obtained by repeated causal discovery and graph aggregation. PC and FCI are each run 50 times, producing 100 adjacency estimates; these are aggregated into a probabilistic adjacency matrix $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$6, from which a DAG is sampled, made acyclic if necessary, and then fitted with DoWhy-GCM additive noise models. Numerical nodes use regressors, categorical nodes use classifiers, and synthetic generation proceeds in topological order (Bühler et al., 7 Jan 2026). The authors explicitly treat these graphs as correlational or structural rather than making strong causal claims, which is an important qualifier (Bühler et al., 7 Jan 2026).

Empirically, on 33 TabArena classification datasets and over 2300 fine-tuning runs, CausalMixFT improved median normalized ROC-AUC from $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$7 for standard fine-tuning to $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$8, while CTGAN, TabEBM, and TableAugment yielded medians of $\tau(x) = \mathbb{E}[Y(1)-Y(0)\mid X=x].$9, $\hat{\tau}(x)$0, and $\hat{\tau}(x)$1, respectively (Bühler et al., 7 Jan 2026). It also narrowed the median validation–test performance gap from $\hat{\tau}(x)$2 to $\hat{\tau}(x)$3, making validation-based early stopping substantially more reliable (Bühler et al., 7 Jan 2026). A plausible implication is that, in this usage, “CausalFM” refers less to a specific pretrained causal transformer than to causal regularization of the adaptation distribution.

5. Temporal and panel-data generalizations

The most explicit temporal generalization treats static CausalFM as insufficient for panel data with temporal dynamics, time-varying treatments, irregular sampling, and streaming data (Talupula et al., 18 Jun 2026). TCPFN extends the static causal-PFN paradigm into temporal panel data by introducing discrete-token representations of unit–time pairs, horizon-safe cross-attention masking, a mixed temporal prior, and a Causal Judgment Head (Talupula et al., 18 Jun 2026). The temporal CATE estimand is

$\hat{\tau}(x)$4

TCPFN’s mixed prior includes six causal regimes—independent, direct, confounded, mediated, time-varying confounded, and feedback—together with CausalFM-style back-door, front-door, and IV priors (Talupula et al., 18 Jun 2026). Its Causal Judgment Head outputs null-effect probability, confounding strength, identifiability probability, mediation fraction, and causal regime (Talupula et al., 18 Jun 2026). The paper emphasizes that $\hat{\tau}(x)$5 is a learned proxy rather than a formal guarantee, because identifiability is a property of the causal model and assumptions, not something determinable from data patterns alone (Talupula et al., 18 Jun 2026).

On 19 benchmark datasets across five domains, TCPFN achieved AUROC $\hat{\tau}(x)$6 on Tennessee Eastman, $\hat{\tau}(x)$7 on SWaT, $\hat{\tau}(x)$8 on Causal Rivers, and $\hat{\tau}(x)$9 on CAUSRCA; its null detector reached NullF1 $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$0 and AUROC $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$1 (Talupula et al., 18 Jun 2026). The model also scaled to $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$2 variables on a proprietary Kraft pulp-and-paper dataset in 6 hours on a single GPU, while PCMCI on a $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$3 sub-panel took 81.5 hours on CPU, extrapolating by $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$4 to approximately 12.5 days at $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$5 (Talupula et al., 18 Jun 2026).

CausalTimePrior supplies the synthetic training infrastructure for temporal causal PFNs at the prior level. It samples time-lagged DAGs with $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$6, $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$7, $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$8, $\mathcal C = \bigl(Z,\;\mathcal P_{\mathrm{obs}\times\mathcal P_{\mathrm{int}},\;Q\bigr),$9, sparse edge probabilities $Z$0, lag-decay $Z$1, nonlinear mechanisms drawn from $Z$2, and intervention types that are approximately 50% hard, 30% soft, and 20% time-varying (Thumm et al., 11 Mar 2026). It also supports regime-switching TSCMs. This establishes a direct pathway from static CausalFM to time-series causal foundation models (Thumm et al., 11 Mar 2026).

6. Multimodal reinterpretations, adjacent usages, and limitations

Outside tabular and temporal effect estimation, foundation-model-powered causal generative modeling provides a more modular interpretation of CausalFM. FM-CGM formalizes an end-to-end visual causal pipeline as

$Z$3

where $Z$4 is a semantic SCM over concepts, $Z$5 is a concept extractor, $Z$6 is a concept manipulator, and $Z$7 is a counterfactual generator (Komanduri et al., 22 May 2026). The framework uses a reasoning vision-LLM for concept extraction and intervention reasoning and Stable Diffusion XL for image generation, while Causal Semantic Guidance applies cross-attention-based masking so that interventions propagate to descendants and preserve non-descendants (Komanduri et al., 22 May 2026). On 75 images each from CelebA-HQ and MS-COCO, CSG achieved average VLM-Eff $Z$8 and average LPIPS $Z$9, compared with $(P_{\mathrm{obs}},P_{\mathrm{int}})$0 for DDIM inversion and $(P_{\mathrm{obs}},P_{\mathrm{int}})$1 for DDPM inversion (Komanduri et al., 22 May 2026). In this setting, the term “CausalFM” no longer denotes a PFN; it denotes a foundation-model system whose modules collectively implement discovery, intervention, and counterfactual generation.

The label also appears in adjacent literatures where “FM” means factor models rather than foundation models. A panel-data paper develops a factor-model framework for causal inference in which treatment effects are represented as structural changes in treated units’ exposure to latent common shocks, without imposing parallel trends, and targets the systematic component

$(P_{\mathrm{obs}},P_{\mathrm{int}})$2

rather than the raw unit-time realized effect (Bai et al., 29 Jun 2026). A financial time-series paper similarly describes a Time-series Neural Causal Model VAE with a DAG-constrained decoder and causal Wasserstein training as a step toward genuinely causal financial models, using “CausalFM” in the sense of causal market simulators rather than PFN-based foundation models (Thumm et al., 6 Nov 2025). These usages are related by emphasis on structural interventions and counterfactuals, but they are not architecturally equivalent.

Several limitations recur across the literature. Static CausalFM models are restricted to cross-sectional settings and do not handle trajectories, time-varying treatments, or irregular sampling unless extended (Talupula et al., 18 Jun 2026). SCM priors and discovery modules can be misspecified; some systems explicitly treat learned graphs as structural approximations rather than strong causal claims (Bühler et al., 7 Jan 2026). Learned reliability scores, such as identifiability probabilities, are heuristic proxies rather than proofs (Talupula et al., 18 Jun 2026). In multimodal generation, causal graphs inferred by a vision-LLM can be wrong or biased, and semantic SCMs usually omit explicit numeric mechanisms $(P_{\mathrm{obs}},P_{\mathrm{int}})$3, which limits formal causal guarantees (Komanduri et al., 22 May 2026). More generally, the effectiveness of any CausalFM variant depends on the match between its causal prior, its target identification regime, and the data regime in which it is deployed (Ma et al., 12 Jun 2025).

A common misconception is that CausalFM names a single architecture. The literature instead supports a more precise interpretation: CausalFM is a family of causally structured foundation-model paradigms whose shared core is the use of SCMs, identification-aware priors, or causal constraints to make pretrained models perform interventional prediction, effect estimation, counterfactual generation, or robust adaptation under distributional and sample-size limitations (Ma et al., 12 Jun 2025, Bühler et al., 7 Jan 2026, Thumm et al., 11 Mar 2026, Komanduri et al., 22 May 2026, Talupula et al., 18 Jun 2026).