---
title: Causality-Aware Transformer (CAT)
url: https://www.emergentmind.com/topics/causality-aware-transformer-cat
type: topic
---

# Causality-Aware Transformer (CAT)

Searching arXiv for the specified CAT paper and closely related transformer-based temporal causality work.
Causality-Aware Transformer (CAT) most commonly denotes a transformer-based framework for temporal causal discovery and inference that couples a multi-layer time-series forecaster with post-training gradient analysis and an attention-masking mechanism for prior knowledge integration. In the formulation introduced in "Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration," CAT is designed to address two stated difficulties in temporal causal analysis: complex nonlinear dependencies and spurious correlations. Its workflow is to train a forecaster on historical multivariate data, extract a directed causal graph and associated lags from the trained model through gradient-based sensitivity analysis, and optionally constrain attention with user-excluded links so that forbidden causal paths are suppressed throughout the network [2508.15928].

## 1. Conceptual definition and problem setting

CAT is a temporal causal discovery model built around a forecasting-first paradigm. Rather than estimating a causal graph directly, it first trains a multi-layer Transformer-based time-series forecaster to model long-range, nonlinear temporal relationships among variables, and then interprets the trained predictor by measuring how perturbations in lagged inputs affect target forecasts [2508.15928].

The underlying causal object is a directed graph with lag annotations. In CAT, a directed edge \(X_i \to Y_j\) is declared when the normalized gradient sensitivity of the forecast for \(Y_j\) to some lagged value of \(X_i\) exceeds a user-defined threshold \(\theta\). The corresponding causal lag is the \(\arg\max\) over lagged gradient magnitudes. This makes CAT a representation-learning approach to temporal causality in which the causal graph is reconstructed from the behavior of a trained predictor rather than from explicit structural constraints in the training objective alone [2508.15928].

Within the broader transformer-for-causality literature, this places CAT alongside forecasting-based discovery models such as CausalFormer, which interprets a causality-aware transformer using regression relevance propagation [2406.16708], and sparse-attention Granger-style models such as the Sparse Attention Transformer, also referred to as CAT in the supplied material, which recovers causal influence by masking variables and comparing residual variances [2411.13264]. A related but more general claim appears in "Transformer Is Inherently a Causal Learner," where decoder-only transformers trained autoregressively are shown to encode lagged causal structure in gradient sensitivities under assumptions A1-A4 [2601.05647].

## 2. Transformer architecture

CAT’s core forecaster tokenizes both static variables and temporal variables. Given \(N\) source time-series variables \(X=(X_1,\dots,X_N)\), each of length \(S\), each series is normalized and split into \(T\) patches of length \(P\). Each patch is linearly projected into a \(D\)-dimensional embedding, while static variables are projected to a single \(D\)-dimensional vector. Standard sinusoidal positional encoding is then added so that temporal order is retained in the token sequence [2508.15928].

The encoder is organized as \(L\) stacked Transformer encoder levels, each containing \(K\) identical Transformer blocks. If \(Z^{\ell-1}\) denotes the input to level \(\ell\), a block computes multi-head self-attention,
\[
Q = Z^{\ell-1}W^Q_{(\ell,b)},\quad
K = Z^{\ell-1}W^K_{(\ell,b)},\quad
V = Z^{\ell-1}W^V_{(\ell,b)},
\]
followed by residual connection, layer normalization, a position-wise feed-forward network,
\[
\mathrm{FFN}(\tilde Z) = \mathrm{ReLU}(\tilde ZW_1 + b_1)\,W_2 + b_2,
\]
and a second residual-plus-normalization step. For attention head \(m\), the attention weight from token \(j\) to token \(i\) is
\[
\alpha_{ij}^{(\ell,b,m)} =
\frac{\exp\bigl((q_iW^Q_m)(k_jW^K_m)^T/\sqrt{d_k}\bigr)}
{\sum_{j'}\exp\bigl((q_iW^Q_m)(k_{j'}W^K_m)^T/\sqrt{d_k}\bigr)}.
\]
After \(L\) levels, each time-series variable is reduced to a single token by a temporal “strided convolution” procedure such that \(T^L=1\) [2508.15928].

The output stage linearly projects the final token for each target variable \(Y_j\) to the forecast window \(\hat Y_j\). Numerical targets use L1 loss, and categorical targets use cross-entropy. This choice keeps the causal extraction stage decoupled from target type: the transformer is optimized as a forecaster, and causal interpretation is imposed after training through sensitivity analysis rather than through a bespoke graph-structured decoder [2508.15928].

## 3. Gradient-based causal graph and lag recovery

After forecaster training, CAT extracts causal relations by finite-difference approximation of partial derivatives. Let \(\mathbf F_j(X)\) denote the forecast for target \(Y_j\). For each source variable \(X_i\) and lag \(\tau\in\{0,\dots,S-1\}\),
\[
g_{j,i}(\tau)\approx
\frac{\mathbf F_j(\dots,X_i(t-\tau)+\epsilon,\dots)-\mathbf F_j(\dots,X_i(t-\tau)-\epsilon,\dots)}{2\epsilon}.
\]
These values are stacked into a gradient-score matrix \(G_j\in\mathbb{R}^{N\times S}\) with entries \(G_j[i,\tau]=g_{j,i}(\tau)\) [2508.15928].

To make scores comparable across variables and lags, CAT normalizes by the global maximum:
\[
S_j[i,\tau]=
\frac{|G_j[i,\tau]|}
{\max_{i',\tau'}|G_j[i',\tau']|}.
\]
A directed edge \(X_i\to Y_j\) is declared when
\[
\max_{\tau} S_j[i,\tau]\ge \theta,
\]
and the inferred lag is the \(\arg\max\) of \(|G_j[i,\tau]|\) [2508.15928].

This extraction mechanism is closely aligned with a broader methodological shift in transformer-based causal discovery from raw attention inspection to differential or attributional criteria. SAT interprets masked attention and prediction-error changes as a Granger-causality analogue [2411.13264]. CausalFormer argues that analyzing attention weights alone is an incomplete mapping from model parameters to causality and instead propagates relevance through the whole network [2406.16708]. "Transformer Is Inherently a Causal Learner" goes further by establishing a score-gradient energy criterion,
\[
H_j=\mathbb E[s_j(Y,X)^2],
\]
with \(H_j>0\Longleftrightarrow j\in\operatorname{Pa}(i,t)\) under assumptions A1-A4, and then estimates a practical surrogate using aggregated relevance scores [2601.05647]. Taken together, these works indicate that CAT’s gradient-based graph recovery belongs to a technically coherent line of research in which causal structure is inferred from input-output sensitivities of trained transformers.

## 4. Prior knowledge integration and spurious-correlation control

A distinctive feature of CAT is the integration of user-specified excluded links directly into multi-head attention. Let \(A\in\{0,1\}^{(N+M)\times(N+M)}\) be a binary adjacency-mask matrix over variables, where \(A[p,q]=0\) means that the link “variable \(q\not\to\) variable \(p\)” has been excluded by prior knowledge. CAT expands this variable-level matrix into a token-level mask \(M^{(\ell)}\in\{0,1\}^{T_\ell\times T_\ell}\) by replicating rows and columns according to the number of tokens per variable in layer \(\ell\) [2508.15928].

The standard attention score is then replaced by a masked score,
\[
\widetilde A^{(\ell)}
=
\mathrm{softmax}\!\Bigl(\bigl(QK^T+\lambda\,M^{(\ell)}\bigr)/\sqrt{d_k}\Bigr),
\]
with \(\lambda\to -\infty\) for prohibited entries so that forbidden attention weights become zero. The mask is applied consistently across all heads and blocks in a level. Because source and target variables are kept in distinct token groups, the paper states that prohibited edges never slip through via multi-layer attention propagation [2508.15928].

The stated purpose of this mechanism is to mitigate spurious causal relationships. CAT couples hard attention exclusion with standard weight decay on all projection matrices and a relatively high gradient threshold \(\theta\), with \(\theta=0.15\) given as an example, to prune weak, noise-driven sensitivities [2508.15928]. This architecture-level masking differs from related approaches that also encode causal constraints in attention but target different tasks. "CaTs and DAGs" masks cross-attention according to a known DAG so that predictions remain causally compliant at inference time [2410.14485], while the DAG-aware Transformer for causal effect estimation masks self-attention using ancestor reachability in a known causal graph [2410.10044]. CAT instead uses exclusion masks within a temporal discovery pipeline, where the graph is still being inferred from data but some edges may be ruled out a priori.

## 5. Empirical evaluation

CAT was benchmarked against six state-of-the-art baselines: cMLP, cLSTM, TCDF, DVGNN, CUTS, and CausalFormer. The evaluation used three dataset families: Basic Synthetic with “fork,” “V-structure,” “mediator,” and “diamond” motifs and 1,000-point series; Lorenz96 nonlinear climate with 10 variables and 1,000 points; and NetSim fMRI network simulations with node counts 5–50 and lengths 50–5,000. Performance was measured using F1-score on recovered edges and Precision of Delay (PoD), defined as the fraction of edges with correctly estimated lag within one time-step [2508.15928].

The reported aggregate result is a 12.8% absolute improvement in F1 over the next best method, together with 98.9% accuracy in estimating causal lags. A specific example given in the paper is Lorenz96, where CAT attains 0.84 F1 versus 0.69 by CausalFormer. Ablation studies further report that removing the prior-masking mechanism drops F1 by 6–8 points, while replacing the multi-layer stack with a single layer, as in CausalFormer, loses an additional 4–5 points [2508.15928].

These results situate CAT within a competitive transformer-based discovery landscape. CausalFormer reports state-of-the-art performance on synthetic, simulated, and real datasets using a causality-aware transformer plus a decomposition-based causality detector [2406.16708]. SAT reports AUC-ROC of 0.70–0.78 and F1-score of 0.63–0.72 across synthetic benchmark regimes, compared with 0.47–0.67 AUC and 0.51–0.66 F1 for VAR-based Granger causality [2411.13264]. "Transformer Is Inherently a Causal Learner" reports that its transformer-based discovery method maintains \(\mathrm{F1}\approx 0.9\) at \(p=50\), preserves \(\mathrm{F1}>0.9\) up to \(L=50\), and achieves AUROC/AUPRC of \((0.87/0.70)\), \((0.70/0.57)\), and \((0.72/0.65)\) on AQI, Traffic, and Medical, ranking top-3 among 16 methods on the CausalTime benchmark [2601.05647]. A plausible implication is that CAT’s strongest empirical distinction within this cluster is not merely the use of transformers for causal recovery, but the combination of gradient-based lag extraction with an explicit prior-knowledge masking interface.

## 6. Terminology, related usages, and interpretive issues

The acronym “CAT” is not unique in the literature. In addition to the temporal causal discovery framework described above [2508.15928], it has been used for Causal Attention Tuning in large language models [2509.01535], CAT-XPLAIN for inherently explainable vision transformers [2206.14841], and Causality-Aware Transformer Networks for robotic navigation [2409.02669]. In adjacent work, “CaT” also denotes Causal Transformers constrained by a predefined DAG [2410.14485]. For precise interpretation, the acronym therefore has to be resolved by task domain and paper context.

A recurring interpretive issue across these works is whether transformer attention itself should be read as causality. SAT is explicitly built on the intuition that self-attention scores can be re-interpreted as measures of causal influence [2411.13264]. CausalFormer takes the opposite methodological stance that parameters such as attention weights and convolution weights alone are insufficient, because other components such as fully connected layers and activation functions also matter for causal discovery [2406.16708]. The 2026 identifiability result for autoregressive transformers reframes the question again: under conditional exogeneity, no instantaneous effects, lag-window coverage, and faithfulness, nonzero score-gradient energy rather than raw attention is what exactly recovers lagged parents [2601.05647]. CAT, as defined in [2508.15928], is methodologically closer to this latter view, since its graph extraction is based on gradient sensitivity and thresholding, while attention masking is used primarily for prior knowledge integration and spurious-link suppression.

This distribution of viewpoints suggests that “causality-aware” is best understood as a family resemblance rather than a single architectural recipe. In the supplied literature, the designation covers at least four distinct strategies: forecasting-plus-gradient extraction [2508.15928], sparse attention with Granger-style masking [2411.13264], whole-model relevance decomposition [2406.16708], and explicit attention constraints from known causal graphs [2410.14485; 2410.10044]. What unifies them is the attempt to move beyond correlation-only sequence modeling by embedding temporal priority, structural constraints, or attributional identifiability into transformer computation.

Source: https://www.emergentmind.com/topics/causality-aware-transformer-cat