Causally-Guided Pairwise Transformer (CGPT)

• CGPT is a Transformer architecture that integrates a known causal graph to guide pairwise modeling for multivariate time-series forecasting.
• It decomposes data into directed target-context pairs and uses shared, channel-agnostic modules, ensuring adaptability without scaling with the channel number.
• Empirical results show strong long-term forecasting performance in industrial digital twin applications by effectively balancing channel-dependent and channel-independent modeling.

The Causally-Guided Pairwise Transformer (CGPT) is a Transformer architecture for industrial multivariate time-series forecasting that integrates a known causal graph as an inductive bias and resolves the channel-dependent (CD) versus channel-independent (CI) trade-off through a pairwise modeling paradigm (Mayr et al., 18 Aug 2025). In this formulation, directed target-context pairs aligned with a causal graph are processed by shared, channel-agnostic modules whose parameter dimensions are independent of the number of variables, so that CGPT is CD at the pair level and CI-like across pairs (Mayr et al., 18 Aug 2025). The resulting architecture is designed for “any-variate” adaptability, multivariate-to-univariate forecasting, and digital twin settings in which explicit cross-variable interactions, lagged effects, and changing sensor configurations are all operationally relevant (Mayr et al., 18 Aug 2025).

1. Conceptual basis and problem setting

CGPT is motivated by a stated tension in industrial multivariate time-series modeling. Channel-Dependent models learn a single function over all variables jointly and can capture explicit cross-variable interactions, multivariate lags, and non-linearities, but their parameterization typically scales with the number of channels $C$, and any change in the sensor set or task forces architectural reconfiguration or retraining (Mayr et al., 18 Aug 2025). Channel-Independent models learn a shared function applied to each variable individually and therefore generalize well across datasets and are robust to noise and distribution shifts, but they are structurally blind to explicit cross-channel dependencies that matter in physically coupled systems (Mayr et al., 18 Aug 2025).

CGPT addresses this by decomposing multidimensional data into directed target-context pairs $(i \to j)$ aligned with a causal graph $G=(V,E)$, where each node corresponds to a channel and an edge $(i \to j)\in E$ indicates that channel $i$ is a potential causal parent of channel $j$ (Mayr et al., 18 Aug 2025). For a target $j$, the parent set is defined as

$\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$

and pair formation is restricted to these edges, which is central to both computational tractability and inductive bias (Mayr et al., 18 Aug 2025).

The forecasting problem is posed on a multivariate time series $X\in\mathbb{R}^{T\times C}$, optionally with exogenous covariates $U\in\mathbb{R}^{T\times D_U}$ (Mayr et al., 18 Aug 2025). With context length $(i \to j)$0 and prediction horizon $(i \to j)$1, the model maps a history to future targets as

$(i \to j)$2

where $(i \to j)$3 is the set of predicted channels (Mayr et al., 18 Aug 2025). The reported use case emphasizes multivariate-to-univariate tasks because of computational feasibility and prevalence in industrial digital twin applications (Mayr et al., 18 Aug 2025).

A central design claim is that the model enforces CD information flow at the pair level and CI-like generalization across pairs, yielding an architecture whose learned modules do not scale with $(i \to j)$4 (Mayr et al., 18 Aug 2025). This is the technical sense in which CGPT is presented as an “any-variate” model.

2. Architecture and mathematical formulation

CGPT operates in three stages: channel-independent temporal encoding, pairwise interaction modeling, and target-wise aggregation and prediction (Mayr et al., 18 Aug 2025).

In Stage 1, each channel $(i \to j)$5 is processed independently using shared modules. RevIN normalization is applied per series to mitigate distribution shift, then patch tokenization segments $(i \to j)$6 into $(i \to j)$7 non-overlapping patches of length $(i \to j)$8 with stride $(i \to j)$9, typically $G=(V,E)$0 (Mayr et al., 18 Aug 2025). A patch embedding layer produces token representations $G=(V,E)$1, and a shared Transformer encoder with $G=(V,E)$2, heads $G=(V,E)$3, $G=(V,E)$4, and FFN $G=(V,E)$5 yields encoded channel tokens $G=(V,E)$6 (Mayr et al., 18 Aug 2025). The encoding step is written as

$G=(V,E)$7

The self-attention inside this encoder uses standard scaled dot-product attention,

$G=(V,E)$8

with $G=(V,E)$9, $(i \to j)\in E$0, and $(i \to j)\in E$1 (Mayr et al., 18 Aug 2025). The paper notes explicitly that, as implemented, the causal graph $(i \to j)\in E$2 is not used to mask attention inside this encoder; causal guidance enters instead through pair selection in the next stage (Mayr et al., 18 Aug 2025).

In Stage 2, the model constructs a pairwise influence representation for each directed edge $(i \to j)\in E$3. First, encoder outputs are pooled: $(i \to j)\in E$4 For each directed pair $(i \to j)\in E$5, CGPT computes

$(i \to j)\in E$6

where the same pair module is reused for all pairs (Mayr et al., 18 Aug 2025). This shared pair module is the locus of channel-dependent interaction modeling: the target and one parent are jointly processed to produce an influence vector (Mayr et al., 18 Aug 2025).

In Stage 3, influences from all parents of a target are aggregated additively. For target $(i \to j)\in E$7,

$(i \to j)\in E$8

and the prediction head maps this representation to the forecast,

$(i \to j)\in E$9

Training uses a regression objective, typically MSE, with MAE also reported: $i$0 The decomposition into base representation and influences is learned implicitly; the paper states that no explicit regularizers on influences are added in the reported model (Mayr et al., 18 Aug 2025).

This architecture has two notable properties. First, all learned modules are shared across channels and pairs, so the encoder, pair MLP, and prediction head remain channel-agnostic (Mayr et al., 18 Aug 2025). Second, causal structure is enforced not by masking token-token attention but by selecting which parent-target pairs are instantiated, a design distinction that separates CGPT from later graph-masked Transformer variants (Mayr et al., 18 Aug 2025).

3. Channel-agnostic parameterization, algorithms, and complexity

The computational argument for CGPT follows directly from its pairwise decomposition. Let $i$1 be the number of channels, $i$2 the number of directed edges, $i$3, $i$4 the number of patches, and $i$5 (Mayr et al., 18 Aug 2025). The CI encoder over all channels has per-layer complexity $i$6, the pairwise module over all edges has complexity $i$7, and aggregation plus prediction for target $i$8 costs $i$9 (Mayr et al., 18 Aug 2025). Naively modeling all pairs would scale as $j$0, whereas graph sparsity reduces this to $j$1 (Mayr et al., 18 Aug 2025).

The model pipeline is reported as follows. For each window in the dataloader, RevIN is applied per channel, patch embeddings are computed, the shared encoder produces $j$2, pooled channel vectors $j$3 are formed, pairwise influences are computed for each target and each parent in $j$4, the target representation $j$5 is aggregated, the head predicts $j$6, and optimization proceeds with MSE under AdamW and cosine annealing (Mayr et al., 18 Aug 2025). The reported hyperparameters are patch size/stride $j$7, $j$8, FFN $j$9, one encoder layer, one attention head, batch size $j$0, $j$1 epochs, early stopping patience $j$2, and learning rate $j$3 (Mayr et al., 18 Aug 2025).

The term “any-variate adaptability” is tied to the fact that parameterization is independent of $j$4: adapting to a new dataset or task requires changing the channel set and the graph $j$5, not the architecture (Mayr et al., 18 Aug 2025). A plausible implication is that the model is intended to support operational environments in which sensor availability and target definitions change over time, provided the new configuration can still be expressed in terms of directed parent-target pairs.

The same paper also emphasizes that CGPT is computationally most natural for multivariate-to-univariate forecasting, since unrestricted multivariate target prediction would multiply the number of instantiated pairs and thereby raise runtime and memory costs (Mayr et al., 18 Aug 2025). In this sense, the architecture’s scalability depends not only on channel count but also on causal sparsity.

4. Empirical evaluation and reported behavior

CGPT is evaluated on two synthetic datasets and four industrial datasets: Synthetic Additive, Synthetic Interactive, ETTh1, Multi-Stage Factory, Amino Emissions, and Refial (Mayr et al., 18 Aug 2025). Two task settings are reported: long-term forecasting with context $j$6 and horizon $j$7, and one-step forecasting with context $j$8 and horizon $j$9 (Mayr et al., 18 Aug 2025). Metrics are MSE and MAE, reported as mean $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$0 std over five runs (Mayr et al., 18 Aug 2025).

The baselines are DLinear as a CI model and an MLP baseline as a CD model, together with three CGPT variants: LeakyPairwise, StrictPairwise, and PureInfluence (Mayr et al., 18 Aug 2025). LeakyPairwise is the full CGPT, StrictPairwise restricts intra-module gradient flow by using a generic learnable embedding for the target’s influence path, and PureInfluence removes the target’s own history in the final prediction step (Mayr et al., 18 Aug 2025).

For long-term forecasting, the reported highlights include the following (Mayr et al., 18 Aug 2025):

• On Synthetic Additive, LeakyPairwise achieves MAE/MSE $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$1 with RevIN off, slightly better than DLinear $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$2 and comparable to MLP $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$3.
• On Synthetic Interactive, LeakyPairwise records $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$4, better than DLinear $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$5 and comparable to MLP $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$6.
• On ETTh1 with RevIN on, LeakyPairwise gives $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$7, while DLinear is slightly better on MSE at $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$8.
• On Multi-Stage Factory with RevIN on, LeakyPairwise yields $\mathrm{Pa}(j)=\{i \mid (i\to j)\in E\},$9, approximately matching DLinear $X\in\mathbb{R}^{T\times C}$0.
• On Refial, with RevIN on, LeakyPairwise achieves the best MSE at $X\in\mathbb{R}^{T\times C}$1 versus MLP $X\in\mathbb{R}^{T\times C}$2 and DLinear $X\in\mathbb{R}^{T\times C}$3.

The paper’s interpretation is that, in long-horizon tasks, pairwise modeling of causal drivers provides a consistent advantage over CI baselines, especially in industrial datasets with lagged, non-linear dynamics, while remaining competitive with end-to-end CD baselines (Mayr et al., 18 Aug 2025).

For one-step forecasting, the reported conclusion is different. On ETTh1 with RevIN on, DLinear dominates with MAE/MSE $X\in\mathbb{R}^{T\times C}$4, whereas LeakyPairwise records $X\in\mathbb{R}^{T\times C}$5 (Mayr et al., 18 Aug 2025). On Synthetic Interactive, DLinear is best at $X\in\mathbb{R}^{T\times C}$6, with LeakyPairwise around $X\in\mathbb{R}^{T\times C}$7, and on Refial DLinear is approximately $X\in\mathbb{R}^{T\times C}$8, compared with LeakyPairwise at approximately $X\in\mathbb{R}^{T\times C}$9 (Mayr et al., 18 Aug 2025). The reported conclusion is that, for one-step predictions, strong autoregression dominates and CI models excel (Mayr et al., 18 Aug 2025).

The ablation results are equally central to the definition of CGPT. Removing access to target history through PureInfluence causes severe degradation across datasets, which the paper presents as evidence that forecasting cannot be treated effectively as purely extrinsic regression (Mayr et al., 18 Aug 2025). By contrast, the difference between StrictPairwise and LeakyPairwise is marginal, suggesting that the primary performance driver is not fine-grained gradient routing inside the pair module but the presence of target history in the overall architecture (Mayr et al., 18 Aug 2025). The effect of causal guidance is discussed qualitatively: replacing the graph-guided pair set by fully connected pairs would inflate $U\in\mathbb{R}^{T\times D_U}$0, runtime, and the risk of negative transfer (Mayr et al., 18 Aug 2025).

5. Relation to causal guidance, pairwise masking, and adjacent Transformer families

Within the broader literature, CGPT occupies a specific point in the design space of causally informed Transformers. In CGPT as reported, causal guidance is used to select parent-target pairs, not to mask self-attention (Mayr et al., 18 Aug 2025). This differs from the DAG-aware Transformer for causal effect estimation, which encodes a user-specified DAG directly into the attention mechanism via a hard mask: $U\in\mathbb{R}^{T\times D_U}$1 and

$U\in\mathbb{R}^{T\times D_U}$2

thereby enforcing parent-only attention inside each head (Liu et al., 2024). That model is aimed at ATE and CATE estimation rather than time-series forecasting, but it formalizes a “pairwise and causally guided” attention mechanism in a way that is conceptually adjacent to CGPT (Liu et al., 2024).

A second neighboring line comes from autoregressive time-series transformers used for causal discovery. “Transformer Is Inherently a Causal Learner” proves, under assumptions A1–A4 and regularity, that the gradient sensitivities of transformer outputs with respect to lagged inputs recover direct lagged parents through the score gradient energy

$U\in\mathbb{R}^{T\times D_U}$3

iff the edge $U\in\mathbb{R}^{T\times D_U}$4 exists (Wang et al., 9 Jan 2026). That paper culminates in a concrete blueprint for a Causally-Guided Pairwise Transformer in which an extracted adjacency $U\in\mathbb{R}^{T\times D_U}$5 is used to restrict attention and regularize the forecast objective (Wang et al., 9 Jan 2026). This suggests a direct route from data-driven causal discovery to graph-guided pair construction, even though the reported CGPT paper assumes a known graph rather than learning one (Mayr et al., 18 Aug 2025, Wang et al., 9 Jan 2026).

A third related model is the Causally Guided Transformer (CGT) for multivariate time-series anomaly detection. CGT uses a time-lagged causal graph prior and a per-target hard parent mask $U\in\mathbb{R}^{T\times D_U}$6 built by PCMCI so that

$U\in\mathbb{R}^{T\times D_U}$7

which zeroes non-parent inputs before attention and learning (Khosravinia et al., 20 Apr 2026). The paper states explicitly that CGT is “essentially the same idea” as a Causally-Guided Pairwise Transformer in spirit and mechanism, although it implements causal guidance at the feature level for each target variable’s forecasting block rather than by creating variable-lag pair tokens and masking the attention matrix across them (Khosravinia et al., 20 Apr 2026). It also provides an explicit attention-mask interpretation for a stricter pairwise variant: $U\in\mathbb{R}^{T\times D_U}$8 with $U\in\mathbb{R}^{T\times D_U}$9 for allowed causal parent interactions and $(i \to j)$00 otherwise (Khosravinia et al., 20 Apr 2026).

Taken together, these adjacent papers clarify a useful taxonomy. CGPT, in the narrow sense of (Mayr et al., 18 Aug 2025), is a graph-guided pair-selection architecture with shared pair modules. DAG-aware Transformers and CGT instantiate harder forms of causal guidance through attention masks or feature masks (Liu et al., 2024, Khosravinia et al., 20 Apr 2026). The causal-discovery paper supplies a theoretical basis for deriving such masks from trained autoregressive Transformers rather than assuming them a priori (Wang et al., 9 Jan 2026). A plausible implication is that later “causally guided” Transformer variants can be read as architectural generalizations of the same underlying principle: causal structure restricts which pairwise interactions are permitted or emphasized.

6. Interpretability, digital twins, limitations, and future directions

CGPT is presented as a model for foundational digital twins because it operationalizes domain causal knowledge through sparse, directed influence paths while remaining adaptable to arbitrary sensor configurations without architectural changes (Mayr et al., 18 Aug 2025). The architecture decomposes prediction into a target base representation and additive pairwise influences, which the paper frames as interpretable components of system dynamics (Mayr et al., 18 Aug 2025). Influence vectors $(i \to j)$01 can be examined through their norms, their contributions to $(i \to j)$02, and sensitivity analyses, while attention maps within the CI encoder can be inspected per channel to analyze temporal patterns (Mayr et al., 18 Aug 2025). In the Refial case, the paper specifically points to gas flow, door state, hull temperature, and exhaust temperature as channels whose influences can be examined for insights into control efficacy, lagged effects, and thermal inertia (Mayr et al., 18 Aug 2025).

The principal limitation is reliance on a known causal graph $(i \to j)$03 (Mayr et al., 18 Aug 2025). The paper states that incorrect or incomplete graphs may omit important drivers or include spurious ones, affecting performance (Mayr et al., 18 Aug 2025). It therefore proposes several future directions: pruning by regularization on influence magnitudes, data-driven graph refinement followed by expert validation, and learnable influence aggregation at Stage 3 to downweight spurious parents (Mayr et al., 18 Aug 2025). Additional proposed extensions include attention-based aggregation over parents,

$(i \to j)$04

cross-attention at the pair level with graph-guided masks, self-supervised pretraining to encode causal proximity, probabilistic forecasting, and multi-hop pairs with counterfactual forecasting conditioned on planned control inputs (Mayr et al., 18 Aug 2025).

The paper also notes computational trade-offs when $(i \to j)$05 is large and recommends graph-guided sparsity, degree capping, edge sampling during training, and multi-hop selection with pruning (Mayr et al., 18 Aug 2025). These are practical consequences of the pairwise design rather than contradictions of it. In encyclopedia terms, CGPT should therefore be understood not as a generic Transformer with causal terminology added post hoc, but as a specific architecture whose defining features are graph-guided pair formation, shared channel-agnostic parameterization, additive influence aggregation, and a forecasting orientation tailored to industrial digital twin use cases (Mayr et al., 18 Aug 2025).

In the broader development of causal Transformers, CGPT marks a distinct architectural answer to a recurring problem: how to preserve explicit cross-variable structure without binding the model irreversibly to a fixed channel set. The reported evidence indicates that this answer is most effective in long-term forecasting and industrial settings where sparse causal drivers matter, while one-step forecasting remains dominated by strong autoregression and favors simpler CI baselines (Mayr et al., 18 Aug 2025).