---
title: 'GOODFormer: Graph Transformer for OOD Learning'
url: https://www.emergentmind.com/topics/graph-out-of-distribution-generalized-transformer-goodformer
type: topic
---

# GOODFormer: Graph Transformer for OOD Learning

Searching arXiv for the specified papers and closely related context.
arXiv search query: GOODFormer 2508.00304 and 2506.20575.
Graph Out-Of-Distribution generalized Transformer (GOODFormer) is a graph-transformer architecture for out-of-distribution graph classification under distribution shifts. It is introduced to learn generalized graph representations by capturing invariant relationships between predictive graph structures and labels through jointly optimizing three modules: an entropy-guided invariant subgraph disentangler, an evolving subgraph positional and structural encoder, and an invariant learning module [2508.00304]. The model sits within a broader line of work on graph-transformer OOD generalization in which GT and hybrid GT-MPNN backbones were found to generalize more strongly than MPNNs across several GOOD benchmark shifts, thereby motivating architectures that combine global attention, structural encodings, and explicit invariance mechanisms [2506.20575].

## 1. Problem formulation and research setting

The OOD setting considered for GOODFormer assumes covariate shifts in which $\mathbb P(G)$ changes across domains while $\mathbb P(Y\mid G)$ remains invariant. This formulation appears in the GOOD benchmark of Gui et al. ’22, which provides graph-classification tasks with training domains $\mathcal E_{\rm train}$ and held-out OOD domains $\mathcal E_{\rm test}$, together with train/ID-val/ID-test and OOD-val/OOD-test splits for measuring both in-distribution and out-of-distribution accuracy [2506.20575].

The benchmarked shift types include CMNIST-color, where node features are colored digits and OOD shifts the background/foreground color distribution; Motif-size, where the graph motif size distribution shifts in test domains; Motif-basis, where the structural basis shifts; and SST2-length, where sentence-derived graphs undergo length-distribution shift. In the broader GOODFormer paper, the empirical scope is extended to GOOD-Motif, GOOD-HIV, DrugOOD, GOOD-SST2, and GOOD-Twitter, with accuracy used on GOOD-Motif/SST2/Twitter and ROC-AUC used on GOOD-HIV/DrugOOD [2508.00304].

A recurrent assumption in earlier graph-transformer practice was that strong in-distribution performance would transfer to OOD conditions. The reported evidence does not support that assumption. The exploratory study on GT backbones shows that OOD behavior requires direct evaluation, while the GOODFormer paper states that existing GTs focus on training and testing graph data originated from the same distribution, but fail to generalize under distribution shifts. Taken together, these results suggest that transformer expressivity alone is insufficient, and that OOD robustness depends on how invariance is encoded into the architecture and objective.

## 2. Backbone formulation and end-to-end architecture

The graph-transformer formalism underlying this line of work starts from a graph $G=(V,E)$ with node features $\{x_i\}_{i\in V}$ and node states $h_i^{(\ell)}\in\mathbb R^d$. A standard GT backbone combines learnable input embeddings, positional or structural encodings, and global self-attention:
\[
h_i^{(0)} = W_X x_i + \mathrm{PE}_i.
\]
Single-head attention is defined by
\[
e_{ij}=\mathrm{LeakyReLU}\Bigl(a^{T}[W h_i^{(\ell)} \Vert W h_j^{(\ell)}]\Bigr),\qquad
\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k\in\mathcal N(i)}\exp(e_{ik})},
\]
and multi-head attention aggregates head-wise value projections before output projection. In the hybrid GPS formulation of Rampášek et al., an MPNN branch and an MHA branch are combined and passed through an MLP, thereby coupling local message passing with global attention [2506.20575].

GOODFormer adopts this GT lineage but reorganizes it around invariance. It accepts an input graph $G=(\mathcal V,\mathcal E)$ with adjacency matrix $\mathbf A$ and optionally node features $\mathbf X$, and produces an OOD-robust prediction by three jointly optimized modules. The end-to-end pipeline is: encode the raw graph via initial GT layers to obtain intermediate node representations $\mathbf Z$; disentangle the graph into an invariant subgraph $G_{\tilde c}$ and a variant subgraph $G_{\tilde s}$; encode both subgraphs with an evolving positional and structural encoder; fuse subgraph node representations and PSEs through a small hybrid GT to obtain subgraph-level predictions $\hat y_{\tilde c}$ and $\hat y_{\tilde s}$; train with an invariant-risk objective, but at test time return only $\hat y_{\tilde c}$ [2508.00304].

This organization is technically significant because it changes the role of attention. Rather than serving only as a feature aggregation mechanism, attention is made responsible for subgraph separation into predictive invariant structure and spurious or environment-specific variant structure. A plausible implication is that GOODFormer treats OOD generalization as a structured decomposition problem rather than only as domain alignment in the latent space.

## 3. Entropy-guided invariant subgraph disentangler

The entropy-guided invariant subgraph disentangler $\Phi$ operates on intermediate node embeddings $\mathbf Z\in\mathbb R^{|\mathcal V|\times d}$. It first computes raw attention logits
\[
\mathbf E=\frac{\mathbf Z\mathbf W_Q(\mathbf Z\mathbf W_K)^\top}{\sqrt{d_K}} \in \mathbb R^{|\mathcal V|\times |\mathcal V|}.
\]
Two complementary attentions are then formed:
\[
\mathbf Z_{\mathrm{Attn},\tilde c}=\mathrm{Softmax}(\mathbf E)\,\mathbf Z\,\mathbf W_V,
\qquad
\mathbf Z_{\mathrm{Attn},\tilde s}=\mathrm{Softmax}(-\mathbf E)\,\mathbf Z\,\mathbf W_V.
\]
The associated soft adjacency masks are
\[
\mathbf M=\sigma(\mathbf E),\qquad
\mathbf A_{\tilde c}=\mathbf M\odot \mathbf A,\qquad
\mathbf A_{\tilde s}=(1-\mathbf M)\odot \mathbf A.
\]
An attention-guided MPNN then refines the two parts:
\[
\mathbf Z_{\mathrm{MPNN},\tilde c}=\mathrm{MPNN}(\mathbf Z,\mathbf A_{\tilde c}),\qquad
\mathbf Z_{\mathrm{MPNN},\tilde s}=\mathrm{MPNN}(\mathbf Z,\mathbf A_{\tilde s}),
\]
followed by
\[
\mathbf Z_{\tilde c}=\mathrm{MLP}\bigl(\mathbf Z_{\mathrm{Attn},\tilde c}+\mathbf Z_{\mathrm{MPNN},\tilde c}\bigr),\qquad
\mathbf Z_{\tilde s}=\mathrm{MLP}\bigl(\mathbf Z_{\mathrm{Attn},\tilde s}+\mathbf Z_{\mathrm{MPNN},\tilde s}\bigr).
\]

A central claim of GOODFormer is that sharpness of $\mathrm{Softmax}$ is crucial for separating invariant and variant edges. To enforce sharpness, the model introduces an entropy regularizer during training,
\[
\mathcal L_E=\mathbb E_{\mathbf E}\bigl[\mathcal H(\mathrm{Softmax}(\mathbf E))\bigr],
\]
where $\mathcal H(p)=-\sum_i p_i\ln p_i$. Minimizing $\mathcal L_E$ makes the attention distributions peaky, widening the gap between invariant and variant edges. At test time, GOODFormer calibrates attention via temperature scaling:
\[
\mathrm{Softmax}_t(e_i)=\frac{\exp(e_i/t)}{\sum_k\exp(e_k/t)},
\qquad
t_G=\arg\min_{t>0}\Bigl|\overline H-\mathcal H(\mathrm{Softmax}_t(\mathbf E_G))\Bigr|,
\]
where $\overline H$ is the average entropy observed during training [2508.00304].

The theoretical justification is given through two results. Theorem 1 states that, under mild assumptions and with a unique non-trivial invariant subgraph, a plain complementary-softmax disentangler cannot simultaneously minimize average interventional risk and its variance, because the softmax gap can become arbitrarily small when variant or invariant edge counts grow large. Theorem 2 states that the Shannon entropy of $\mathrm{Softmax}_t(\mathbf e)$ is strictly increasing in $t$, so lowering $t$ always sharpens attention. In combination, these results place the disentangler at the center of GOODFormer’s OOD design rather than treating it as a heuristic add-on.

## 4. Evolving subgraph positional and structural encoding

GOODFormer’s second module addresses a specific difficulty introduced by subgraph disentanglement: the subgraphs $G_{\tilde c}$ and $G_{\tilde s}$ evolve each epoch, so classical positional and structural encodings computed once on a fixed graph become mismatched or expensive to recompute. The proposed solution is an evolving subgraph positional and structural encoder, implemented as a light MPNN-based encoder $h_{\mathrm{PSE}}$ [2508.00304].

The procedure samples random node features $\mathbf X\sim\mathcal N(0,\mathbf I)$ and builds two auxiliary graphs,
\[
G_{\tilde c}'=(\mathbf X,\mathcal E_{\tilde c}),\qquad
G_{\tilde s}'=(\mathbf X,\mathcal E_{\tilde s}),
\]
where $\mathcal E_{\tilde c}$ and $\mathcal E_{\tilde s}$ are induced by the learned masks. The encoder produces
\[
\mathbf Z_{\mathrm{PSE},\tilde c}=h_{\mathrm{PSE}}(G_{\tilde c}'),\qquad
\mathbf Z_{\mathrm{PSE},\tilde s}=h_{\mathrm{PSE}}(G_{\tilde s}').
\]
To make these learned encodings preserve graph positional and structural information, a decoder $w_{\mathrm{PSE}}$ is trained to reconstruct a hand-crafted encoding $\mathbf z_{\mathrm{PSE}}$ such as the top-$k$ Laplacian eigenvectors computed once on the original graph:
\[
\mathcal L_{\mathrm{PSE}}
=
\mathbb E_{G_{\tilde c}}\!\left[\left\|w_{\mathrm{PSE}}(\mathbf Z_{\mathrm{PSE},\tilde c})-\mathbf z_{\mathrm{PSE}}\right\|\right]
+
\mathbb E_{G_{\tilde s}}\!\left[\left\|w_{\mathrm{PSE}}(\mathbf Z_{\mathrm{PSE},\tilde s})-\mathbf z_{\mathrm{PSE}}\right\|\right].
\]

A notable implementation detail is that backpropagation is blocked from $\mathcal L_{\mathrm{PSE}}$ into the disentangler so as not to leak label–PSE correlations. This design indicates that GOODFormer treats positional encoding not merely as an initialization choice but as a dynamic component of the invariant-learning pipeline. The reported ablation evidence is consistent with that interpretation: removing the encoder causes a drop of 10–30%, while replacing it with hand-crafted LapPE or SignNet at input gives only a slight gain over the encoder-free variant and remains far below the full model.

## 5. Invariant learning objective and optimization

The invariant learning module consumes the disentangled node representations and their PSEs. For the invariant branch, node features $[\mathbf Z_{\tilde c},\mathbf Z_{\mathrm{PSE},\tilde c}]$ are processed by a small hybrid GT $h'$, pooled, and mapped by a classifier $w_{\tilde c}$:
\[
\hat y_{\tilde c}
=
w_{\tilde c}\Bigl(\mathrm{Pool}\bigl(h'([\mathbf Z_{\tilde c},\mathbf Z_{\mathrm{PSE},\tilde c}])\bigr)\Bigr),
\]
with an analogous definition for $\hat y_{\tilde s}$. To simulate an intervention $do(G_S=\tilde s)$, the two predictions are combined as
\[
\hat y = \hat y_{\tilde c}\odot \sigma(\hat y_{\tilde s}),
\]
where $\sigma$ is sigmoid. During inference, only $\hat y_{\tilde c}$ is returned in order to shield spurious variant effects [2508.00304].

The core training criterion is an interventional invariant-risk objective:
\[
\mathcal L_I
=
\mathbb E_{G_s}\bigl[\ell(h(G),Y\mid do(G_S=G_s))\bigr]
+
\lambda\,\mathrm{Var}_{G_s}\Bigl\{\ell(h(G),Y\mid do(G_S=G_s))\Bigr\},
\]
supplemented with a classification loss on the variant branch, $\mathcal L_{\tilde S}=\mathbb E[\ell(\hat y_{\tilde s},Y)]$. The full objective is
\[
\min_{h,w_{\tilde s},w_{\mathrm{PSE}}}
\quad
\mathcal L_I+\alpha_S\mathcal L_{\tilde S}+\alpha_E\mathcal L_E+\alpha_{\mathrm{PSE}}\mathcal L_{\mathrm{PSE}}.
\]
Typical hyperparameters are $\alpha_S=1$, $\alpha_E\in\{10^{-1},10^{-2}\}$, $\alpha_{\mathrm{PSE}}=10^{-2}$, and $\lambda\in\{10^2,10^1,1,10^{-1}\}$.

The implementation reported for GOODFormer uses GraphGPS layers as the backbone GT$^M$ with 2 blocks, a 3-layer PSE MPNN, a 1-block hybrid GT $h'$, mean pooling, ReLU activations, and node and graph representations of dimension $d=128$, while the GNN baselines use $d=300$. Optimization uses Adam for 100 epochs with learning rate tuned per dataset. The paper summarizes this design by stating that GOODFormer is the first Graph Transformer built under invariant-learning principles [2508.00304].

## 6. Empirical behavior, diagnostics, and interpretation

The empirical evaluation compares GOODFormer against general OOD methods—ERM, IRM, and VREx—graph OOD methods—DIR, GIL, GSAT, CIGA, and iMoLD—and graph transformers used under IID assumptions—Graphormer, GraphGPS, and Exphormer. The main result reported is that GOODFormer outperforms all baselines on every dataset, often by large margins, including +11–14 percentage points on GOOD-Motif splits and +2–3 percentage points on real datasets. The same section reports that standard GTs collapse under OOD shift, invariant-learning GNNs only partly help, and GOODFormer combines GT expressivity with invariance [2508.00304].

Ablation studies identify three components as especially important. Removing the entropy term and test-time tuning causes a large performance drop. Replacing the disentangler with GIL’s subgraph generator is inferior. Removing the evolving encoder produces a 10–30% drop, and injecting LapPE or SignNet only at input remains far below the full model. Hyper-parameter sensitivity is reported as stable over $\lambda\in[10^{-1},10^2]$, $\alpha_E\in[10^{-2},10^{-1}]$, and $\alpha_{\mathrm{PSE}}\approx 10^{-2}$. Runtime on GOOD-Motif with a Tesla V100 is reported as approximately 28.6 s per training epoch and 7.1 s per test epoch for GOODFormer, compared with approximately 7.7 s/2.1 s for Graphormer, 9.5 s/1.7 s for GraphGPS, and 11.9 s/5.6 s for Exphormer; the complexity remains $O(|E|d+|V|^2d^2)$ plus a small $O(|V|^2T)$ test-time loop for fitting $t_G$ [2508.00304].

Related diagnostic analysis from the earlier graph-transformer OOD study provides an informative lens for interpreting why hybrid transformer backbones can be strong OOD candidates. After training, penultimate-layer embeddings on the entire ID and OOD test sets are evaluated using domain alignment, measured by MMD with kernel $k(x,y)=\exp(-\gamma\|x-y\|_1)$, and class separation, measured by the Silhouette score. Lower MMD indicates better alignment of ID and OOD embedding distributions, while higher Silhouette indicates tighter intra-class clusters and clearer inter-class margins across domains. In that study, GPS achieved approximately 86.6% OOD accuracy versus approximately 70.6% for MHA and approximately 55.6% for vGIN; GPS also had the lowest MMD at approximately 0.02, compared with approximately 0.05 for MHA and approximately 0.2–0.3 for vGIN [2506.20575].

These results do not imply that any hybrid GT is automatically OOD-robust. Rather, they suggest that hybridization supplies a favorable architectural substrate, while GOODFormer adds explicit invariant subgraph disentanglement, evolving subgraph encoding, and intervention-based training to convert that substrate into a dedicated OOD method. In that sense, GOODFormer can be read as a synthesis of two lines of evidence: first, that GT and hybrid GT-MPNN backbones can outperform MPNNs under distribution shift; second, that OOD robustness improves further when invariance is imposed at the level of subgraph decomposition and interventional risk rather than inferred indirectly from accuracy alone.

Source: https://www.emergentmind.com/topics/graph-out-of-distribution-generalized-transformer-goodformer