GOODFormer: Graph Transformer for OOD Learning

• The paper introduces GOODFormer, which jointly optimizes an entropy-guided invariant subgraph disentangler, an evolving positional encoder, and an invariant learning module to robustly classify graphs under OOD shifts.
• GOODFormer integrates global attention with explicit invariance mechanisms, separating predictive invariant structure from spurious variant features to combat covariate shifts.
• Empirical evaluations show that GOODFormer outperforms traditional GT and MPNN baselines, with ablations confirming the critical role of subgraph disentanglement and evolving encodings in achieving superior OOD performance.

Searching arXiv for the specified papers and closely related context. arXiv search query: GOODFormer (Liao et al., 1 Aug 2025) and (Niv et al., 25 Jun 2025). Graph Out-Of-Distribution generalized Transformer (GOODFormer) is a graph-transformer architecture for out-of-distribution graph classification under distribution shifts. It is introduced to learn generalized graph representations by capturing invariant relationships between predictive graph structures and labels through jointly optimizing three modules: an entropy-guided invariant subgraph disentangler, an evolving subgraph positional and structural encoder, and an invariant learning module (Liao et al., 1 Aug 2025). The model sits within a broader line of work on graph-transformer OOD generalization in which GT and hybrid GT-MPNN backbones were found to generalize more strongly than MPNNs across several GOOD benchmark shifts, thereby motivating architectures that combine global attention, structural encodings, and explicit invariance mechanisms (Niv et al., 25 Jun 2025).

1. Problem formulation and research setting

The OOD setting considered for GOODFormer assumes covariate shifts in which $\mathbb P(G)$ changes across domains while $\mathbb P(Y\mid G)$ remains invariant. This formulation appears in the GOOD benchmark of Gui et al. ’22, which provides graph-classification tasks with training domains $\mathcal E_{\rm train}$ and held-out OOD domains $\mathcal E_{\rm test}$, together with train/ID-val/ID-test and OOD-val/OOD-test splits for measuring both in-distribution and out-of-distribution accuracy (Niv et al., 25 Jun 2025).

The benchmarked shift types include CMNIST-color, where node features are colored digits and OOD shifts the background/foreground color distribution; Motif-size, where the graph motif size distribution shifts in test domains; Motif-basis, where the structural basis shifts; and SST2-length, where sentence-derived graphs undergo length-distribution shift. In the broader GOODFormer paper, the empirical scope is extended to GOOD-Motif, GOOD-HIV, DrugOOD, GOOD-SST2, and GOOD-Twitter, with accuracy used on GOOD-Motif/SST2/Twitter and ROC-AUC used on GOOD-HIV/DrugOOD (Liao et al., 1 Aug 2025).

A recurrent assumption in earlier graph-transformer practice was that strong in-distribution performance would transfer to OOD conditions. The reported evidence does not support that assumption. The exploratory study on GT backbones shows that OOD behavior requires direct evaluation, while the GOODFormer paper states that existing GTs focus on training and testing graph data originated from the same distribution, but fail to generalize under distribution shifts. Taken together, these results suggest that transformer expressivity alone is insufficient, and that OOD robustness depends on how invariance is encoded into the architecture and objective.

2. Backbone formulation and end-to-end architecture

The graph-transformer formalism underlying this line of work starts from a graph $G=(V,E)$ with node features $\{x_i\}_{i\in V}$ and node states $h_i^{(\ell)}\in\mathbb R^d$. A standard GT backbone combines learnable input embeddings, positional or structural encodings, and global self-attention: $h_i^{(0)} = W_X x_i + \mathrm{PE}_i.$ Single-head attention is defined by

$$e_{ij}=\mathrm{LeakyReLU}\Bigl(a^{T}[W h_i^{(\ell)} \Vert W h_j^{(\ell)}]\Bigr),\qquad \alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k\in\mathcal N(i)}\exp(e_{ik})},$$

and multi-head attention aggregates head-wise value projections before output projection. In the hybrid GPS formulation of Rampášek et al., an MPNN branch and an MHA branch are combined and passed through an MLP, thereby coupling local message passing with global attention (Niv et al., 25 Jun 2025).

GOODFormer adopts this GT lineage but reorganizes it around invariance. It accepts an input graph $G=(\mathcal V,\mathcal E)$ with adjacency matrix $\mathbb P(Y\mid G)$0 and optionally node features $\mathbb P(Y\mid G)$1, and produces an OOD-robust prediction by three jointly optimized modules. The end-to-end pipeline is: encode the raw graph via initial GT layers to obtain intermediate node representations $\mathbb P(Y\mid G)$2; disentangle the graph into an invariant subgraph $\mathbb P(Y\mid G)$3 and a variant subgraph $\mathbb P(Y\mid G)$4; encode both subgraphs with an evolving positional and structural encoder; fuse subgraph node representations and PSEs through a small hybrid GT to obtain subgraph-level predictions $\mathbb P(Y\mid G)$5 and $\mathbb P(Y\mid G)$6; train with an invariant-risk objective, but at test time return only $\mathbb P(Y\mid G)$7 (Liao et al., 1 Aug 2025).

This organization is technically significant because it changes the role of attention. Rather than serving only as a feature aggregation mechanism, attention is made responsible for subgraph separation into predictive invariant structure and spurious or environment-specific variant structure. A plausible implication is that GOODFormer treats OOD generalization as a structured decomposition problem rather than only as domain alignment in the latent space.

3. Entropy-guided invariant subgraph disentangler

The entropy-guided invariant subgraph disentangler $\mathbb P(Y\mid G)$8 operates on intermediate node embeddings $\mathbb P(Y\mid G)$9. It first computes raw attention logits

$\mathcal E_{\rm train}$0

Two complementary attentions are then formed: $\mathcal E_{\rm train}$1 The associated soft adjacency masks are

$\mathcal E_{\rm train}$2

An attention-guided MPNN then refines the two parts: $\mathcal E_{\rm train}$3 followed by

$\mathcal E_{\rm train}$4

A central claim of GOODFormer is that sharpness of $\mathcal E_{\rm train}$5 is crucial for separating invariant and variant edges. To enforce sharpness, the model introduces an entropy regularizer during training,

$\mathcal E_{\rm train}$6

where $\mathcal E_{\rm train}$7. Minimizing $\mathcal E_{\rm train}$8 makes the attention distributions peaky, widening the gap between invariant and variant edges. At test time, GOODFormer calibrates attention via temperature scaling: $\mathcal E_{\rm train}$9 where $\mathcal E_{\rm test}$0 is the average entropy observed during training (Liao et al., 1 Aug 2025).

The theoretical justification is given through two results. Theorem 1 states that, under mild assumptions and with a unique non-trivial invariant subgraph, a plain complementary-softmax disentangler cannot simultaneously minimize average interventional risk and its variance, because the softmax gap can become arbitrarily small when variant or invariant edge counts grow large. Theorem 2 states that the Shannon entropy of $\mathcal E_{\rm test}$1 is strictly increasing in $\mathcal E_{\rm test}$2, so lowering $\mathcal E_{\rm test}$3 always sharpens attention. In combination, these results place the disentangler at the center of GOODFormer’s OOD design rather than treating it as a heuristic add-on.

4. Evolving subgraph positional and structural encoding

GOODFormer’s second module addresses a specific difficulty introduced by subgraph disentanglement: the subgraphs $\mathcal E_{\rm test}$4 and $\mathcal E_{\rm test}$5 evolve each epoch, so classical positional and structural encodings computed once on a fixed graph become mismatched or expensive to recompute. The proposed solution is an evolving subgraph positional and structural encoder, implemented as a light MPNN-based encoder $\mathcal E_{\rm test}$6 (Liao et al., 1 Aug 2025).

The procedure samples random node features $\mathcal E_{\rm test}$7 and builds two auxiliary graphs,

$\mathcal E_{\rm test}$8

where $\mathcal E_{\rm test}$9 and $G=(V,E)$0 are induced by the learned masks. The encoder produces

$G=(V,E)$1

To make these learned encodings preserve graph positional and structural information, a decoder $G=(V,E)$2 is trained to reconstruct a hand-crafted encoding $G=(V,E)$3 such as the top-$G=(V,E)$4 Laplacian eigenvectors computed once on the original graph: $G=(V,E)$5

A notable implementation detail is that backpropagation is blocked from $G=(V,E)$6 into the disentangler so as not to leak label–PSE correlations. This design indicates that GOODFormer treats positional encoding not merely as an initialization choice but as a dynamic component of the invariant-learning pipeline. The reported ablation evidence is consistent with that interpretation: removing the encoder causes a drop of 10–30%, while replacing it with hand-crafted LapPE or SignNet at input gives only a slight gain over the encoder-free variant and remains far below the full model.

5. Invariant learning objective and optimization

The invariant learning module consumes the disentangled node representations and their PSEs. For the invariant branch, node features $G=(V,E)$7 are processed by a small hybrid GT $G=(V,E)$8, pooled, and mapped by a classifier $G=(V,E)$9: $\{x_i\}_{i\in V}$0 with an analogous definition for $\{x_i\}_{i\in V}$1. To simulate an intervention $\{x_i\}_{i\in V}$2, the two predictions are combined as

$\{x_i\}_{i\in V}$3

where $\{x_i\}_{i\in V}$4 is sigmoid. During inference, only $\{x_i\}_{i\in V}$5 is returned in order to shield spurious variant effects (Liao et al., 1 Aug 2025).

The core training criterion is an interventional invariant-risk objective: $\{x_i\}_{i\in V}$6 supplemented with a classification loss on the variant branch, $\{x_i\}_{i\in V}$7. The full objective is

$\{x_i\}_{i\in V}$8

Typical hyperparameters are $\{x_i\}_{i\in V}$9, $h_i^{(\ell)}\in\mathbb R^d$0, $h_i^{(\ell)}\in\mathbb R^d$1, and $h_i^{(\ell)}\in\mathbb R^d$2.

The implementation reported for GOODFormer uses GraphGPS layers as the backbone GT$h_i^{(\ell)}\in\mathbb R^d$3 with 2 blocks, a 3-layer PSE MPNN, a 1-block hybrid GT $h_i^{(\ell)}\in\mathbb R^d$4, mean pooling, ReLU activations, and node and graph representations of dimension $h_i^{(\ell)}\in\mathbb R^d$5, while the GNN baselines use $h_i^{(\ell)}\in\mathbb R^d$6. Optimization uses Adam for 100 epochs with learning rate tuned per dataset. The paper summarizes this design by stating that GOODFormer is the first Graph Transformer built under invariant-learning principles (Liao et al., 1 Aug 2025).

6. Empirical behavior, diagnostics, and interpretation

The empirical evaluation compares GOODFormer against general OOD methods—ERM, IRM, and VREx—graph OOD methods—DIR, GIL, GSAT, CIGA, and iMoLD—and graph transformers used under IID assumptions—Graphormer, GraphGPS, and Exphormer. The main result reported is that GOODFormer outperforms all baselines on every dataset, often by large margins, including +11–14 percentage points on GOOD-Motif splits and +2–3 percentage points on real datasets. The same section reports that standard GTs collapse under OOD shift, invariant-learning GNNs only partly help, and GOODFormer combines GT expressivity with invariance (Liao et al., 1 Aug 2025).

Ablation studies identify three components as especially important. Removing the entropy term and test-time tuning causes a large performance drop. Replacing the disentangler with GIL’s subgraph generator is inferior. Removing the evolving encoder produces a 10–30% drop, and injecting LapPE or SignNet only at input remains far below the full model. Hyper-parameter sensitivity is reported as stable over $h_i^{(\ell)}\in\mathbb R^d$7, $h_i^{(\ell)}\in\mathbb R^d$8, and $h_i^{(\ell)}\in\mathbb R^d$9. Runtime on GOOD-Motif with a Tesla V100 is reported as approximately 28.6 s per training epoch and 7.1 s per test epoch for GOODFormer, compared with approximately 7.7 s/2.1 s for Graphormer, 9.5 s/1.7 s for GraphGPS, and 11.9 s/5.6 s for Exphormer; the complexity remains $h_i^{(0)} = W_X x_i + \mathrm{PE}_i.$0 plus a small $h_i^{(0)} = W_X x_i + \mathrm{PE}_i.$1 test-time loop for fitting $h_i^{(0)} = W_X x_i + \mathrm{PE}_i.$2 (Liao et al., 1 Aug 2025).

Related diagnostic analysis from the earlier graph-transformer OOD study provides an informative lens for interpreting why hybrid transformer backbones can be strong OOD candidates. After training, penultimate-layer embeddings on the entire ID and OOD test sets are evaluated using domain alignment, measured by MMD with kernel $h_i^{(0)} = W_X x_i + \mathrm{PE}_i.$3, and class separation, measured by the Silhouette score. Lower MMD indicates better alignment of ID and OOD embedding distributions, while higher Silhouette indicates tighter intra-class clusters and clearer inter-class margins across domains. In that study, GPS achieved approximately 86.6% OOD accuracy versus approximately 70.6% for MHA and approximately 55.6% for vGIN; GPS also had the lowest MMD at approximately 0.02, compared with approximately 0.05 for MHA and approximately 0.2–0.3 for vGIN (Niv et al., 25 Jun 2025).

These results do not imply that any hybrid GT is automatically OOD-robust. Rather, they suggest that hybridization supplies a favorable architectural substrate, while GOODFormer adds explicit invariant subgraph disentanglement, evolving subgraph encoding, and intervention-based training to convert that substrate into a dedicated OOD method. In that sense, GOODFormer can be read as an overview of two lines of evidence: first, that GT and hybrid GT-MPNN backbones can outperform MPNNs under distribution shift; second, that OOD robustness improves further when invariance is imposed at the level of subgraph decomposition and interventional risk rather than inferred indirectly from accuracy alone.