GSTBench: Graph SSL Transferability Benchmark

• GSTBench is a benchmark that evaluates the cross-dataset transferability of graph SSL methods using a controlled pretrain-then-transfer protocol.
• It uses unified LLM-derived node embeddings from SentenceBERT to reduce feature heterogeneity across diverse graph datasets.
• Empirical findings show generative objectives like GraphMAE outperform contrastive methods, sometimes even surpassing random initialization.

GSTBench, short for Graph SSL Transferability Benchmark, is a benchmark for evaluating the cross-dataset transferability of graph self-supervised learning (SSL) methods under large-scale pretraining and unified textual node features. Introduced in "GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning" (Song et al., 28 Aug 2025), it addresses a specific gap in graph representation learning: most graph SSL methods had been developed and evaluated in a single-dataset setting, leaving open the question of whether pretraining on a very large source graph yields representations that generalize to new graphs, domains, and tasks. GSTBench frames this question within a controlled pretrain-then-transfer protocol, standardizing architecture, features, and adaptation so that observed differences can be attributed primarily to the pretraining objective.

1. Motivation and benchmark scope

GSTBench is motivated by three problems identified in existing graph pretraining work. First, many graph foundation model pipelines entangle multiple design choices, including pretext task, architecture, prompting or adaptation module, task formulation, and special handling for heterogeneous graph inputs. This makes it difficult to isolate the effect of the SSL objective itself. Second, evaluation practices had been inconsistent across datasets, splits, feature encoders, adaptation strategies, and downstream settings, weakening cross-paper comparability. Third, most prior studies pretrained on modest-sized graphs rather than on truly large corpora that could test whether graph SSL scales in a manner comparable to SSL in NLP or computer vision (Song et al., 28 Aug 2025).

The benchmark therefore focuses explicitly on graph SSL transferability in LLM-unified feature spaces. Its central design choice is to represent nodes with LLM-derived node embeddings, so that graphs from different domains share a common feature space. In the benchmark, all node features are extracted using SentenceBERT. This reduces feature heterogeneity and makes cross-graph transfer a more direct test of whether the pretrained graph encoder has acquired transferable structural and semantic regularities.

A plausible implication is that GSTBench is intended not merely as an accuracy leaderboard, but as a controlled test of whether graph SSL objectives produce transferable knowledge once common confounders are removed.

2. Pretraining corpus and target datasets

All SSL methods in GSTBench are pretrained on ogbn-papers100M, a large citation graph containing roughly 100 million nodes and 1.6 billion edges (Song et al., 28 Aug 2025). Because the graph is too large for GPU memory, it is preprocessed into about 10,000 METIS partitions, and training is performed with mini-batches over these subgraphs. The reported partition statistics are an average of 10,000.90 nodes per subgraph and 61,357.03 edges per subgraph, with node counts ranging from 303 to 45,748 and edge counts ranging from 328 to 122,644. The benchmark is therefore explicitly designed as a large-scale pretraining setting rather than a small-graph setting.

Transfer is evaluated on eight target datasets, divided into in-domain and cross-domain settings. The in-domain datasets are Cora, Citeseer, WikiCS, DBLP, and PubMed. The cross-domain datasets are Amazon Ratings, Child, and Photo. WikiCS is treated as in-domain because it is structurally and semantically similar to citation-family data, despite being derived from Wikipedia rather than a citation network.

The target graphs differ substantially in size, homophily, and feature shift. The reported statistics are as follows. Cora has 2,708 nodes, 10,556 edges, 7 classes, MMD 0.07, and homophily 0.81, with domain Citation. Citeseer has 3,186 nodes, 8,450 edges, 6 classes, MMD 0.06, and homophily 0.78, also in Citation. WikiCS has 11,701 nodes, 431,726 edges, 10 classes, MMD 0.10, and homophily 0.65, in Wikipedia. DBLP has 14,376 nodes, 431,326 edges, 4 classes, MMD 0.13, and homophily 0.67, in Citation. PubMed has 19,717 nodes, 88,648 edges, 3 classes, MMD 0.15, and homophily 0.80, again in Citation. Amazon Ratings has 24,492 nodes, 186,100 edges, 5 classes, MMD 0.23, and homophily 0.38, in E-commerce. Child has 76,875 nodes, 2,325,044 edges, 24 classes, MMD 0.10, and homophily 0.42, in E-commerce. Photo has 48,362 nodes, 873,782 edges, 12 classes, MMD 0.13, and homophily 0.75, also in E-commerce.

These statistics are used in the benchmark to contextualize transfer difficulty. The combination of MMD and homophily differences indicates that transfer is being evaluated under both feature shift and structural shift.

3. Controlled benchmark design

GSTBench compares five representative pretraining objectives, chosen to span the major families of graph SSL: GraphMAE, VGAE, DGI, GRACE, and LP (Song et al., 28 Aug 2025). The paper characterizes them as follows: GraphMAE performs masked feature reconstruction using local neighborhood context; VGAE performs structure reconstruction with KL regularization; DGI maximizes mutual information between local node embeddings and a global summary; GRACE uses contrastive learning with feature-drop and edge-drop augmentations; and LP uses edge-existence prediction as the pretraining objective.

A central feature of the benchmark is architectural control. All methods use the same encoder families, GCN and GAT, and both backbones use 2 convolution layers with hidden dimension 384. The benchmark excludes BatchNorm and LayerNorm, because they did not help in transfer. This choice is intended to ensure that model capacity and normalization do not confound comparisons among SSL objectives.

Dataset control is imposed by evaluating all methods on the same eight downstream graphs with the same SentenceBERT-derived node embeddings. Adaptation protocol control is imposed by using the same downstream strategies across methods. For node classification, GSTBench studies linear probing, fine-tuning, and in-context learning (ICL). For link prediction, it uses fine-tuning, with the explicit rationale that LP is already a self-supervised downstream objective and benefits from fully adapting the encoder.

This controlled setup is central to the benchmark’s interpretation: the benchmark is designed to measure transferability of SSL objectives rather than variation induced by feature engineering, architectural changes, or inconsistent downstream protocols.

4. Formal pretrain-and-transfer formulation and evaluation protocol

GSTBench defines a general pretrain-and-transfer setup in terms of a GNN encoder $f(\cdot;\theta)$ and a pretraining head $h(\cdot;\phi)$. Pretraining on a source graph $\mathcal{G}^s$ is formulated as

$$\min_{\theta,\phi} \,\mathcal{L}^s\bigl(f(\cdot;\theta),\, h(\cdot;\phi),\, \mathcal{G}^s\bigr).$$

The pretrained encoder is then transferred to a target graph $\mathcal{G}^t$.

For linear probing, the encoder is frozen and only the classifier is learned:

$$\theta^* \;=\; \Bigl(\theta,\; \arg\min_{\phi}\,\mathcal{L}^t\bigl(f(\cdot;\theta),\,h(\cdot;\phi),\,\mathcal{G}^t\bigr)\Bigr).$$

For fine-tuning, both encoder and head are updated:

$$\theta^* \;=\; \arg\min_{\theta,\phi}\,\mathcal{L}^t\bigl(f(\cdot;\theta),\,h(\cdot;\phi),\,\mathcal{G}^t\bigr).$$

For in-context learning, the paper follows a graph ICL formulation with class nodes. For an $N$-way $K$-shot problem, the augmented graph contains class nodes linked to support examples, and prediction is based on cosine similarity:

$$\hat{y} = \arg\max_{i \in \{1, \dots, N\}} \cos(t, s_i).$$

For node classification, GSTBench uses a 5-shot protocol: 5 labeled nodes per class for training, 500 validation nodes, and the remaining nodes for testing. Each dataset is evaluated over 5 random splits, and the reported quantities are mean accuracy and standard deviation. For link prediction, edges are split 40\% train / 10\% validation / 50\% test; the metric is MRR; and results are averaged over 3 random seeds (Song et al., 28 Aug 2025).

Pretraining uses AdamW, with learning rate searched from 1e-5 to 1e-2, a cosine schedule with warmup in the first epoch, and a maximum of 5 epochs. SSL-specific hyperparameters follow the original implementations as closely as possible. In downstream evaluation, linear probing uses learning rate 1e-2 and weight decay 1e-4. Fine-tuning searches learning rate in 1e-2 to 1e-4 and weight decay in 0 to 1e-5.

5. Principal empirical results

The central empirical result of GSTBench is that many graph SSL methods transfer poorly, and some can perform worse than random initialization (Song et al., 28 Aug 2025). This finding is notable because the input features are already strong: with SentenceBERT embeddings, a randomly initialized GNN is surprisingly competitive. On Cora, for example, linear probing with random initialization already achieves about 0.737–0.744 accuracy depending on whether the backbone is GCN or GAT. The benchmark therefore tests whether pretraining improves the model’s use of graph structure, rather than whether it merely recovers informative node features.

Across node classification and link prediction, GraphMAE is the only method reported to consistently improve transfer performance and to remain stable across settings. In linear probing for node classification, the average accuracies over in-domain datasets are 0.683 (GCN) and 0.711 (GAT) for random initialization, compared with 0.714 (GCN) and 0.728 (GAT) for GraphMAE. Over cross-domain datasets, the corresponding values are 0.326 (GCN) and 0.351 (GAT) for random initialization, versus 0.363 (GCN) and 0.365 (GAT) for GraphMAE. Under in-context learning, GraphMAE again leads, with in-domain mean accuracy of 0.673 (GCN) and 0.696 (GAT), and cross-domain mean accuracy of 0.314 (GCN) and 0.341 (GAT). In link prediction, GraphMAE attains the strongest reported overall in-domain result, with mean MRR 0.749 (GCN).

By contrast, the contrastive methods DGI and GRACE are reported as often unstable and sometimes harmful. In node classification they often fail to outperform random initialization meaningfully, and DGI in particular tends to degrade as pretraining progresses. In link prediction their MRR values are usually close to the random baseline. This is described as evidence of negative transfer, meaning that a pretraining strategy can degrade downstream performance when moved to a different graph.

VGAE is described as inconsistent. It sometimes helps, especially in some cross-domain node-classification settings, but performs especially poorly in link prediction. The benchmark highlights a specific mismatch: LP pretraining helps link prediction, while VGAE harms link prediction, even though both are edge-oriented objectives. The proposed explanation is architectural mismatch: VGAE uses a dot-product decoder, whereas downstream link prediction uses a learnable MLP decoder.

The fine-tuning results are especially restrictive. In Table 5, all methods cluster around nearly the same performance for 5-shot node classification: Random 0.722 mean, GraphMAE 0.720, VGAE 0.720, GRACE 0.720, DGI 0.720, and LP 0.718. Under this benchmark, fine-tuning therefore does not significantly benefit from large-scale pretraining. The paper interprets this as evidence that, when the encoder is fully updated, much of the pretraining signal is overwritten.

6. Transferability analysis and broader implications

GSTBench also analyzes why transferability differs across objectives. The reported pattern is that generative objectives transfer better than contrastive ones. GraphMAE and VGAE, both generative, are generally stronger and more stable than contrastive methods. The paper argues that reconstruction-based objectives encourage a more robust understanding of the data distribution, analogizing this to masked image modeling in vision and masked language modeling in NLP (Song et al., 28 Aug 2025). GraphMAE is identified as especially effective because it jointly learns neighborhood interaction patterns and preserves rich semantic information from LLM features.

By contrast, augmentation-based contrastive learning is described as brittle because it depends heavily on augmentation choice, perturbation strength, and negative sample quality. If these are poorly matched to the target data, the learned invariances may fail to transfer and may even remove useful semantics. This suggests that strong in-domain pretext performance is not sufficient evidence of cross-dataset utility.

The benchmark studies the relationship between pretext optimization and downstream transfer by computing Spearman correlation between SSL error and validation accuracy across checkpoints for GraphMAE, after ranking values within each pretraining trajectory. The reported conclusion is a statistically significant negative correlation with $h(\cdot;\phi)$0: lower SSL loss generally correlates with better transfer performance. However, the relationship is not strictly monotonic. Accuracy can plateau or decline while SSL loss continues to improve, which motivates early stopping or other regularization during graph SSL pretraining.

Adaptation strategy and architecture also affect transfer. Linear probing is described as the most stable way to reveal the usefulness of pretrained representations. ICL can work, but may not fully exploit the representation. Fine-tuning often washes out the benefit of pretraining. With respect to backbone choice, GAT often gives higher absolute accuracy in linear probing and ICL, whereas in fine-tuning for link prediction it tends to adapt more aggressively and can overwrite pretrained knowledge more than GCN. The paper supports this with a training-dynamics analysis showing larger parameter drift for GAT during fine-tuning and lower cosine similarity between pretrained and fine-tuned embeddings. This suggests greater flexibility but lower stability as a transfer mechanism.

Domain similarity is another reported determinant of transfer. Cross-domain transfer to e-commerce graphs is substantially harder than in-domain transfer to citation-like graphs, and the paper uses the reported MMD and homophily statistics to show substantial shift in both feature and structural distributions. A plausible implication is that pretraining helps most when source and target graphs are reasonably aligned in both semantics and graph topology.

At a broader level, GSTBench advances a cautionary interpretation of graph foundation models. Strong SSL performance on the pretraining graph does not imply transferable representations; pretraining can cause negative transfer; and large-scale pretraining alone is insufficient. With rich SentenceBERT features, a randomly initialized GNN can already be strong, so SSL must contribute genuinely transferable structural bias rather than merely reconstructing easy semantic signals. The paper therefore concludes that future progress should emphasize new masked reconstruction objectives, hybrid objectives that combine generative and contrastive strengths, improved adaptation strategies such as prompt tuning or improved ICL methods, architectures that better preserve pretrained knowledge during transfer, and further study of alternative LLM feature encoders beyond SentenceBERT.