A Fair Evaluation of Graph Foundation Models for Node Property Prediction

Abstract: Due to the wide use of graph-structured data in different fields of industry and science, the development of Graph Foundation Models (GFMs) has recently attracted a lot of attention. While many different types of models are called GFMs, particular interest has been paid to GFMs designed for node property prediction tasks, which is one of the most popular settings in Graph ML with lots of real-world applications from fraud detection in financial and social networks to recommendation systems for e-commerce and user-generated content platforms. While a number of GFMs for this task have been recently proposed, the field has not converged to a unified evaluation setting, and different works evaluate their models in widely different ways, preventing reliable comparison of GFMs with each other and with other types of models. In this work, we conduct a fair and rigorous reevaluation of 9 recent GFMs for node property prediction, comparing them to strong Graph Neural Network (GNN) baselines. We find that, among these GFMs, only the most recent ones based on the Prior-data Fitted Networks paradigm outperform well-tuned GNNs in predictive performance, although at a higher inference cost.

• The paper demonstrates that only PFN-based GFMs significantly outperform optimized GNN baselines in node prediction tasks.
• It employs a comprehensive framework using the GraphLand benchmark to assess predictive accuracy and resource demands across ten diverse datasets.
• The study reveals that enhanced performance with PFN-based models comes with increased inference time and memory usage, emphasizing a trade-off between accuracy and efficiency.

Rigorous Evaluation of Graph Foundation Models for Node Property Prediction

Introduction

Node property prediction on graphs is an extensively studied problem in Graph Machine Learning (GML), central to applications such as fraud detection, churn prediction, recommendation systems, and traffic forecasting. While Graph Neural Networks (GNNs) have historically dominated methodological development, recent movements toward Graph Foundation Models (GFMs)—large, pre-trained models designed for transfer across diverse tasks and domains—have resulted in inconsistent evaluation practices and ambiguous claims regarding superiority to GNNs.

This paper thoroughly reevaluates nine GFMs for node property prediction, benchmarking them against meticulously tuned GNN baselines. It demonstrates that only the most recent PFN-based GFMs achieve superior predictive performance, though such gains incur significant computational overhead.

Methodological Framework

Dataset Selection and Diversity

Standard graph ML benchmarks have suffered from domain narrowness, brittle task definitions, and problematic data splits. To address these, the study employs the GraphLand benchmark, a suite of ten sub-million-node datasets spanning realistic and diverse industrial node property prediction tasks. This choice is justified by extensive discussion of known flaws in traditional datasets and the need for practical representativeness.

Model Classes

• GNN Baselines: Models include GCN, GraphSAGE, GAT, and Local Graph Transformer (LGT), each deployed with architectural enhancements such as skip-connections and normalization (layer/batch). Extensive per-dataset hyperparameter tuning is performed using TPE across 100 trials, following best practices in baseline optimization.
• GFMs: Nine GFMs are evaluated, encompassing both non-PFN-based models and PFN-based models (G2T-FM, TAG, GraphPFN). PFN-based GFMs leverage the Prior-data Fitted Networks (PFN) paradigm, originally formulated for tabular data, enabling in-context learning (ICL) for node property prediction via dataset-wide input and Transformer-based aggregation.

Evaluation Protocols

• Performance Metrics: Models are compared using average rank and normalized score across tasks. Experimental results include node classification and regression tasks.
• Computational Efficiency: Resource demands are benchmarked in terms of training, inference time, and memory (VRAM). PFN-based GFMs are also evaluated for inference-time ensembling, which boosts predictive outcomes at substantial computational cost.

Empirical Findings

Predictive Performance

Experimental evidence clearly bifurcates GFMs into two categories:

• Non-PFN-based GFMs consistently underperform relative to well-tuned GNNs.
• PFN-based GFMs (notably GraphPFN) achieve the highest predictive accuracy on all datasets, outperforming GNNs and all other GFMs by substantial margins, especially in the fine-tuning regime. Fine-tuning provides further boosts over pure in-context learning for PFN-based models.

Significantly, architectural details matter: GraphPFN, which natively incorporates graph structure in its PFN, achieves stronger performance than models (e.g., G2T-FM, TAG) that rely on graph tabularization and unmodified tabular PFN backbones.

Computational Trade-offs

PFN-based GFMs require more inference time and memory compared to GNNs. Typical PFN inference times range from seconds to tens of minutes, versus sub-second times for GNNs. These models also incur higher VRAM consumption due to tokenization and larger model sizes. While inference-time ensembling enhances prediction, it multiplies resource demands. This constitutes a major practical consideration for deployment in low-latency, industrial environments.

Conversely, PFNs offer advantages for rapid model development and adaptation: they require minimal dataset-specific engineering and—under the ICL regime—can make predictions without any retraining.

Implications and Future Directions

These results have several key implications:

• Benchmarking Practice: The lack of standardized evaluation and dataset diversity in prior GFM work means that many published GFM comparisons are unreliable. Properly tuned GNNs remain highly competitive against older GFMs, counter to mainstream assumptions.
• GFM Design Paradigms: For node property prediction, only PFN-based GFMs with graph-native architectures are presently capable of achieving statistically significant gains over GNNs. This underscores the importance of architectural adaptation, prior distribution design, and synthetic dataset generation in foundation model training.
• Computational Constraints: Inference efficiency and scalability are major barriers to PFN-based GFM adoption. Research toward scalable, distributed inference strategies or more efficient PFN designs is necessary if GFMs are to be practical for large-scale or real-time applications.
• Task and Domain Generalization: The ability of PFN-based GFMs to generalize across diverse tasks and domains is theoretically compelling, but empirical performance remains heavily shaped by prior design and resource budgets. Future work should focus on improved prior distributions, robust transfer learning methods, and expanded domain diversity.

Conclusion

A rigorous comparison between GFMs and improved GNN baselines demonstrates that only PFN-based GFMs currently outperform well-tuned GNNs for node property prediction, and only when significant computational resources are available. While PFN-based approaches set a promising direction for foundation modeling in graphs, practical constraints must be overcome to ensure effective, wide-scale deployment. Standardization in benchmarking, continued architectural innovation, and emphasis on resource-efficient modeling remain priorities for future research in GML foundation models.