GOOD: A Graph Out-of-Distribution Benchmark (2206.08452v2)

Abstract: Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences in designing shifts. Overall, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods with 10 random runs. This results in 510 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/.

• The paper introduces a novel benchmark that isolates covariate and concept shifts in graph data evaluation.
• It presents an extensive suite of 11 datasets, 17 domain selections, and 51 splits to test model generalization under OOD conditions.
• Baseline tests reveal significant performance gaps, highlighting the need for tailored graph learning methods to handle OOD challenges.

Analyzing "GOOD: A Graph Out-of-Distribution Benchmark"

The paper "GOOD: A Graph Out-of-Distribution Benchmark" addresses the emerging field of out-of-distribution (OOD) learning with a focus on graph data. Traditional OOD research has primarily concentrated on covariate and concept shifts in simpler structure models or computer vision tasks. Graph data, characterized by its irregular and connected topology, introduces unique challenges that necessitate specialized OOD methods and benchmarks.

Summary of Contributions

The authors present "GOOD," a benchmark specifically designed to evaluate graph OOD methods. This benchmark provides a comprehensive suite for testing the capability of models to generalize in situations where training and test data distributions differ. A key distinction in GOOD is its systematic approach to differentiating between covariate and concept shifts, which are explored separately across a variety of datasets and domain selections.

GOOD comprises 11 datasets with 17 domain selections and results in 51 distinct splits when combining covariate, concept, and no shifts. This extensive range enables a detailed analysis of OOD performance. The benchmark also includes baseline performance results from 10 existing OOD methods, applied across these dataset-model combinations, leading to a robust comparison framework.

Methodology

The benchmark distinguishes between covariate and concept shifts:

• Covariate Shift: Changes in input distributions while maintaining conditional distributions $P(Y|X)$. This is particularly challenging in graph data due to potential shifts in node/edge features or adjacency structures.
• Concept Shift: Variations in the conditional distribution $P(Y|X)$. This can occur amidst unchanged input distributions, posing a challenge for the model to adjust its understanding of the relationship between input features and labels.

To manage these shifts, GOOD engages in meticulous dataset splitting, ensuring that empirical evaluations faithfully reflect real-world distribution discrepancies.

Experimental Insights

The benchmark’s results indicate substantial performance differences between in-distribution (ID) and OOD scenarios, revealing the existing gaps in current OOD methodologies. Notably, methods like VREx and GroupDRO demonstrated adaptability in handling these variations, though no single method excelled universally across all scenarios. This pattern underscores the nuanced nature of graph OOD problems and the need for tailored solutions.

Implications and Future Work

The implications of this research are twofold:

1. Practical: For practitioners, the benchmark provides a set of reliable datasets and performance baselines that can guide the development of more robust graph learning methods.
2. Theoretical: The findings emphasize the complexity of graph-based OOD challenges and point to the necessity for more sophisticated models or learning strategies that can dynamically adjust to several types of distribution shifts.

As an ongoing project, GOOD anticipates expanding to include additional datasets and methods, particularly those addressing graph-specific challenges. This evolution aims to foster advancements in AI that leverage graph structures, offering insights into more generalized OOD capabilities.

In conclusion, GOOD serves as a critical tool for advancing OOD research by providing a structured means to evaluate graph learning algorithms. It spotlights the intricacies associated with graph data and sets the stage for innovations aimed at overcoming OOD challenges.