Combining Label Propagation and Simple Models Out-performs Graph Neural Networks (2010.13993v2)

Published 27 Oct 2020 in cs.LG and cs.SI

Abstract: Graph Neural Networks (GNNs) are the predominant technique for learning over graphs. However, there is relatively little understanding of why GNNs are successful in practice and whether they are necessary for good performance. Here, we show that for many standard transductive node classification benchmarks, we can exceed or match the performance of state-of-the-art GNNs by combining shallow models that ignore the graph structure with two simple post-processing steps that exploit correlation in the label structure: (i) an "error correlation" that spreads residual errors in training data to correct errors in test data and (ii) a "prediction correlation" that smooths the predictions on the test data. We call this overall procedure Correct and Smooth (C&S), and the post-processing steps are implemented via simple modifications to standard label propagation techniques from early graph-based semi-supervised learning methods. Our approach exceeds or nearly matches the performance of state-of-the-art GNNs on a wide variety of benchmarks, with just a small fraction of the parameters and orders of magnitude faster runtime. For instance, we exceed the best known GNN performance on the OGB-Products dataset with 137 times fewer parameters and greater than 100 times less training time. The performance of our methods highlights how directly incorporating label information into the learning algorithm (as was done in traditional techniques) yields easy and substantial performance gains. We can also incorporate our techniques into big GNN models, providing modest gains. Our code for the OGB results is at https://github.com/Chillee/CorrectAndSmooth.

View on arXiv

Authors (5)

Qian Huang (55 papers)
Horace He (12 papers)
Abhay Singh (6 papers)
Ser-Nam Lim (116 papers)
Austin R. Benson (65 papers)

Citations (270)

View on Semantic Scholar

Summary

An Analysis of "Combining Label Propagation and Simple Models out-performs Graph Neural Networks"

The landscape of machine learning for graphs has been predominantly dominated by Graph Neural Networks (GNNs), which have been the focal point for practitioners aiming to harness relational data for tasks such as node classification. However, this paper challenges the necessity of complex GNN architectures by proposing a method that leverages simple models combined with classical techniques of label propagation. The paper demonstrates that these methods can match or even surpass the performance benchmarks set by state-of-the-art GNNs across various datasets.

Main Contributions

The authors present a refined pipeline, termed "Correct and Smooth" (C), which emphasizes post-processing node classification predictions through label propagation techniques. The primary components of this pipeline are as follows:

Simple Base Models: The authors utilize shallow models (such as linear models and MLPs) which are devoid of any graph structural inputs to predict initial node classifications. These models operate on node features alone and are extraordinarily fast in both training and inference phases.
Error Correction via Label Propagation: The paper introduces a novel approach to enhance the base predictions by correcting errors through a diffusion-based process that effuses predicted residual errors across the graph. This error correction leverages the positive correlation between errors on connected nodes, yielding predictions that show significantly improved accuracy.
Final Smoothing: A secondary layer of label propagation is applied to the corrected predictions, enforcing a smoothness assumption over the graph, which is supported by network theory concepts like homophily.

Performance and Efficiency

Numerically, the results are impressive. On several benchmark datasets, the proposed C framework not only outperforms GNNs but does so with a fraction of the computational resources. Particularly striking is the reduction in parameters and training time — with some models achieving over 100 times faster training times compared to complex GNN architectures like UniMP on OGB datasets.

The efficiency of the approach is enhanced by the method's capacity to directly incorporate both training and validation label information to improve inference, a strategy that is typically not feasible in GNN methodologies. According to the authors, one crucial factor contributing to this performance gain is the direct and explicit use of label information in the learning process — a facet that seems underutilized in modern GNN architectures.

Theoretical and Practical Implications

Theoretically, this paper poses an essential analysis of the fundamental assumptions behind GNNs, questioning the implied necessity of intricate graph convolutions for effective graph-based learning. The work advocates for a return to simpler, more interpretable methodologies while still achieving state-of-the-art performance, thus broadening the scope of transductive node classification tasks wherein simpler methods could suffice.

Practically, these findings suggest a paradigm shift in how researchers can approach graph learning tasks, advocating for an approach that is both scalable and efficient. This scaling facilitation opens the door to applying these methodologies to even larger graph structures where traditional GNNs can face computational bottlenecks.

Future Directions

The implementation of this C framework reveals potential avenues for future research. There is an opportunity to explore its applicability in other types of graph-based learning tasks beyond transductive node classification, such as inductive learning or link prediction. Moreover, integrating such classical label propagation techniques with cutting-edge advancements in GNN technology could yield novel hybrid models that capitalize on the strengths of both simplistic and complex paradigms.

In conclusion, the work underscores a critical insight: exploring simplicity in model design does not necessarily entail a trade-off in performance. This paper serves to remind the community of the untapped potential residing in traditional methods, particularly when combined innovatively, to enhance the efficiency and efficacy of machine learning for graphs.

PDF Markdown