Provably Powerful Graph Networks (1905.11136v4)

Abstract: Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressive power of graph neural networks (GNN). It was shown that the popular message passing GNN cannot distinguish between graphs that are indistinguishable by the 1-WL test (Morris et al. 2018; Xu et al. 2019). Unfortunately, many simple instances of graphs are indistinguishable by the 1-WL test. In search for more expressive graph learning models we build upon the recent k-order invariant and equivariant graph neural networks (Maron et al. 2019a,b) and present two results: First, we show that such k-order networks can distinguish between non-isomorphic graphs as good as the k-WL tests, which are provably stronger than the 1-WL test for k>2. This makes these models strictly stronger than message passing models. Unfortunately, the higher expressiveness of these models comes with a computational cost of processing high order tensors. Second, setting our goal at building a provably stronger, simple and scalable model we show that a reduced 2-order network containing just scaled identity operator, augmented with a single quadratic operation (matrix multiplication) has a provable 3-WL expressive power. Differently put, we suggest a simple model that interleaves applications of standard Multilayer-Perceptron (MLP) applied to the feature dimension and matrix multiplication. We validate this model by presenting state of the art results on popular graph classification and regression tasks. To the best of our knowledge, this is the first practical invariant/equivariant model with guaranteed 3-WL expressiveness, strictly stronger than message passing models.

• The paper introduces k-order graph networks that match higher-order WL tests to overcome conventional GNN limitations.
• The methodology combines invariant/equivariant architectures with simplified MLP and matrix operations to achieve practical 3-WL strength.
• Empirical results show robust performance in graph classification and regression tasks on social network and molecular datasets.

Provably Powerful Graph Networks

The paper "Provably Powerful Graph Networks" by Maron et al. explores enhancing the expressive power of Graph Neural Networks (GNNs) by constructing models that surpass the limitations inherent in conventional message-passing frameworks. The authors build upon Weisfeiler-Lehman (WL) graph isomorphism tests, particularly emphasizing the restrictions of 1-WL tests for distinguishing non-isomorphic graphs and propose models leveraging higher order WL tests.

Key Contributions and Results

1. $k$-Order Graph Networks: The authors extend $k$-order invariant and equivariant networks, proving that these architectures match the discrimination power of $k$-WL tests, which surpass the expressiveness of standard message-passing GNNs for $k > 2$. While proving superior graph distinction abilities, they recognize the increased computational demands due to high-order tensor processing that complicates scalability.
2. Simple Scalable Models: Addressing the constraints of computationally intensive high-order networks, the authors introduce a simplified model. This model, a reduced 2-order network, utilizes scaled identity operators with matrix multiplication, demonstrating a 3-WL expressiveness. Crucially, this model melds traditional Multilayer Perceptrons (MLP) with matrix operations, making it the first practical invariant/equivariant model with verified 3-WL strength, moving beyond message-passing models.
3. Empirical Validation: Numerical results on various graph classification and regression tasks affirm the effectiveness of this approach, achieving competitive or superior performance compared to state-of-the-art methods. Specifically, the model showcases standout results in datasets involving social network graphs and molecular properties, such as the QM9 dataset, indicating its robustness and versatility across varying applications.

Theoretical Insights

The theoretical foundation lies in the relationship between $k$-order networks and the hierarchy of WL tests. By characterizing the representational capacity of these networks, the work fundamentally links GNN architectures to classic graph isomorphism tests, offering a structured pathway to enhance GNN's distinguishing capacity.

Furthermore, the utilization of Power-sum Multi-symmetric Polynomials (PMP) provides a novel mechanism for representing multisets within networks, enabling a refined analysis of neighborhood representations critical for the successful application of WL tests.

Practical Implications and Future Directions

This research offers a compelling alternative to traditional GNNs, particularly in scenarios where superior graph discrimination is crucial, such as in molecular chemistry or network analysis. Although the introduced models address some scalability issues, the complexity remains a concern for very large graphs.

Future work may explore optimizing computational efficiency and exploring hybrid models that balance between the depth and width of network architectures to exploit both expressive power and computational feasibility. Additionally, integrating these approaches with attention mechanisms or other modern neural network components could further enhance the models' capability to capture complex interactions in graph data.

In conclusion, this paper provides a significant theoretical and empirical framework for developing more powerful and expressive graph neural network models, paving the way for advances in graph representation learning.