How Attentive are Graph Attention Networks? (2105.14491v3)

Abstract: Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GAT, every node attends to its neighbors given its own representation as the query. However, in this paper we show that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node. We formally define this restricted kind of attention as static attention and distinguish it from a strictly more expressive dynamic attention. Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data. To remove this limitation, we introduce a simple fix by modifying the order of operations and propose GATv2: a dynamic graph attention variant that is strictly more expressive than GAT. We perform an extensive evaluation and show that GATv2 outperforms GAT across 11 OGB and other benchmarks while we match their parametric costs. Our code is available at https://github.com/tech-srl/how_attentive_are_gats . GATv2 is available as part of the PyTorch Geometric library, the Deep Graph Library, and the TensorFlow GNN library.

Overview of "How Attentive are Graph Attention Networks?"

In the paper titled "How Attentive are Graph Attention Networks?" by Brody, Alon, and Yahav, the authors critically examine the effectiveness of Graph Attention Networks (GATs), a prominent architecture within the broader class of Graph Neural Networks (GNNs). Despite GATs' widespread recognition as a state-of-the-art framework for representation learning on graphs, the paper exposes a fundamental limitation in their attention mechanism and introduces GATv2, a more expressive variant.

Static vs. Dynamic Attention

The crux of the paper revolves around the distinction between static and dynamic attention mechanisms. In GATs, each node updates its representation by attending to its neighbors using its representation as the query. However, the authors demonstrate that GATs essentially compute a form of static attention, where the ranking of attention scores is unconditioned on the query node. This means that the ranking of neighbors' importance is the same for all nodes, severely limiting the expressiveness of GATs.

The authors formally define static attention and compare it to dynamic attention. Dynamic attention, unlike static attention, allows the ranking of a node's neighbors to change based on the query node's representation. This flexibility is crucial for capturing more complex and nuanced relationships within graph-structured data.

The Limitation in GATs

The paper provides a theoretical analysis showing that GATs cannot compute dynamic attention due to their inherent design. Specifically, the attention scores in GATs are derived from a value that is monotonic with respect to the neighbors, making the attention mechanism unconditioned on the specific query node. This static nature was demonstrated using a synthetic problem and showed that GATs could not adequately fit the training data for even simple graph problems requiring dynamic node selection.

Introducing GATv2

To address GAT's limitations, the authors propose GATv2, a modified version of GAT that computes dynamic attention. By simply altering the order of operations within the attention mechanism—namely, applying the learned transformations after the nonlinearity instead of before—GATv2 achieves significantly higher expressiveness. The authors prove that GATv2 can compute dynamic attention theoretically and show through extensive empirical evaluation that GATv2 outperforms GAT across multiple benchmarks while maintaining similar computational complexity.

Empirical Evaluation

The authors conducted comprehensive experiments comparing GAT and GATv2 across 12 benchmarks, including node-, link-, and graph-prediction tasks. Key findings include:

• Synthetic Benchmark: In a controlled synthetic benchmark specifically designed to test the ability of a GNN to perform dynamic attention, GATv2 achieved 100% accuracy, while GAT struggled.
• Robustness to Noise: GATv2 was found to be significantly more robust to structural noise compared to GAT. This robustness was particularly evident in datasets with noisy edges, where GAT's performance degraded more rapidly.
• Graph-Prediction: In the QM9 dataset for graph property prediction, GATv2 showed an 11.5% improvement in error rates over GAT.
• Node-Prediction Tasks: Across several node-prediction tasks, GATv2 consistently outperformed GAT. For example, in the ogbn-proteins dataset, GATv2 achieved a higher ROC-AUC score.

Practical Implications and Future Work

The results suggest that the more expressive, dynamic attention mechanism of GATv2 makes it a superior choice for many real-world applications involving complex graph structures. The findings have significant implications for developing future GNN architectures and highlight the importance of reevaluating widely accepted models like GAT.

The paper’s release of GATv2 implementation, integrated into popular libraries like PyTorch Geometric and TensorFlow GNN, facilitates its adoption and further experimentation by the research community. Future research may explore additional modifications and optimizations to GATv2, as well as its application to new types of graph-structured data.

In conclusion, by rigorously identifying and addressing the limitations in GATs, this paper provides the community with a more powerful tool for graph-based learning tasks, establishing a foundation for future advancements in the domain of graph neural networks.