Graph Neural Networks with Learnable Structural and Positional Representations (2110.07875v2)

Abstract: Graph neural networks (GNNs) have become the standard learning architectures for graphs. GNNs have been applied to numerous domains ranging from quantum chemistry, recommender systems to knowledge graphs and natural language processing. A major issue with arbitrary graphs is the absence of canonical positional information of nodes, which decreases the representation power of GNNs to distinguish e.g. isomorphic nodes and other graph symmetries. An approach to tackle this issue is to introduce Positional Encoding (PE) of nodes, and inject it into the input layer, like in Transformers. Possible graph PE are Laplacian eigenvectors. In this work, we propose to decouple structural and positional representations to make easy for the network to learn these two essential properties. We introduce a novel generic architecture which we call LSPE (Learnable Structural and Positional Encodings). We investigate several sparse and fully-connected (Transformer-like) GNNs, and observe a performance increase for molecular datasets, from 1.79% up to 64.14% when considering learnable PE for both GNN classes.

An Overview of Graph Neural Networks with Learnable Structural and Positional Representations

Graph Neural Networks (GNNs) have emerged as a critical approach for learning and information extraction from structured data across various domains. In the paper "Graph Neural Networks with Learnable Structural and Positional Representations," the authors aim to address a central challenge in GNNs: integrating structural and positional information to create more expressive node representations, thereby improving the accuracy of these networks on complex graph-based tasks.

Decoupling Structural and Positional Features

A significant limitation of standard GNNs is their local structure dependence, which often fails to differentiate nodes in isomorphic or symmetric positions. The absence of explicit positional information limits the networks' ability in distinguishing such nodes, hindering performance on tasks like molecular property prediction. Current solutions attempt to inject positional encodings (PE) akin to those used in Transformers, but these methods often merge structural with positional features at the input phase, not considering their potential separate roles throughout the network pipeline.

This paper proposes a novel architecture named LSPE (Learnable Structural and Positional Encodings), which explicitly decouples structural and positional representations, allowing GNNs to learn these aspects independently across layers. The authors introduce a random-walk-based positional encoding as an effective and computationally feasible method to initialize positional representations, ensuring robustness against ambiguity such as sign flips seen in Laplacian eigenvectors.

Evaluation of LSPE Architectures

The researchers evaluate their LSPE model across various benchmarks, focusing particularly on molecular datasets like ZINC, where existing GNN models have struggled to outperform certain baselines. Empirical results show LSPE brings performance improvements ranging from 1.79% to 64.14% across tested datasets. Notably, sparse GNNs like GatedGCN, when equipped with decoupled LSPEs, achieve lower errors in prediction tasks, outperforming some of the state-of-the-art architectures tailored for these benchmarks.

Implications and Future Directions

The introduction of LSPE is a stepping stone toward exploiting full graph attention capabilities without sacrificing GNN efficiency. By enabling separate but integrated learning of positional information, LSPEs enhance graph embeddings' expressiveness, making them suitable for complex tasks, including knowledge graph completion and traffic flow prediction, with a linear complexity favorable for large-scale systems.

Future work could extend this framework to generalize other forms of graph-based learning tasks, including heterogeneous and dynamic graphs. Additionally, exploring alternative encoding approaches, such as more sophisticated graph diffusion methods, may bolster the task-level adaptivity and performance of GNNs under LSPE architectures.

In conclusion, this paper's findings underscore the potential of decoupling structural and positional representations in elevating GNN capabilities, aligning with ongoing efforts to build more comprehensive and intelligent graph-based models across disciplines.