Supra-Laplacian Encoding for Transformer on Dynamic Graphs (2409.17986v2)

Abstract: Fully connected Graph Transformers (GT) have rapidly become prominent in the static graph community as an alternative to Message-Passing models, which suffer from a lack of expressivity, oversquashing, and under-reaching. However, in a dynamic context, by interconnecting all nodes at multiple snapshots with self-attention, GT loose both structural and temporal information. In this work, we introduce Supra-LAplacian encoding for spatio-temporal TransformErs (SLATE), a new spatio-temporal encoding to leverage the GT architecture while keeping spatio-temporal information. Specifically, we transform Discrete Time Dynamic Graphs into multi-layer graphs and take advantage of the spectral properties of their associated supra-Laplacian matrix. Our second contribution explicitly model nodes' pairwise relationships with a cross-attention mechanism, providing an accurate edge representation for dynamic link prediction. SLATE outperforms numerous state-of-the-art methods based on Message-Passing Graph Neural Networks combined with recurrent models (e.g LSTM), and Dynamic Graph Transformers, on 9 datasets. Code is available at: github.com/ykrmm/SLATE.

• The paper introduces SLATE, which leverages spectral supra-Laplacian encoding to capture both structural and temporal features for improved dynamic link prediction.
• It transforms discrete time dynamic graphs into multi-layer representations and employs a fully connected spatio-temporal transformer to model node dependencies.
• Experimental results across 11 datasets demonstrate SLATE's superior performance over baseline methods in handling high node isolation and complex temporal dynamics.

Overview of "Supra-Laplacian Encoding for Transformer on Dynamic Graphs"

The paper "Supra-Laplacian Encoding for Transformer on Dynamic Graphs" by Yannis Karmim et al. introduces an innovative method for integrating structural and temporal information in Transformers applied to dynamic graphs. By leveraging the spectral properties of supra-Laplacian matrices, the proposed SLATE (Supra-LAplacian Encoding for spatio-temporal TransformErs) model adopts a unique approach, transforming Discrete Time Dynamic Graphs (DTDGs) into multi-layer graphs. This transformation supports enhanced spatio-temporal encoding, significantly improving dynamic link prediction tasks.

Motivation and Background

Dynamic graphs represent crucial data structures in fields such as social sciences, computational biology, and recommender systems. The accurate modeling of these graphs involves predicting the formation of links over time, a task hampered by traditional Message-Passing Graph Neural Networks (MP-GNNs) due to issues like oversquashing and under-reaching. Transformers, with their fully connected attention mechanisms, have emerged as compelling alternatives in static graph contexts. However, their application to dynamic graphs has been limited due to insufficient incorporation of temporal and structural information. This limitation prompts the need for a model like SLATE, which effectively integrates these dimensions.

Methodology

Supra-Laplacian Encoding

The cornerstone of the SLATE model is the supra-Laplacian encoding, derived from the multi-layer representation of DTDGs. The first step involves transforming DTDGs into connected multi-layer graphs by removing isolated nodes and introducing virtual nodes and temporal self-connections. This process ensures the resulting supra-graph is connected, enabling a meaningful spectral analysis. The eigenvectors associated with the smallest non-zero eigenvalues of the supra-Laplacian matrix encapsulate significant spatio-temporal information, used as positional encoding.

Spatio-temporal Transformer

SLATE employs a fully connected spatio-temporal transformer, leveraging the constructed positional encoding. This transformer captures dependencies between nodes across multiple temporal steps within a defined time window, outputting dynamic node representations. These representations are subsequently refined by a cross-attention mechanism focused on node pairs, creating edge-specific embeddings critical for the link prediction task.

Experimental Results

SLATE's efficacy is validated through comprehensive experiments on 11 dynamic graph datasets. The method outperforms baseline MP-GNNs combined with recurrent models and existing dynamic graph transformers comfortably. Particularly notable is the superior performance of SLATE in datasets characterized by high node isolation and complex temporal dynamics. The experiments also underscore the efficiency of the supra-Laplacian encoding and cross-attention edge module, affirming their contributions to robust spatio-temporal modeling.

Analysis

Implications

Practically, SLATE's ability to maintain both structural and temporal graph characteristics translates into more accurate predictions in applications like social network analysis, bioinformatics, and financial forecasting. Theoretically, this approach addresses the significant challenge of integrating temporal dimensions with graph structures in transformer models, paving the way for future explorations in dynamic graph learning.

Future Directions

Future research may extend the SLATE model's applicability to larger graphs and experiment with varying transformer architectures to further optimize performance. Additionally, integrating SLATE with other graph-based encoding techniques, such as positional encoding from graph convolutional networks, may offer even richer node representations.

Conclusion

The SLATE model represents a substantive advancement in dynamic graph learning. By integrating supra-Laplacian encodings with fully connected transformer architectures, this work addresses the challenges of temporal and structural integration in dynamic graphs. Researchers and practitioners in dynamic graph analysis can look towards SLATE as a potent solution for enhancing the expressiveness and accuracy of their models.