Recurrent Structure-Reinforced Graph Transformer

• The paper introduces a recurrent Transformer that explicitly models edge temporal states via a two-stage process and structure-aware attention, significantly advancing dynamic link prediction.
• The methodology employs a structure-reinforced design combining global self-attention with topological and path-based feature encoding to effectively capture both local and global graph dynamics.
• Empirical results demonstrate that RSGT outperforms state-of-the-art baselines on various dynamic graph datasets by mitigating over-smoothing and incorporating historical structural cues.

The Recurrent Structure-reinforced Graph Transformer (RSGT) is a framework for discrete dynamic graph representation learning designed to capture the evolving structural and temporal properties of time-evolving graphs. It addresses limitations in previous approaches that combine recurrent neural networks (RNNs) and graph neural networks (GNNs)—notably their inability to adequately encode edge temporal states and their susceptibility to over-smoothing, which collectively hinder the modeling of dynamic node relationships and the extraction of global structural features. RSGT introduces explicit edge temporal-state modeling and an advanced structure-reinforced transformer architecture within a recurrent paradigm, enabling superior local and global feature integration over discrete graph snapshots (Hu et al., 2023).

1. Edge Temporal States Modeling

At the core of RSGT is a two-stage process for each time step $t$. First, it converts the current graph snapshot $G_t=(V_t,E_t,W_t)$ together with the previous snapshot $G_{t-1}$ into a weighted multi-relation “difference” graph $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$. Here, $\hat E_t = E_{t-1} \cup E_t$ ensures even vanished edges are considered for their residual effects. Each edge $(i,j) \in \hat E_t$ receives a temporal-type $tp_{ij}^t$ among \textbf{emerging} ($\mathcal{e}$), \textbf{persisting} ($\mathcal{p}$), or \textbf{disappearing} ($\mathcal{d}$):

$G_t=(V_t,E_t,W_t)$0

Edge weights $G_t=(V_t,E_t,W_t)$1 encode long-term interaction memory:

$G_t=(V_t,E_t,W_t)$2

where $G_t=(V_t,E_t,W_t)$3 is the consecutive persistence count and $G_t=(V_t,E_t,W_t)$4 are hyperparameters. This construction yields a multi-relation weighted graph whose topology integrates both dynamic and structural cues, addressing the insufficient edge-state modeling in prior methods.

2. Structure-reinforced Graph Transformer Design

The Structure-reinforced Graph Transformer (SGT) operates at each time step $G_t=(V_t,E_t,W_t)$5 on the current $G_t=(V_t,E_t,W_t)$6 and the previous hidden node embeddings $G_t=(V_t,E_t,W_t)$7. SGT stacks $G_t=(V_t,E_t,W_t)$8 identical encoding layers with the following components:

(a) Global Self-Attention: Standard Transformer attention is computed: $G_t=(V_t,E_t,W_t)$9 with query, key, and value projections $G_{t-1}$0, $G_{t-1}$1, $G_{t-1}$2 using learnable weights.

(b) Graph Structural Encoding: For every ordered node pair $G_{t-1}$3, two sets of features are extracted:

• Topological attributes: $G_{t-1}$4
• Temporal path features along the shortest path $G_{t-1}$5 from $G_{t-1}$6: $G_{t-1}$7, embedded and encoded with 1D convolution after positional encoding.

These are concatenated to yield $G_{t-1}$8.

(c) Structure-aware Attention Reinforcement: Raw self-attention scores $G_{t-1}$9 are modulated by an affine map dependent on $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$0: $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$1

(d) Update and Residuals: Updated node values $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$2 are produced by normalizing $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$3 and multiplying by $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$4; standard residual and feed-forward connections apply. After $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$5 layers, an outer residual is added: $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$6.

This architecture enables the transformer to capture both semantic and structure/path-aware dependencies, directly incorporating dynamic edge information into the self-attention mechanism.

3. Recurrent Learning Over Snapshots

RSGT models dynamic graph representation as a shallow recurrence across $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$7 discrete graph snapshots. With $\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$8 (initial features), the recurrence is:

$\hat G_t=(V_t,\hat E_t,TP_t, \hat W_t)$9

This sum accumulates past structural-temporal updates, allowing each $\hat E_t = E_{t-1} \cup E_t$0 to encode the full dynamic context up to snapshot $\hat E_t = E_{t-1} \cup E_t$1. The approach ensures both historical persistence and adaptation to new graph structures.

4. Training Objective, Algorithm, and Complexity

The primary supervised task is dynamic link prediction. For each candidate edge $\hat E_t = E_{t-1} \cup E_t$2 at step $\hat E_t = E_{t-1} \cup E_t$3, its feature vector is $\hat E_t = E_{t-1} \cup E_t$4, with prediction via a shallow MLP:

$\hat E_t = E_{t-1} \cup E_t$5

and binary cross-entropy loss with $\hat E_t = E_{t-1} \cup E_t$6 regularization:

$\hat E_t = E_{t-1} \cup E_t$7

The optimization uses AdamW over all parameters, including $\hat E_t = E_{t-1} \cup E_t$8 if learned.

Computational Complexity: For one snapshot, per-layer cost is $\hat E_t = E_{t-1} \cup E_t$9, dominated by attention ($(i,j) \in \hat E_t$0) and path encoding ($(i,j) \in \hat E_t$1), where $(i,j) \in \hat E_t$2 is the shortest-path length horizon and $(i,j) \in \hat E_t$3 the edge embedding dimension. Total runtime is $(i,j) \in \hat E_t$4 in the number of snapshots, and practical scalability is maintained by constraining $(i,j) \in \hat E_t$5, $(i,j) \in \hat E_t$6 (history window), and $(i,j) \in \hat E_t$7.

5. Empirical Performance and Ablation Results

RSGT has been empirically validated on four real-world dynamic graphs:

Dataset	$(i,j) \in \hat E_t$8	Edges	Train/Test
twi-Tennis	1,000	40,839	100/20
CollegeMsg	1,899	59,835	25/63
cit-HepTh	7,577	51,315	77/1
sx-MathOF	24,818	506,550	64/15

On dynamic link prediction, RSGT outperforms ten strong baselines (DeepWalk, node2vec, GraphSAGE, EvolveGCN, CoEvoSAGE, ROLAND, CTDNE, TGAT, CAW, TREND):

• twi-Tennis: Accuracy 87.6% vs TREND 74.0% (+18.3% absolute)
• CollegeMsg: 86.8% vs 74.6% (+16.4%)
• cit-HepTh: 87.2% vs 80.4% (+8.5%)
• sx-MathOF: 87.9% vs 79.8% (+10.1%)

F1 scores demonstrate commensurate improvements.

Ablation analysis confirms two architectural choices as essential: (a) explicit edge temporal-state modeling (types and weights), (b) structure-aware attention (pairwise topological and path-based features). Removal of either leads to up to 15% performance drop. RSGT maintains robustness across variations in window size, number of transformer layers, attention heads, and shortest-path horizon.

6. Significance, Limitations, and Context

RSGT addresses critical shortcomings of existing dynamic graph embedding algorithms by providing a unified, recurrent, and structure-aware Transformer architecture with explicit modeling of edge temporal states. The integration of dynamic edge types, long-term edge weights, and structure-conditioned attention sets RSGT apart regarding the quality of representations and task performance. This design mitigates GNN over-smoothing, enables extraction of global graph structure, and provides scalable procedures for graphs of moderate to large size.

By consistently outperforming contemporary baselines in dynamic link prediction and demonstrating necessary ablation-verified design advances, RSGT substantiates the importance of fine-grained temporal-state modeling and structure-aware attention in dynamic graph learning. A plausible implication is that further refinements of Transformer-based recurrent paradigms, potentially with deeper recurrence, online inference, or continuous-time extensions, could continue to advance state-of-the-art performance on evolving graph data (Hu et al., 2023).