Dyn-VGAE: Dynamic Variational Graph Autoencoder

• The paper introduces a dynamic variational graph autoencoder that uses a Gaussian random walk prior to enforce temporal smoothness across graph snapshots.
• It employs a hybrid GCN-RNN encoder to capture both local graph structures and temporal dependencies efficiently.
• Empirical studies demonstrate that Dyn-VGAE outperforms static and conventional dynamic models in node classification, link prediction, and recommendation tasks.

Dyn-VGAE (Dynamic Variational Graph Autoencoder) is a generative model framework for network representation learning on evolving graphs. It extends variational graph autoencoders (VGAEs) to the dynamic setting, learning node embeddings that simultaneously reconstruct the topology of each graph snapshot and capture temporal dependencies, enabling accurate node classification, link prediction, and recommendation tasks on networks subject to change over time (Mahdavi et al., 2019, Salha-Galvan, 2022).

1. Problem Formulation and Graph Temporal Structure

Dyn-VGAE is defined over a sequence of graph snapshots $\{G^{(t)}\}_{t=1}^T$, where each $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$ comprises a node set $V^{(t)}$, edge set $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$, adjacency matrix $A^{(t)}$, and optional node feature matrix $X^{(t)}$. The primary objective is, for each time step $t$, to produce low-dimensional node embeddings $Z^{(t)}\in\mathbb{R}^{|V^{(t)}|\times d}$ that:

• Reconstruct the observed local structure $A^{(t)}$ via probabilistic decoding.
• Vary smoothly across time, modeling the graph's temporal evolution.

This setup covers various real-world scenarios such as social, communication, and transaction networks, where both structure and attributes may evolve (Mahdavi et al., 2019, Salha-Galvan, 2022).

2. Model Architecture

Dyn-VGAE augments the standard VGAE by integrating mechanisms for temporal smoothness within both the encoder and the latent variable prior.

Encoder

The encoder constructs an approximate posterior:

$$q_\phi(Z^{(t)}|A^{(t)}, X^{(t)}) = \prod_{i=1}^{N_t} \mathcal{N}(z_i^{(t)} | \mu_i^{(t)}, \operatorname{diag}(\sigma_i^{(t)2}))$$

where:

• $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$0
• $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$1

Each GCN is two-layered, with ReLU activations and symmetric normalization of the adjacency Laplacian:

$G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$2

In recent formulations, a hybrid GCN-RNN encoder is used, combining GCN-based extraction for local structures with node-wise recurrent (GRU/LSTM) units to aggregate temporal information:

• $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$3
• $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$4

Parameters of the variational distribution are then projected from the RNN state (Salha-Galvan, 2022).

Decoder

The decoder reconstructs $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$5 using an inner-product Bernoulli model:

$G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$6

3. Temporal Dependency Mechanisms

A key innovation of Dyn-VGAE is the incorporation of a temporal random walk prior over latents to capture the smooth evolution of node embeddings. At each time $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$7:

$G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$8

At $G^{(t)} = (V^{(t)}, E^{(t)}, X^{(t)})$9, an isotropic Gaussian prior is used:

$V^{(t)}$0

This random walk prior couples the latent space across time, enforcing temporal consistency. Extensions include using a learned RNN-VAE prior or including longer Markovian histories:

$V^{(t)}$1

Explicit regularization (e.g., $V^{(t)}$2) may also be included, though the KL term generally suffices (Mahdavi et al., 2019, Salha-Galvan, 2022).

4. Variational Objective and Optimization

For each snapshot, Dyn-VGAE maximizes an ELBO adapted for temporal dynamics:

$V^{(t)}$3

The total loss summed over all $V^{(t)}$4 time steps is:

$V^{(t)}$5

where $V^{(t)}$6 balances reconstruction and temporal smoothness. All terms are fully differentiable and can be evaluated via reparameterization, closed-form KL for Gaussians, and sigmoid cross-entropy (Mahdavi et al., 2019).

Optimization is performed with Adam, typically for 200 epochs and learning rates near $V^{(t)}$7 to $V^{(t)}$8 (Mahdavi et al., 2019, Salha-Galvan, 2022).

5. Training Workflow and Implementation

The canonical training loop is as follows:

• For each epoch:
  • For $V^{(t)}$9:
  • Compute Laplacian-normalized adjacency $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$0.
  • Extract spatial features via parallel GCNs for $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$1 and $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$2 or GCN+RNN architectures.
  • Infer variational parameters and sample $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$3 via the reparameterization trick.
  • Decode the adjacency, compute per-timestep reconstruction loss and temporal KL.
  • Accumulate loss over all $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$4 and perform joint backpropagation.

GCN parameters $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$5, $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$6 can either be shared across $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$7 or snapshot-specific; latent codes are tied only via the KL term. For large graphs, subgraph sampling/minibatch decoding may be employed for scalability (Salha-Galvan, 2022). Parallelism across $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$8 is possible, with dependencies on $E^{(t)} \subseteq V^{(t)}\times V^{(t)}$9 potentially enabling alternating or round-robin update schemes (Mahdavi et al., 2019).

6. Empirical Performance and Benchmarks

Comprehensive studies demonstrate the superiority of Dyn-VGAE over static and conventional dynamic baselines:

• Node Classification: Dyn-VGAE outperforms DeepWalk, node2vec, and static VGAE by 3–10 F1 points on streaming co-authorship graphs.
• Dynamic Link Prediction: On evolving citation and social networks (e.g., Hep-Th, AS, St-Ov, Enron, UCI-social), Dyn-VGAE exceeds static VGAE and random-walk methods by 3–5 AUC points and competes with advanced temporal GNNs (JODIE, DyRep, EvolveGCN, TGN) with lower parameter counts.
• Recommendation: In top-$A^{(t)}$0 co-author recommendation, Dyn-VGAE yields higher Precision@$A^{(t)}$1/Recall@$A^{(t)}$2 across all tested values of $A^{(t)}$3.

Scalability is similar to that of static VGAE, scaling linearly in $A^{(t)}$4 and $A^{(t)}$5 per epoch, and is notably faster than RNN/LSTM-heavy architectures such as dynAERNN (Mahdavi et al., 2019, Salha-Galvan, 2022).

7. Extensions, Limitations, and Research Directions

Strengths:

• Joint temporal learning without the need for post-hoc alignment.
• Gaussian random walk prior offers simple, effective temporal regularization.
• Applicable to attributed, directed, and multi-modal graphs with decoder modifications.
• ELBO-based training ensures balanced learning between local reconstruction and global temporal smoothness.

Limitations:

• Assumes large overlap in node set $A^{(t)}$6 across time; birth/death of nodes requires explicit padding or inductive GCN variants.
• Temporal prior is Markovian and Gaussian, limiting expressivity for non-Gaussian or long-range dynamics.
• Selection of hyperparameters ($A^{(t)}$7, history length $A^{(t)}$8, prior variance) is dataset-dependent.

Potential Extensions:

• Replace the Gaussian prior with a learned RNN-VAE prior $A^{(t)}$9 for more complex temporal dependencies.
• Incorporate edge or node attributes, model heterogeneity, enable multi-modal feature fusion.
• Adapt to very large-scale or distributed training using multi-GPU or parameter-server architectures.
• Apply gravity-inspired or MLP-based decoders for additional flexibility (Salha-Galvan, 2022).

• "Dynamic Joint Variational Graph Autoencoders" (Mahdavi et al., 2019)
• "Contributions to Representation Learning with Graph Autoencoders and Applications to Music Recommendation" (Salha-Galvan, 2022)