Score Modeling Perspective in Permutation Invariant Graph Generation

Last updated: June 11, 2025

Certainly! Here is a fact-faithful, well-sourced, and stylistically polished overview of the Score Modeling Perspective grounded solely in the evidence and formulations from "Permutation Invariant Graph Generation via Score-Based Generative Modeling" (Niu et al., 2020 ° ).

1. Score-Based Generative Modeling for Graphs

The score-based generative modeling ° framework aims to learn complex probability distributions by estimating the score function °—i.e., the gradient of the log-density ° with respect to the data. For graphs, the data is most naturally represented by their adjacency matrices °.

Key definition:

$\text{Score function: }\quad \nabla_A \log p(A)$

where $A$ is the symmetric adjacency matrix representing an undirected graph.

Rather than directly modeling (or parameterizing) $p(A)$, the model learns to estimate the score of the data at different noise levels $\sigma$: $s_\theta(A; \sigma) \approx \nabla_A \log p_{\text{data}}(A)$ By estimating the score, we can define a generative process ° that stochastically transforms noise into graph samples via Langevin dynamics ° using the learned score field.

2. Enforcing Permutation Invariance

Challenge:

Graphs are combinatorial objects; any relabeling (permutation) of node indices yields an equivalent graph under a different adjacency matrix. Permutation invariance ° is the requirement that the generative model assign equal probability to all such representations.

Limitation of prior approaches:

Previous models are only approximately permutation invariant, often relying on data augmentation, with no guarantees that the generated distributions are strictly invariant to node relabelings.

Solution:

The proposed approach achieves true permutation invariance by:

• Permutation Equivariant Score Network °:

The score function itself is modeled by a GNN ° that is permutation equivariant:

$s(A^{[\pi]}) = (s(A))^{[\pi]}$

where $A^{[\pi]}$ is the adjacency matrix permuted by $\pi$, and the same permutation is applied to the score field. This ensures that permuting nodes in the input graph applies the same permutation to the output scores.

• Network architecture details:
  • Predicts edge features ° directly (not just node features).
  • Utilizes multiple edge channels (akin to feature maps in CNNs), increasing expressivity.
  • Applies message passing and feature update mechanisms ° that operate identically under any permutation of node indices.

Typical layer operations (for channel $c$ and step $m$), as outlined in the work:

for c in range(C):
    # Edge update via message passing per channel
    Z_tilde[c] = A[c] @ Z
Z_next = MLP_node(concat([Z_tilde[c][i] + (1+ϵ)*Z[i] for c in 0..C-1]))
A_tilde = MLP_edge(concat([A[:,i,j], Z_next[i], Z_next[j]]))
A_next = A_tilde + A_tilde.T  # symmetry for undirected graphs

• Guarantee:

The paper proves that if the score network is permutation equivariant, then the implicitly defined distribution is permutation invariant:

$\log p(A) = \int_{\gamma[0, A]} \langle s(X), dX \rangle_F + \log p(0)$

where the integration path is independent of node ordering.

3. Training and Graph Generation

Training:

• Minimizes the score matching ° loss at various noise levels, using symmetrically-noised versions of the adjacency matrix: [ \mathcal{L}(\theta;{\sigma_i}) = \frac{1}{2L}\sum_{i=1}^L \sigma_i² \mathbb{E}\left[ \left| s_\theta(\tilde{A}; \sigma_i) + \frac{\tilde{A} - A}{\sigma_i^2} \right|_2² \right] ] with $\tilde{A} = A + \text{Gaussian noise}$, ensuring undirectedness is preserved.
• Conditioning on noise level ($\sigma$) is done via modulation (learned gain and bias terms in each layer), allowing the network to learn robust score fields across scales.

Generation (Sampling):

• Starts from a random noisy adjacency matrix.
• For a decreasing sequence of noise levels ($\{\sigma_i\}$), applies annealed Langevin dynamics:

$\tilde{A}_t \leftarrow \tilde{A}_{t-1} + \frac{\alpha_i}{2} s_\theta(\tilde{A}_{t-1}; \sigma_i) + \sqrt{\alpha_i} z_t$

with $z_t$ Gaussian noise ° symmetrically applied.

• After the final noise level, the result is thresholded to yield a binary graph:

$ A^{\text{(sample)}}_{i,j} = \mathbbm{1}_{\tilde{A}_{i,j} > 0.5}$

4. Empirical Evaluation

Datasets:

• Community-small (graphs with planted communities)
• Ego-small (ego-networks from real datasets)

Baselines:

• GraphRNN, GNF (graph normalizing flow), GraphVAE, DeepGMG

Metrics:

• Maximum Mean Discrepancy ° (MMD °) for degree distributions, clustering coefficients, subgraph orbits.

Results:

• EDP-GNN matches or outperforms all baselines by MMD metrics on both datasets.
• For graph algorithm learning (e.g., shortest path, MST), EDP-GNN outperforms standard GNNs ° in accuracy.

Ablation studies confirm that:

• Learning edgewise features (rather than node-only) greatly improves expressiveness.
• Multi-channel edge modeling is critical for learning algorithmic tasks ° and capturing diverse distributions.

5. Practical Considerations & Implications

• Sample efficiency:

The permutation equivariant design avoids the need for data augmentation or explicit node matching at training or evaluation time.

• Applicability:

The approach is generic for undirected graphs ° and can be extended to domains like molecular generation, social, or infrastructural networks where relational invariance is crucial.

• Limitations:
  • Sampling becomes computationally expensive for large graphs due to the full adjacency matrix ° representation and Langevin steps.
  • Scaling could be addressed by integrating hierarchical architectures ° or sparse message-passing.
• Deployment:

The method is ready for integration into systems where unbiased, order-invariant graph generation ° or edge-centric learning is required.

Summary Table

In conclusion, this work demonstrates a theoretically principled and practically effective method for permutation invariant score-based generative modeling of graphs, using permutation equivariant GNNs ° to produce flexible, unbiased generative models performant on both realistic and algorithmic benchmarks °. The EDP-GNN, its loss formulation, and sampling scheme ° provide a robust blueprint for further development in structure-aware generative modeling.