--- title: Discrete Graph Auto-Encoder (DGAE) Overview url: https://www.emergentmind.com/topics/discrete-graph-auto-encoder-dgae type: topic --- # Discrete Graph Auto-Encoder (DGAE) Overview Searching arXiv for recent and related papers on “Discrete Graph Auto-Encoder” and acronym ambiguities. {"query":"all:(\"Discrete Graph Auto-Encoder\" OR DGAE graph auto-encoder discrete)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} {"query":"ti:\"Discrete Graph Auto-Encoder\" OR ti:\"Variational Disentangled Graph Auto-Encoders for Link Prediction\" OR ti:\"Graph Representation Learning with Diffusion Generative Models\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} The **Discrete Graph Auto-Encoder (DGAE)** is a two-stage graph generative model that addresses a central difficulty in graph generation: a graph does not have an intrinsic node ordering, so naïve sequential models are highly sensitive to arbitrary permutations. DGAE combines two strategies that had largely been pursued separately: permutation-equivariant graph neural auto-encoding, and autoregressive sequence modeling after canonicalization. In the first stage, a permutation-equivariant auto-encoder maps a graph to a **set** of discrete latent node representations, where each node is represented by a sequence of quantized vectors. In the second stage, that latent set is sorted lexicographically and modeled with a specifically designed autoregressive Transformer. The result is a graph-to-set-to-sequence framework intended to retain permutation symmetry where it is essential, while exploiting efficient sequence modeling once the latent representation has been converted into a sortable set [2306.07735]. ## 1. Representation non-uniqueness and the generative problem A graph with \(n\) nodes admits up to \(n!\) equivalent representations under node permutation. DGAE is motivated by the observation that this non-uniqueness makes graph generation difficult: a model should not treat all of these equivalent adjacency-matrix or node-list representations as different objects. The paper frames two broad responses. One is to reduce representational ambiguity by imposing an order, typically by sorting or traversal heuristics such as Breadth-First Search. The other is to use permutation-invariant or permutation-equivariant graph neural networks, which respect graph symmetries directly [2306.07735]. The paper adopts the standard definitions. A function \(f\) is permutation-invariant if \[ f(\mathcal{G}) = x \iff f(\pi(\mathcal{G})) = x, \] and permutation-equivariant if \[ f(\pi(\mathcal{G})) = \pi(f(\mathcal{G})). \] This distinction is important because graph-level summaries should generally be invariant, whereas node-wise latent representations should permute with the input graph. DGAE treats node-level latent representations as an equivariant object in stage 1, and only later converts them into a sequence [2306.07735]. The paper’s critique of existing strategies is two-sided. Ordering-based autoregressive methods can still inherit many equivalent representations, may create long-range dependencies between graph-near but sequence-distant nodes, and can be slow at sampling. Pure GNN-based permutation-equivariant models avoid ad hoc node orderings, but standard Message Passing Neural Networks have limited receptive field, oversmoothing issues, and expressiveness bounded by the Weisfeiler–Lehman \(1\)-WL test. DGAE is positioned as a hybrid response: use GNNs to solve the graph-to-latent mapping without ordering assumptions, then sort the latent set—where sorting is easy—and learn its distribution autoregressively [2306.07735]. ## 2. Permutation-equivariant graph auto-encoding The first stage is a permutation-equivariant graph auto-encoder. The encoder maps \[ f_{\text{encoder}} : \mathbb{G} \mapsto \mathbb{Z}^h, \] from graph space to sets of continuous node embeddings, while the decoder maps \[ f_{\text{decoder}} : \mathbb{Z}^q \mapsto \mathbb{G}, \] from quantized latent node representations back to graphs. The encoder is an \(L\)-layer MPNN with edge and node updates \[ e^{l+1}_{i, j} = \text{bn}(f_{\text{edge}}([h^{l}_i, h^{l}_j, e^{l}_{i, j}])), \] \[ h^{l+1}_{i} = \text{bn}\left(h^{l}_{i} + \sum_{j \in \mathcal{N}(i)}f_{\text{node}}([h^{l}_i, h^{l}_j, e^{l}_{i, j}])\right), \] and the final continuous latent representation of node \(i\) is \[ z_i^h = h_i^L \in \mathbb{R}^{d_{\text{latent}}}. \] The graph-level latent object is therefore a set \[ \mathcal{Z}^h = \{z_1^h,\dots,z_n^h\}. \] Because the encoder is permutation-equivariant, permuting node labels in the input graph only permutes the output set elements [2306.07735]. To strengthen local structural information before encoding, the model augments node and edge features with **\(p\)-path features**. In the reported experiments \(p=3\). The augmentation introduces virtual edges between nodes connected by paths of length \(2\) to \(p\), edge features counting the number of paths of lengths \(1,\dots,p\) between endpoints, and node features counting the number of emanating paths of those lengths. The path-count matrices are \[ P_1 = A,\qquad P_2 = A^2 - D,\qquad P_3 = A^3 - AD - (D-I)A. \] Node features are augmented with \(p_j = P_j \mathbf{1}\), and edge features with the corresponding path-count vectors. This augmentation is presented as a way to compensate for the limited expressiveness of ordinary message passing [2306.07735]. The decoder reconstructs the graph from the set of quantized node latents. Because the explicit input graph structure is no longer present in the latent set, the decoder assumes a fully connected graph over latent nodes and applies another MPNN, without feature augmentation. For simple graphs, the final edge representation is a scalar and \[ \sigma(e_{i,j}^L)=p_\theta(\epsilon_{i,j}=1 \mid \mathcal{Z}^q). \] For annotated graphs, node and edge outputs are decoded by softmax: \[ \text{softmax}(h_i^L)=p_\theta(x_i \mid \mathcal{Z}^q),\qquad \text{softmax}(e_{i,j}^L)=p_\theta(e_{i,j} \mid \mathcal{Z}^q). \] For undirected graphs, symmetry is enforced by averaging the output adjacency tensor with its transpose [2306.07735]. ## 3. Discrete latent node representations The defining operation of DGAE is the conversion of continuous node embeddings into discrete node-wise codes. Each continuous latent vector \(z_i^h \in \mathbb{R}^{d_{\text{latent}}}\) is partitioned into \(C\) subvectors, \[ z_i^h = (z_{i,1}^h,\dots,z_{i,C}^h), \qquad z_{i,c}^h \in \mathbb{R}^{d_{\text{latent}}/C}, \] and each partition is quantized independently with its own codebook \(H_c \in \mathbb{R}^{m \times d_{\text{latent}}/C}\). The partitionwise quantization rule is \[ q_c(z^h_{i,c}) = z^q_{i,c} = h_{k,c} \quad \text{with} \quad k = \argmin_g \left( \|z^h_{i,c} - h_{g,c}\|_2 \right). \] Equivalently, each node may be represented by a sequence of codebook indices \[ (k_{i,1},\dots,k_{i,C}), \] with \(k_{i,c}\in\{1,\dots,m\}\) [2306.07735]. The paper gives several reasons for discretization. Graphs are discrete objects; quantization produces a latent space with known finite support; and the discrete bottleneck acts as an information bottleneck that simplifies the second-stage density model. Partitioning is used because a single very large codebook would induce an unwieldy categorical latent space. With \(C\) partitions and codebook size \(m\) per partition, the effective dictionary size per node is \[ M = m^C. \] Different codebooks are used for different partitions because no invariance across partition positions is assumed [2306.07735]. Quantization is non-differentiable, so the encoder is trained with a straight-through estimator. Codebooks are updated by an Exponential Moving Average scheme following the VQ-VAE setting, and are initialized by \(k\)-means++ after a warm-up period without quantization. A commitment loss keeps encoder outputs close to their selected codewords: \[ \mathcal{L}_{\text{commit.}} = \frac{1}{nC}\sum_{i=1}^n\sum_{c=1}^C \|z^h_{i,c} - \text{sg}[z^q_{i,c}]\|_2^2, \] where \(\text{sg}[\cdot]\) denotes stop-gradient. The appendix reports a commitment cost \(\beta = 0.25\) and loss weight \(\gamma = 0.1\) [2306.07735]. A central implication is that DGAE does not use a continuous latent prior of the usual VAE type. The first stage instead learns a permutation-equivariant discrete node-level coding of graphs, and the second stage learns the distribution of those codes directly [2306.07735]. ## 4. Sorting the latent set and the 2D-Transformer prior Once a graph has been mapped to a set of discrete node codes, the paper argues that the difficult part of canonicalization has been transferred from graphs to sets. Unlike graphs, sets can be sorted uniquely by any deterministic sorting rule. DGAE sorts the latent node index tuples \[ (k_{i,1},\dots,k_{i,C}) \] in increasing lexicographic order. This yields a sequence \[ \mathrm{K} \in \mathbb{R}^{n \times C}, \] or equivalently a sequence of codeword vectors \[ Z \in \mathbb{R}^{n \times C \times d_{\text{latent}}/C}. \] Sorting is a preprocessing step for stage 2; there is no backpropagation through sorting [2306.07735]. The prior over sorted latent sequences is factored autoregressively over two dimensions: node index \(i\) and partition index \(c\). The paper writes \[ P\left((k_{1,1}\ldots,k_{1,C}),\ldots,(k_{n,1},\ldots,k_{n,C})\right) = \prod_{i=1}^n \prod_{c=1}^C P(k_{i,c}\mid z_{