Bayesian inference on random simple graphs with power law degree distributions (1702.08239v2)

Abstract: We present a model for random simple graphs with a degree distribution that obeys a power law (i.e., is heavy-tailed). To attain this behavior, the edge probabilities in the graph are constructed from Bertoin-Fujita-Roynette-Yor (BFRY) random variables, which have been recently utilized in Bayesian statistics for the construction of power law models in several applications. Our construction readily extends to capture the structure of latent factors, similarly to stochastic blockmodels, while maintaining its power law degree distribution. The BFRY random variables are well approximated by gamma random variables in a variational Bayesian inference routine, which we apply to several network datasets for which power law degree distributions are a natural assumption. By learning the parameters of the BFRY distribution via probabilistic inference, we are able to automatically select the appropriate power law behavior from the data. In order to further scale our inference procedure, we adopt stochastic gradient ascent routines where the gradients are computed on minibatches (i.e., subsets) of the edges in the graph.

• The paper introduces a Bayesian model that leverages truncated BFRY variables to naturally produce power-law degree distributions characterized by a tunable exponent.
• It employs scalable variational Bayesian inference with stochastic gradient ascent and minibatch training to efficiently estimate latent node weights and hyperparameters.
• Experiments on simulated and real-world networks demonstrate improved predictive performance over traditional Gamma-based models.

This paper introduces a Bayesian model for simple random graphs that naturally exhibits power-law degree distributions, a characteristic found in many real-world networks like social networks and biological interaction graphs. Unlike previous theoretical work that established conditions for scale-free graphs within frameworks like the generalized random graph (GRG) model, this paper provides a practical construction and a scalable inference method.

The core of the proposed model is the use of truncated Bertoin--Fujita--Roynette--Yor (BFRY) random variables to define node-specific weights, denoted as $W_i$ for node $i$. In a GRG model, the probability of an edge between nodes $i$ and $j$ ($p_{i,j}$) is determined by node parameters, often through the odds ratio $r_{i,j} = p_{i,j} / (1-p_{i,j})$. This paper sets $r_{i,j} = U_i U_j$, where $U_i = W_i / \sqrt{L}$ and $L = \sum_k W_k$.

Key aspects of the model include:

• Power-Law Generation: BFRY random variables possess heavy tails, which are essential for generating power-law degree distributions in the resulting graph. The density function of the truncated BFRY variables depends on a discount parameter $\alpha \in (0,1)$. The model proves that the graph's degree distribution follows a power law with exponent $\tau = 1 + \alpha$, allowing for exponents in the range (1, 2).
• Truncation and Sparsity: The BFRY variables are upper truncated at a value $C_n$ (dependent on the number of nodes $n$). This truncation ensures that the node weights have finite moments, a requirement for the specific GRG construction used, and allows for controlling the graph's sparsity. By setting $C_n = n^\beta$ for some $\beta > 0$, the expected number of edges scales as $O(n^{1+\beta(1-\alpha)})$, providing a mechanism to decouple power-law behavior (controlled by $\alpha$) from density (controlled by $\beta$).
• Latent Factor Extension: The model can be naturally extended to incorporate latent factors, similar to stochastic blockmodels. This is achieved by scaling the odds ratios, $r_{i,j} = A_{i,j} U_i U_j$, where $A_{i,j}$ depends on latent variables (e.g., cluster assignments). The paper shows that if $A_{i,j}$ are uniformly bounded, this scaling does not affect the asymptotic power-law degree distribution. Examples include incorporating discrete cluster assignments or mixed memberships.

For statistical inference on model parameters (the latent weights $W_i$ and hyperparameters $\alpha, \beta$), the paper proposes a variational Bayesian approach.

• Variational Inference: An approximate posterior distribution $q(W; \theta)$ over the latent weights $W$ is optimized to maximize a lower bound on the marginal likelihood (the evidence lower bound, ELBO). A mean-field assumption is made, $q(W; \theta) = \prod q(W_i; \theta_i)$, where each marginal $q(W_i; \theta_i)$ is modeled as a rectified gamma distribution (a gamma distribution truncated at $C_n$).
• Stochastic Gradient Ascent: The variational parameters $\theta$ are optimized using stochastic gradient ascent. To handle the expectation in the ELBO, a Monte Carlo approximation based on the reparameterization trick for gamma distributions is employed.
• Minibatch Training: To scale to large networks, the gradient computations are performed on minibatches of edges, approximating the full gradient over all edges. This is a standard technique from stochastic optimization applied to the network setting.
• Hyperparameter Inference: The discount parameter $\alpha$, which dictates the power-law exponent, is inferred via gradient ascent by fixing the latent weights $W$ to their mean under the current variational distribution and maximizing the likelihood with respect to $\alpha$. This requires approximating a normalization constant and its derivative numerically. The sparsity parameter $\beta$ was found difficult to infer directly and is instead selected using cross-validation in the experiments.

The experimental section demonstrates the practical utility of the model and inference procedure.

• Inference of $\alpha$ on simulated data shows that the method can effectively recover the true $\alpha$ values, although with slight overestimation in some regimes.
• Comparisons on simulated data confirm that the BFRY model, which captures the power-law structure, achieves better predictive performance (higher test log-likelihood) than a baseline GRG model using Gamma-distributed weights (which do not produce power-law tails).
• Evaluation on several real-world network datasets (US airports, openflights, polblogs, Facebook) shows that the BFRY model consistently outperforms the Gamma baseline model. The inferred $\alpha$ values provide insights into the heavy-tailedness of the degree distributions in these networks, indicating, for example, that the Facebook network analyzed has heavier tails ($\alpha \approx 0.00$) than the air traffic or political blog networks.

The paper highlights that controlling sparsity via $\beta$ is important for predictive performance and suggests that future work could focus on better inference methods for $\beta$ and implementing the latent factor extensions presented.

In summary, the paper provides a concrete, implementable Bayesian model for power-law simple graphs using truncated BFRY variables, develops a scalable variational inference algorithm leveraging stochastic gradient ascent on minibatches, and demonstrates its effectiveness on both synthetic and real-world network datasets. The model offers a way to statistically infer the power-law exponent from data and provides improved predictive performance compared to models lacking this structure.