Scalable network reconstruction in subquadratic time (2401.01404v5)

Abstract: Network reconstruction consists in determining the unobserved pairwise couplings between $N$ nodes given only observational data on the resulting behavior that is conditioned on those couplings -- typically a time-series or independent samples from a graphical model. A major obstacle to the scalability of algorithms proposed for this problem is a seemingly unavoidable quadratic complexity of $\Omega(N^2)$, corresponding to the requirement of each possible pairwise coupling being contemplated at least once, despite the fact that most networks of interest are sparse, with a number of non-zero couplings that is only $O(N)$. Here we present a general algorithm applicable to a broad range of reconstruction problems that significantly outperforms this quadratic baseline. Our algorithm relies on a stochastic second neighbor search (Dong et al., 2011) that produces the best edge candidates with high probability, thus bypassing an exhaustive quadratic search. If we rely on the conjecture that the second-neighbor search finishes in log-linear time (Baron & Darling, 2020; 2022), we demonstrate theoretically that our algorithm finishes in subquadratic time, with a data-dependent complexity loosely upper bounded by $O(N^{3/2}\log N)$, but with a more typical log-linear complexity of $O(N\log^2N)$. In practice, we show that our algorithm achieves a performance that is many orders of magnitude faster than the quadratic baseline -- in a manner consistent with our theoretical analysis -- allows for easy parallelization, and thus enables the reconstruction of networks with hundreds of thousands and even millions of nodes and edges.

• The paper introduces a scalable algorithm that achieves subquadratic network reconstruction by leveraging a greedy coordinate descent approach tuned for sparse networks.
• The paper exploits a stochastic search for second neighbors using an approximate k-nearest neighbor search (NNDescent) to significantly reduce computational complexity.
• The paper demonstrates practical efficiency on large datasets, offering promising applications in fields like microbiomics and genomics.

Overview of the Paper: Scalable Network Reconstruction in Subquadratic Time

The paper "Scalable Network Reconstruction in Subquadratic Time," authored by Tiago P. Peixoto, addresses a fundamental challenge in network science: the reconstruction of unobserved pairwise interactions from empirical data. Conventionally, network reconstruction algorithms endure at least quadratic complexity $O(N^2)$ because each pairwise coupling between $N$ nodes must be evaluated. However, many real-world networks are sparse, containing a much smaller number of non-zero couplings proportional to $O(N)$. The author proposes an innovative algorithm capable of performing network reconstruction with a complexity that scales subquadratically with the number of nodes, aiming for a practical log-linear average complexity $O(N\log^2N)$.

Contributions and Algorithms

The central contribution of the paper is the development of a general algorithm applicable to various network reconstruction problems. This algorithm exploits a stochastic search technique for second neighbors, which efficiently prioritizes promising edge candidates. The procedure allows for bypassing the exhaustive quadratic search typical in conventional algorithms, enabling the reconstruction of networks of massive size—comprising hundreds of thousands to millions of nodes and edges.

The proposed approach advances beyond the typical coordinate descent (CD) baseline, which iteratively updates each possible pairwise coupling and suffers from at least $O(N^2)$ complexity. To achieve subquadratic complexity, the paper introduces a greedy coordinate descent (GCD) algorithm. The GCD leverages a "m-closest pairs" strategy, significantly refining the selection of potential candidates for update through an approximative $k$-nearest neighbor (KNN) search. The KNN search is approximated by the NNDescent algorithm, which sorts through a potential neighbor graph to find optimal edge candidates rapidly.

Implication and Practical Assessment

This proposed methodology marks a shift in scaling possibilities for network reconstruction. It implies that even for networks represented by a large number of nodes, the traditionally prohibitive $O(N^2)$ scaling can be surpassed, thereby extending network inference feasibility to substantially larger datasets within reasonable computation times. The empirical analyses include a performance evaluation of synthetic test cases and large-scale empirical textual datasets from domains such as microbiomics and genomics.

Future Directions and Theoretical Speculation

Future developments could focus on extending this approach to handle more complex forms of network inference, potentially involving non-convex objectives or intricate dynamics beyond the currently tested Ising models and multivariate Gaussian assumptions. Additionally, while the NNDescent algorithm is empirically robust, its theoretical limits are not fully described. Addressing these theoretical underpinnings could refine the guarantees on convergence and approximate performance, especially under distributional assumptions or sparse graph constraints.

This research opens avenues for much more scalable and efficient network reconstruction algorithms, fundamentally changing how large-scale network data can be analyzed. Further refining the algorithm's robustness and uncovering theoretical guarantees will only enhance its utility across computational network science and related interdisciplinary fields.