Achieving Exact Cluster Recovery Threshold via Semidefinite Programming (1412.6156v2)

Abstract: The binary symmetric stochastic block model deals with a random graph of $n$ vertices partitioned into two equal-sized clusters, such that each pair of vertices is connected independently with probability $p$ within clusters and $q$ across clusters. In the asymptotic regime of $p=a \log n/n$ and $q=b \log n/n$ for fixed $a,b$ and $n \to \infty$, we show that the semidefinite programming relaxation of the maximum likelihood estimator achieves the optimal threshold for exactly recovering the partition from the graph with probability tending to one, resolving a conjecture of Abbe et al. \cite{Abbe14}. Furthermore, we show that the semidefinite programming relaxation also achieves the optimal recovery threshold in the planted dense subgraph model containing a single cluster of size proportional to $n$.

• The paper establishes that SDP relaxations achieve the information-theoretic threshold for exact cluster recovery in graph models.
• It proves that for the binary SBM, exact recovery is possible when √a – √b exceeds √2, meeting previously conjectured limits.
• The study highlights that SDP offers a computationally efficient, polynomial-time method to close the statistical-computational gap in challenging clustering tasks.

Overview of Achieving Exact Cluster Recovery Threshold via Semidefinite Programming

The paper "Achieving Exact Cluster Recovery Threshold via Semidefinite Programming" by Bruce Hajek, Yihong Wu, and Jiaming Xu addresses the challenge of exact cluster recovery in the binary symmetric stochastic block model (SBM) and the planted dense subgraph model. The authors demonstrate that semidefinite programming (SDP) relaxations offer a computationally efficient approach to achieve the information-theoretic limits of cluster recovery.

The paper focuses on two specific models within graph theory. The binary symmetric stochastic block model is concerned with partitioning a graph's vertices into two clusters, with high internal connection probability, $p$, and lower cross-cluster connection probability, $q$. The asymptotic regime considered involves $p = \frac{a \log n}{n}$ and $q = \frac{b \log n}{n}$ for fixed constants $a$ and $b$ as $n \rightarrow \infty$. The semidefinite programming relaxation of maximum likelihood estimators is shown to meet the optimal threshold for exact cluster recovery, originally conjectured but not proved until this paper.

The paper also addresses the planted dense subgraph model, where the semidefinite programming relaxation achieves the optimal recovery threshold in polynomial time. The research provides contrasting results against existing computational barriers, notably related to the planted clique problem, suggesting that the SDP relaxation remains efficient even when the cluster size scales linearly with $n$.

Key Theoretical Contributions

1. Exact Recovery Threshold: The paper establishes conditions under which exact cluster recovery is possible. For the binary symmetric stochastic block model, recovery is achievable if $\sqrt{a} - \sqrt{b} > \sqrt{2}$; otherwise, no algorithm can recover the clusters with high probability.
2. SDP Relaxation Success: It proves that the SDP relaxation of the maximum likelihood (ML) estimator meets the sharp threshold necessary for exact recovery, validating previous conjectures with compelling simulation results.
3. Generalization Across Models: The findings extend beyond the binary SBM to confirm similar threshold results for the planted dense subgraph model, emphasizing SDP's robustness in various planted model scenarios.
4. Computational and Statistical Trade-offs: The analysis highlights that the computational complexity of recovery via SDP can be polynomially bounded, bridging the gap between statistical optimality and feasible computation.

Implications and Future Directions

This paper provides significant implications for both theoretical advancements and practical applications in community detection and network science. The resolution of the recovery threshold through SDP offers a new lens to approach NP-hard problems in graph theory without compromising on recovery accuracy or efficiency.

Future research may delve further into refining the SDP algorithm to handle diverse and asymmetrical block models effectively, potentially broadening its applicability to more complex networks. Additionally, exploring the intersection between the planted models and real-world networks might reveal new insights into natural clustering phenomena and network segmentation.

The methodologies and results presented extend potential applications in signal processing, bioinformatics, and machine learning domains, where community detection plays a crucial role. Furthermore, improving understanding of computational-statistical trade-offs and developing tighter bounds on complexity and performance remain pivotal avenues for further research in exact recovery scenarios.