Distributed Community Detection in Large Networks (2203.06509v2)
Abstract: Community detection for large networks poses challenges due to the high computational cost as well as heterogeneous community structures. In this paper, we consider widely existing real-world networks with grouped communities'' (orthe group structure''), where nodes within grouped communities are densely connected and nodes across grouped communities are relatively loosely connected. We propose a two-step community detection approach for such networks. Firstly, we leverage modularity optimization methods to partition the network into groups, where between-group connectivity is low. Secondly, we employ the stochastic block model (SBM) or degree-corrected SBM (DCSBM) to further partition the groups into communities, allowing for varying levels of between-community connectivity. By incorporating this two-step structure, we introduce a novel divide-and-conquer algorithm that asymptotically recovers both the group structure and the community structure. Numerical studies confirm that our approach significantly reduces computational costs while achieving competitive performance. This framework provides a comprehensive solution for detecting community structures in networks with grouped communities, offering a valuable tool for various applications.
- Emmanuel Abbe. Community detection and stochastic block models: recent developments. J. Mach. Learn. Res., 18:177:1–177:86, 2017.
- Pseudo-likelihood methods for community detection in large sparse networks. The Annals of Statistics, 41(4):2097–2122, 2013.
- A framework for analysis of dynamic social networks. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 523–528. ACM, 2006.
- A nonparametric view of network models and newman–girvan and other modularities. Proceedings of the National Academy of Sciences, pages pnas–0907096106, 2009.
- Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
- Stochastic blockmodels with a growing number of classes. Biometrika, 99(2):273–284, 2012.
- Finding community structure in very large networks. Physical review E, 70(6):066111, 2004.
- A mixture model for random graphs. Statistics and computing, 18(2):173–183, 2008.
- Introduction to informetrics: Quantitative methods in library, documentation and information science. Elsevier Science Publishers, 1990.
- Resolution limit in community detection. Proceedings of the National Academy of Sciences, 104(1):36–41, 2007.
- Community detection in networks: A user guide. Physics Reports, 659:1–44, 2016.
- Modularity from fluctuations in random graphs and complex networks. Physical Review E, 70(2):025101, 2004.
- Model-based clustering for social networks. Journal of the Royal Statistical Society: Series A (Statistics in Society), 170(2):301–354, 2007.
- Stochastic blockmodels: First steps. Social networks, 5(2):109–137, 1983.
- Jiashun Jin et al. Fast community detection by score. The Annals of Statistics, 43(1):57–89, 2015.
- Stochastic blockmodels and community structure in networks. Physical review E, 83(1):016107, 2011.
- Consistency of spectral clustering in stochastic block models. The Annals of Statistics, 43(1):215–237, 2015.
- SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, June 2014.
- Learning to discover social circles in ego networks. In Advances in neural information processing systems, pages 539–547, 2012.
- Recognizing objects in adversarial clutter: Breaking a visual captcha. In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, volume 1, pages I–I. IEEE, 2003.
- Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences, 103(23):8577–8582, 2006.
- Mark EJ Newman. Communities, modules and large-scale structure in networks. Nature physics, 8(1):25, 2012.
- Finding and evaluating community structure in networks. Physical review E, 69(2):026113, 2004.
- Regularized spectral clustering under the degree-corrected stochastic blockmodel. In Advances in Neural Information Processing Systems, pages 3120–3128, 2013.
- Spectral clustering and the high-dimensional stochastic blockmodel. The Annals of Statistics, 39(4):1878–1915, 2011.
- Dynamic social network analysis using latent space models. In Advances in Neural Information Processing Systems, pages 1145–1152, 2006.
- Estimation and prediction for stochastic blockmodels for graphs with latent block structure. Journal of classification, 14(1):75–100, 1997.
- Consistency of spectral clustering. The Annals of Statistics, pages 555–586, 2008.
- Fast network community detection with profile-pseudo likelihood methods. arXiv preprint arXiv:2011.00647, 2020.
- Likelihood-based model selection for stochastic block models. The Annals of Statistics, 45(2):500–528, 2017.
- A spectral clustering approach to finding communities in graphs. In Proceedings of the 2005 SIAM international conference on data mining, pages 274–285. SIAM, 2005.
- Community extraction for social networks. Proceedings of the National Academy of Sciences, 2011.
- Consistency of community detection in networks under degree-corrected stochastic block models. The Annals of Statistics, 40(4):2266–2292, 2012.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.