Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CCFC++: Enhancing Federated Clustering through Feature Decorrelation (2402.12852v1)

Published 20 Feb 2024 in cs.LG

Abstract: In federated clustering, multiple data-holding clients collaboratively group data without exchanging raw data. This field has seen notable advancements through its marriage with contrastive learning, exemplified by Cluster-Contrastive Federated Clustering (CCFC). However, CCFC suffers from heterogeneous data across clients, leading to poor and unrobust performance. Our study conducts both empirical and theoretical analyses to understand the impact of heterogeneous data on CCFC. Findings indicate that increased data heterogeneity exacerbates dimensional collapse in CCFC, evidenced by increased correlations across multiple dimensions of the learned representations. To address this, we introduce a decorrelation regularizer to CCFC. Benefiting from the regularizer, the improved method effectively mitigates the detrimental effects of data heterogeneity, and achieves superior performance, as evidenced by a marked increase in NMI scores, with the gain reaching as high as 0.32 in the most pronounced case.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. L. Fu, H. Zhang, G. Gao, M. Zhang, and X. Liu, “Client selection in federated learning: Principles, challenges, and opportunities,” IEEE Internet of Things Journal, 2023.
  2. G. Long, M. Xie, T. Shen, T. Zhou, X. Wang, and J. Jiang, “Multi-center federated learning: clients clustering for better personalization,” World Wide Web, vol. 26, no. 1, pp. 481–500, 2023.
  3. Y. J. Cho, J. Wang, T. Chirvolu, and G. Joshi, “Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer,” IEEE Journal of Selected Topics in Signal Processing, vol. 17, no. 1, pp. 234–247, 2023.
  4. S. Zhou, H. Xu, Z. Zheng, J. Chen, J. Bu, J. Wu, X. Wang, W. Zhu, M. Ester et al., “A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions,” arXiv preprint arXiv:2206.07579, 2022.
  5. J. Yan, J. Liu, and Z.-Y. Zhang, “Ccfc: Bridging federated clustering and contrastive learning,” arXiv preprint arXiv:2401.06634, 2024.
  6. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  7. P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” Advances in neural information processing systems, vol. 32, 2019.
  8. U. Von Luxburg, “A tutorial on spectral clustering,” Statistics and computing, vol. 17, pp. 395–416, 2007.
  9. Y. Tao, K. Takagi, and K. Nakata, “Clustering-friendly representation learning via instance discrimination and feature decorrelation,” arXiv preprint arXiv:2106.00131, 2021.
  10. J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in International Conference on Machine Learning.   PMLR, 2021, pp. 12 310–12 320.
  11. T. Hua, W. Wang, Z. Xue, S. Ren, Y. Wang, and H. Zhao, “On feature decorrelation in self-supervised learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9598–9608.
  12. Y. Shi, K. Zhou, J. Liang, Z. Jiang, J. Feng, P. H. Torr, S. Bai, and V. Y. Tan, “Mimicking the oracle: An initial phase decorrelation approach for class incremental learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 722–16 731.
  13. Y. Shi, J. Liang, W. Zhang, C. Xue, V. Y. Tan, and S. Bai, “Understanding and mitigating dimensional collapse in federated learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  14. A. Strehl and J. Ghosh, “Cluster ensembles—a knowledge reuse framework for combining multiple partitions,” Journal of machine learning research, vol. 3, no. Dec, pp. 583–617, 2002.
  15. X. Liu, H.-M. Cheng, and Z.-Y. Zhang, “Evaluation of community detection methods,” IEEE Transactions on Knowledge and Data Engineering, vol. 32, no. 9, pp. 1736–1746, 2019.
  16. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020.
  17. A. E. Ezugwu, A. M. Ikotun, O. O. Oyelade, L. Abualigah, J. O. Agushaka, C. I. Eke, and A. A. Akinyelu, “A comprehensive survey of clustering algorithms: State-of-the-art machine learning applications, taxonomy, challenges, and future research prospects,” Engineering Applications of Artificial Intelligence, vol. 110, p. 104743, 2022.
  18. M. Stallmann and A. Wilbik, “Towards federated clustering: A federated fuzzy c𝑐citalic_c-means algorithm (ffcm),” in AAAI 2022 International Workshop on Trustable, Verifiable and Auditable Federated Learning, 2022.
  19. D. K. Dennis, T. Li, and V. Smith, “Heterogeneity for the win: One-shot federated clustering,” in International Conference on Machine Learning.   PMLR, 2021, pp. 2611–2620.
  20. J. Yan, J. Liu, J. Qi, and Z.-Y. Zhang, “Federated clustering with gan-based data synthesis,” arXiv preprint arXiv:2210.16524, 2022.
  21. S. Lloyd, “Least squares quantization in pcm,” IEEE transactions on information theory, vol. 28, no. 2, pp. 129–137, 1982.
  22. J. C. Bezdek, R. Ehrlich, and W. Full, “Fcm: The fuzzy c-means clustering algorithm,” Computers & geosciences, vol. 10, no. 2-3, pp. 191–203, 1984.
  23. J. Yan, J. Liu, J. Qi, and Z.-Y. Zhang, “Privacy-preserving federated deep clustering based on gan,” arXiv preprint arXiv:2211.16965, 2022.
  24. B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards k-means-friendly spaces: Simultaneous deep learning and clustering,” in International Conference on Machine Learning.   PMLR, 2017, pp. 3861–3870.
  25. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  26. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning.   PMLR, 2020, pp. 1597–1607.
  27. J. Chung, K. Lee, and K. Ramchandran, “Federated unsupervised clustering with generative models,” in AAAI 2022 International Workshop on Trustable, Verifiable and Auditable Federated Learning, 2022.
  28. S. Arora, N. Cohen, and E. Hazan, “On the optimization of deep networks: Implicit acceleration by overparameterization,” in International Conference on Machine Learning.   PMLR, 2018, pp. 244–253.
  29. S. Arora, N. Cohen, W. Hu, and Y. Luo, “Implicit regularization in deep matrix factorization,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  30. L. Jing, P. Vincent, Y. LeCun, and Y. Tian, “Understanding dimensional collapse in contrastive self-supervised learning,” arXiv preprint arXiv:2110.09348, 2021.
  31. Z. Ji and M. Telgarsky, “Gradient descent aligns the layers of deep linear networks,” arXiv preprint arXiv:1810.02032, 2018.
  32. L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets