Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning with Only Positive Labels by Exploring Label Correlations (2404.15598v1)

Published 24 Apr 2024 in cs.LG and cs.CR

Abstract: Federated learning aims to collaboratively learn a model by using the data from multiple users under privacy constraints. In this paper, we study the multi-label classification problem under the federated learning setting, where trivial solution and extremely poor performance may be obtained, especially when only positive data w.r.t. a single class label are provided for each client. This issue can be addressed by adding a specially designed regularizer on the server-side. Although effective sometimes, the label correlations are simply ignored and thus sub-optimal performance may be obtained. Besides, it is expensive and unsafe to exchange user's private embeddings between server and clients frequently, especially when training model in the contrastive way. To remedy these drawbacks, we propose a novel and generic method termed Federated Averaging by exploring Label Correlations (FedALC). Specifically, FedALC estimates the label correlations in the class embedding learning for different label pairs and utilizes it to improve the model training. To further improve the safety and also reduce the communication overhead, we propose a variant to learn fixed class embedding for each client, so that the server and clients only need to exchange class embeddings once. Extensive experiments on multiple popular datasets demonstrate that our FedALC can significantly outperform existing counterparts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics.   PMLR, 2017, pp. 1273–1282.
  2. Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  3. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020.
  4. Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: vision, hype and reality for data privacy and protection,” IEEE Transactions on Knowledge and Data Engineering, 2021.
  5. J. Liu, W.-C. Chang, Y. Wu, and Y. Yang, “Deep learning for extreme multi-label text classification,” in Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, 2017, pp. 115–124.
  6. M. Shi, Y. Tang, and X. Zhu, “Mlne: Multi-label network embedding,” IEEE transactions on neural networks and learning systems, vol. 31, no. 9, pp. 3682–3695, 2019.
  7. F. Yu, A. S. Rawat, A. Menon, and S. Kumar, “Federated learning with only positive labels,” in International Conference on Machine Learning.   PMLR, 2020, pp. 10 946–10 956.
  8. D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification,” in Interspeech 2017, 2017.
  9. F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Processing Letters, 2018.
  10. Cao, Kai, Jain, and K. Anil, “Automated latent fingerprint recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
  11. K. Nguyen, C. Fookes, A. Ross, and S. Sridharan, “Iris recognition with off-the-shelf cnn features: A deep learning perspective,” IEEE Access, vol. 6, pp. 18 848–18 855, 2017.
  12. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017, pp. 5998–6008.
  13. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, 2016.
  14. P. Bojanowski and A. Joulin, “Unsupervised learning by predicting noise,” in International Conference on Machine Learning.   PMLR, 2017, pp. 517–526.
  15. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  16. I. Katakis, I. Vlahavas, and G. Tsoumakas, “Multilabel text classification for automated tag suggestion,” 2008.
  17. Q. Wang, N. Jia, and T. Breckon, “A baseline for multi-label image classification using ensemble deep cnn,” arXiv preprint arXiv:1811.08412, 2018.
  18. G. Tsoumakas, I. Katakis, and I. Vlahavas, “Mining multi-label data,” in Data mining and knowledge discovery handbook.   Springer, 2009, pp. 667–685.
  19. M.-L. Zhang and Z.-H. Zhou, “A review on multi-label learning algorithms,” IEEE transactions on knowledge and data engineering, vol. 26, no. 8, pp. 1819–1837, 2013.
  20. Z. Barutcuoglu, R. E. Schapire, and O. G. Troyanskaya, “Hierarchical multi-label prediction of gene function,” Bioinformatics, vol. 22, no. 7, pp. 830–836, 2006.
  21. Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, 2016, pp. 1480–1489.
  22. J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “Cnn-rnn: A unified framework for multi-label image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2285–2294.
  23. T. Gu, Z. Wang, Z. Fang, Z. Zhu, H. Yang, D. Li, and W. Du, “Multilabel convolutional network with feature denoising and details supplement,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–13, 2022.
  24. M.-K. Xie and S.-J. Huang, “Partial multi-label learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  25. ——, “Partial multi-label learning with noisy label identification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  26. Y.-Y. Sun, Y. Zhang, and Z.-H. Zhou, “Multi-label learning with weak label,” in Twenty-fourth AAAI conference on artificial intelligence, 2010.
  27. W. Liu, H. Wang, X. Shen, and I. Tsang, “The emerging trends of multi-label learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  28. H. Jain, Y. Prabhu, and M. Varma, “Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 935–944.
  29. J. Huang, C.-M. Vong, C. L. P. Chen, and Y. Zhou, “Accurate and efficient large-scale multi-label learning with reduced feature broad learning system using label correlation,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2022.
  30. Y. Xu, Q. Zhang, J. Zhang, and D. Tao, “Vitae: Vision transformer advanced by exploring intrinsic inductive bias,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  31. Q. Zhang, Y. Xu, J. Zhang, and D. Tao, “Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond,” arXiv preprint arXiv:2202.10108, 2022.
  32. L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” 2021.
  33. A. Shamsian, A. Navon, E. Fetaya, and G. Chechik, “Personalized federated learning using hypernetworks,” 2021.
  34. T. Li, S. Hu, A. Beirami, and V. Smith, “Ditto: Fair and robust federated learning through personalization,” 2020.
  35. A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–17, 2022.
  36. J. C. Duchi, M. I. Jordan, and M. J. Wainwright, “Privacy aware learning,” 2012.
  37. W. Zhu, P. Kairouz, H. Sun, B. Mcmahan, and W. Li, “Federated heavy hitters discovery with differential privacy,” 2019.
  38. Z. Li, D. Kovalev, X. Qian, and P. Richtárik, “Acceleration for compressed gradient descent in distributed and federated optimization,” 2020.
  39. D. Basu, D. Data, C. Karakus, and S. N. Diggavi, “Qsparse-local-sgd: Distributed sgd with quantization, sparsification, and local computations,” 2020.
  40. J. Xu, W. Du, Y. Jin, W. He, and R. Cheng, “Ternary compression for communication-efficient federated learning,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
  41. Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” 2021.
  42. T. Murata and T. Suzuki, “Bias-variance reduced local sgd for less heterogeneous federated learning,” 2021.
  43. H. Z. Feng, Z. You, M. Chen, T. Zhang, and W. Chen, “Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation,” 2020.
  44. F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” IEEE transactions on neural networks and learning systems, vol. 31, no. 9, pp. 3400–3413, 2019.
  45. D. K. Dennis, T. Li, and V. Smith, “Heterogeneity for the win: One-shot federated clustering,” 2021.
  46. Y. Fraboni, R. Vidal, L. Kameni, and M. Lorenzi, “Clustered sampling: Low-variance and improved representativity for clients selection in federated learning,” 2021.
  47. H. Jamali-Rad, M. Abdizadeh, and A. Singh, “Federated learning with taskonomy for non-iid data,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  48. Z. Li, Y. He, H. Yu, J. Kang, X. Li, Z. Xu, and D. Niyato, “Data heterogeneity-robust federated learning via group client selection in industrial iot,” IEEE Internet of Things Journal, 2022.
  49. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  50. S. J. Reddi, S. Kale, F. Yu, D. Holtmann-Rice, J. Chen, and S. Kumar, “Stochastic negative mining for learning with large output spaces,” in The 22nd International Conference on Artificial Intelligence and Statistics.   PMLR, 2019, pp. 1940–1949.
  51. B. Huang, X. Li, Z. Song, and X. Yang, “FL-NTK: A neural tangent kernel-based framework for federated learning analysis,” in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol. 139.   PMLR, 2021, pp. 4423–4434.
  52. R. Rivest and S. Dusse, “The md5 message-digest algorithm,” 1992.
  53. X. Wang, Y. L. Yin, and H. Yu, “Finding collisions in the full sha-1,” in Annual international cryptology conference.   Springer, 2005, pp. 17–36.
  54. S. M. Bellovin and E. K. Rescorla, “Deploying a new hash algorithm,” 2005.
  55. J.-S. Coron, Y. Dodis, C. Malinaud, and P. Puniya, “Merkle-damgård revisited: How to construct a hash function,” in Annual International Cryptology Conference.   Springer, 2005, pp. 430–448.
  56. Y. Pantazis, D. Paul, M. Fasoulakis, Y. Stylianou, and M. A. Katsoulakis, “Cumulant gan,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2022.
  57. J. Sun, A. Li, B. Wang, H. Yang, H. Li, and Y. Chen, “Soteria: Provable defense against privacy leakage in federated learning from representation perspective,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 9311–9319.
  58. C. Dwork, F. McSherry, K. Nissim, and A. D. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, ser. Lecture Notes in Computer Science, vol. 3876.   Springer, 2006, pp. 265–284.
  59. M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016.   ACM, 2016, pp. 308–318.
  60. B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, and Z. Lin, “When machine learning meets privacy: A survey and outlook,” ACM Comput. Surv., vol. 54, no. 2, pp. 31:1–31:36, 2022.
  61. H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
  62. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision.   Springer, 2014, pp. 740–755.
  63. C. Song, F. Granqvist, and K. Talwar, “FLAIR: federated learning annotated image repository,” in NeurIPS, 2022.
  64. E. L. Mencía and J. Fürnkranz, “Efficient pairwise multilabel classification for large-scale problems in the legal domain,” in ECML/PKDD, 2008.
  65. L. Guan, M. H. Alam, W. Ryu, and S. Lee, “A phrase-based model to discover hidden factors and hidden topics in recommender systems,” in 2016 International Conference on Big Data and Smart Computing, BigComp 2016, Hong Kong, China, January 18-20, 2016.   IEEE Computer Society, 2016, pp. 337–340.
  66. I. Partalas, A. Kosmopoulos, N. Baskiotis, T. Artières, G. Paliouras, É. Gaussier, I. Androutsopoulos, M. Amini, and P. Gallinari, “LSHTC: A benchmark for large-scale text classification,” arXiv, vol. abs/1503.08581, 2015.
  67. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  68. T. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” arXiv, 2019.
  69. S. Chang, B. W. Hsu, T. Chang, and V. S. Tseng, “FLAG: fast label-adaptive aggregation for multi-label classification in federated learning,” arXiv, vol. abs/2302.13571, 2023.
  70. K. Bhatia, H. Jain, P. Kar, M. Varma, and P. Jain, “Sparse local embeddings for extreme multi-label classification.” in NIPS, vol. 29, 2015, pp. 730–738.
  71. Z. Dai, C. Dun, Y. Tang, A. Kyrillidis, and A. Shrivastava, “Federated multiple label hashing (fedmlh): Communication efficient federated learning on extreme classification tasks,” arXiv, vol. abs/2110.12292, 2021.
  72. Q. Wang and K. Chen, “Multi-label zero-shot human action recognition via joint latent ranking embedding,” Neural Networks, vol. 122, pp. 1–23, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets