Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectives (2307.02140v3)

Published 5 Jul 2023 in cs.SE, cs.AI, and cs.LG

Abstract: Traditional Federated Learning (FL) follows a server-dominated cooperation paradigm which narrows the application scenarios of FL and decreases the enthusiasm of data holders to participate. To fully unleash the potential of FL, we advocate rethinking the design of current FL frameworks and extending it to a more generalized concept: Open Federated Learning Platforms, positioned as a crowdsourcing collaborative machine learning infrastructure for all Internet users. We propose two reciprocal cooperation frameworks to achieve this: query-based FL and contract-based FL. In this survey, we conduct a comprehensive review of the feasibility of constructing open FL platforms from both technical and legal perspectives. We begin by reviewing the definition of FL and summarizing its inherent limitations, including server-client coupling, low model reusability, and non-public. In particular, we introduce a novel taxonomy to streamline the analysis of model license compatibility in FL studies that involve batch model reusing methods, including combination, amalgamation, distillation, and generation. This taxonomy provides a feasible solution for identifying the corresponding licenses clauses and facilitates the analysis of potential legal implications and restrictions when reusing models. Through this survey, we uncover the current dilemmas faced by FL and advocate for the development of sustainable open FL platforms. We aim to provide guidance for establishing such platforms in the future while identifying potential limitations that need to be addressed.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (228)
  1. S. E. Whang, Y. Roh, H. Song, and J.-G. Lee, “Data collection and quality challenges in deep learning: A data-centric ai perspective,” The VLDB Journal, pp. 1–23, 2023.
  2. O. Tene, “Privacy: The new generations,” International data privacy law, vol. 1, no. 1, pp. 15–27, 2011.
  3. P. Voigt and A. Von dem Bussche, “The EU general data protection regulation (GDPR): A practical guide,” Springer International Publishing, 2017.
  4. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine (SPM), vol. 37, no. 3, pp. 50–60, 2020.
  5. N. Truong, K. Sun, S. Wang, F. Guitton, and Y. Guo, “Privacy preservation in federated learning: An insightful survey from the gdpr perspective,” Computers & Security, vol. 110, p. 102402, 2021.
  6. K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, H. B. McMahan et al., “Towards federated learning at scale: System design,” in Proceedings of the 2nd SysML Conference, 2019.
  7. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” in NIPS 2016 workshop on Private Multi-Party Machine Learning, 2016.
  8. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017, pp. 1273–1282.
  9. C. Xu, Y. Qu, Y. Xiang, and L. Gao, “Asynchronous federated learning on heterogeneous devices: A survey,” Computer Science Review, vol. 50, p. 100595, 2023.
  10. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proceedings of the 3rd SysML Conference, 2020.
  11. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in Proceedings of the 37th International Conference on Machine Learning (ICML).   PMLR, 2020, pp. 5132–5143.
  12. Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 10 713–10 722.
  13. M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang, “Self-balancing federated learning with global imbalanced data in mobile systems,” IEEE Transactions on Parallel and Distributed Systems (TPDS), vol. 32, no. 1, pp. 59–71, 2020.
  14. F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 31, no. 9, pp. 3400–3413, 2019.
  15. L. Li, D. Liu, M. Duan, Y. Zhang, A. Ren, X. Chen, Y. Tan, and C. Wang, “Federated learning with workload-aware client scheduling in heterogeneous systems,” Neural Networks, vol. 154, pp. 560–573, 2022.
  16. K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS).   ACM, 2017, pp. 1175–1191.
  17. R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” arXiv preprint arXiv:1712.07557, 2017.
  18. K. Cheng, T. Fan, Y. Jin, Y. Liu, T. Chen, D. Papadopoulos, and Q. Yang, “SecureBoost: A lossless federated learning framework,” IEEE Intelligent Systems, vol. 36, no. 6, pp. 87–98, 2021.
  19. I. C. Society, “Ieee guide for architectural framework and application of federated machine learning,” IEEE Std 3652.1-2020, pp. 1–69, 2021.
  20. Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  21. Q. Yang, L. Fan, R. Tong, and A. Lv, “Ieee federated machine learning,” IEEE Federated Machine Learning - White Paper, pp. 1–18, 2021.
  22. M. Aledhari, R. Razzak, R. M. Parizi, and F. Saeed, “Federated learning: A survey on enabling technologies, protocols, and applications,” IEEE Access, vol. 8, pp. 140 699–140 725, 2020.
  23. P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning,” Foundations and Trends® in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021.
  24. C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y. Gao, “A survey on federated learning,” Knowledge-Based Systems (KBS), vol. 216, p. 106775, 2021.
  25. Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: vision, hype and reality for data privacy and protection,” IEEE Transactions on Knowledge and Data Engineering (TKDE), 2021.
  26. V. Kulkarni, M. Kulkarni, and A. Pant, “Survey of personalization techniques for federated learning,” in 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4).   IEEE, 2020, pp. 794–797.
  27. A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS), pp. 1–17, 2022.
  28. X. Ma, J. Zhu, Z. Lin, S. Chen, and Y. Qin, “A state-of-the-art survey on solving non-iid data in federated learning,” Future Generation Computer Systems (FGCS), vol. 135, pp. 244–258, 2022.
  29. Z. Liu, J. Guo, W. Yang, J. Fan, K.-Y. Lam, and J. Zhao, “Privacy-preserving aggregation in federated learning: A survey,” IEEE Transactions on Big Data (TBD), pp. 1–20, 2022.
  30. A. El Ouadrhiri and A. Abdelhadi, “Differential privacy for deep and federated learning: A survey,” IEEE Access, vol. 10, pp. 22 359–22 380, 2022.
  31. X. Yin, Y. Zhu, and J. Hu, “A comprehensive survey of privacy-preserving federated learning: A taxonomy, review, and future directions,” ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1–36, 2021.
  32. L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,” arXiv preprint arXiv:2003.02133, 2020.
  33. D. C. Nguyen, M. Ding, Q.-V. Pham, P. N. Pathirana, L. B. Le, A. Seneviratne, J. Li, D. Niyato, and H. V. Poor, “Federated learning meets blockchain in edge computing: Opportunities and challenges,” IEEE Internet of Things Journal (IoT-J), vol. 8, no. 16, pp. 12 806–12 825, 2021.
  34. Y. Qu, M. P. Uddin, C. Gan, Y. Xiang, L. Gao, and J. Yearwood, “Blockchain-enabled federated learning: A survey,” ACM Computing Surveys (CSUR), vol. 55, no. 4, pp. 1–35, 2022.
  35. J. Zhu, J. Cao, D. Saxena, S. Jiang, and H. Ferradi, “Blockchain-empowered federated learning: Challenges, solutions, and future directions,” ACM Computing Surveys (CSUR), 2022.
  36. C. Niu, F. Wu, S. Tang, L. Hua, R. Jia, C. Lv, Z. Wu, and G. Chen, “Billion-scale federated learning on mobile clients: A submodel design with tunable privacy,” in Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (MobiCom), 2020, pp. 1–14.
  37. S. AbdulRahman, H. Tout, A. Mourad, and C. Talhi, “Fedmccs: multicriteria client selection model for optimal iot federated learning,” IEEE Internet of Things Journal (IoT-J), vol. 8, no. 6, pp. 4723–4735, 2020.
  38. B. G. Tekgul, Y. Xia, S. Marchal, and N. Asokan, “WAFFLE: Watermarking in federated learning,” in Proceedings of the 40th International Symposium on Reliable Distributed Systems (SRDS).   IEEE, 2021, pp. 310–320.
  39. S. Shao, W. Yang, H. Gu, J. Lou, Z. Qin, L. Fan, Q. Yang, and K. Ren, “Fedtracker: Furnishing ownership verification and traceability for federated learning model,” arXiv preprint arXiv:2211.07160, 2022.
  40. H. Yu, Z. Liu, Y. Liu, T. Chen, M. Cong, X. Weng, D. Niyato, and Q. Yang, “A fairness-aware incentive scheme for federated learning,” in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2020, pp. 393–399.
  41. L. Hanzlik, Y. Zhang, K. Grosse, A. Salem, M. Augustin, M. Backes, and M. Fritz, “MLCapsule: Guarded offline deployment of machine learning as a service,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2021, pp. 3300–3309.
  42. E. Hesamifard, H. Takabi, M. Ghasemi, and R. N. Wright, “Privacy-preserving machine learning as a service,” Proceedings on Privacy Enhancing Technologies, vol. 2018, no. 3, pp. 123–142, 2018.
  43. F. Sattler, K.-R. Müller, T. Wiegand, and W. Samek, “On the byzantine robustness of clustered federated learning,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2020, pp. 8861–8865.
  44. A. Act, “Health insurance portability and accountability act of 1996,” Public law, vol. 104, p. 191, 1996.
  45. C. Dwork, “Differential privacy,” in Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33.   Springer, 2006, pp. 1–12.
  46. Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in Proceedings of the 2019 IEEE Conference on Computer Communications (INFOCOM).   IEEE, 2019, pp. 2512–2520.
  47. L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32.   Curran Associates, Inc., 2019.
  48. X. Jin, P.-Y. Chen, C.-Y. Hsu, C.-M. Yu, and T. Chen, “Cafe: Catastrophic data leakage in vertical federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 994–1006.
  49. W. Wei, L. Liu, Y. Wut, G. Su, and A. Iyengar, “Gradient-leakage resilient federated learning,” in 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS).   IEEE, 2021, pp. 797–807.
  50. Z. Li, J. Zhang, L. Liu, and J. Liu, “Auditing privacy defenses in federated learning via generative gradient leakage,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10 132–10 142.
  51. Q. Zhang, T. Wu, P. Zhou, S. Zhou, Y. Yang, and X. Jin, “Felicitas: Federated learning in distributed cross device collaborative frameworks,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 4502–4509.
  52. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” in Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2016, pp. 265–283.
  53. Y. Liu, T. Fan, T. Chen, Q. Xu, and Q. Yang, “FATE: An industrial grade platform for collaborative learning with data protection,” The Journal of Machine Learning Research (JMLR), vol. 22, no. 1, pp. 10 320–10 325, 2021.
  54. D. J. Beutel, T. Topal, A. Mathur, X. Qiu, T. Parcollet, P. P. de Gusmão, and N. D. Lane, “Flower: A friendly federated learning research framework,” arXiv preprint arXiv:2007.14390, 2020.
  55. C. He, S. Li, J. So, X. Zeng, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu et al., “FedML: A research library and benchmark for federated machine learning,” in NeurIPS 2020 Workshop on Scalability, Privacy, and Security in Federated Learning, 2020.
  56. P. Foley, M. J. Sheller, B. Edwards, S. Pati, W. Riviera, M. Sharma, P. N. Moorthy, S.-h. Wang, J. Martin, P. Mirhaji et al., “OpenFL: the open federated learning library,” Physics in Medicine & Biology, vol. 67, no. 21, p. 214001, 2022.
  57. H. R. Roth, Y. Cheng, Y. Wen, I. Yang, Z. Xu, Y.-T. Hsieh, K. Kersten, A. Harouni, C. Zhao, K. Lu et al., “Nvidia flare: Federated learning from simulation to real-world,” in NeurIPS 2022 Workshop on Federated Learning: Recent Advances and New Challenges, 2022.
  58. A. Ziller, A. Trask, A. Lopardo, B. Szymkow, B. Wagner, E. Bluemke, J.-M. Nounahon, J. Passerat-Palmbach, K. Prakash, N. Rose et al., “PySyft: A library for easy federated learning,” Federated Learning Systems: Towards Next-Generation AI, pp. 111–139, 2021.
  59. T. Zhang, L. Gao, C. He, M. Zhang, B. Krishnamachari, and A. S. Avestimehr, “Federated learning for the internet of things: applications, challenges, and opportunities,” IEEE Internet of Things Magazine (IoTM), vol. 5, no. 1, pp. 24–29, 2022.
  60. M. Alazab, S. P. RM, M. Parimala, P. K. R. Maddikunta, T. R. Gadekallu, and Q.-V. Pham, “Federated learning for cybersecurity: concepts, challenges, and future directions,” IEEE Transactions on Industrial Informatics (TII), vol. 18, no. 5, pp. 3501–3509, 2021.
  61. D. Zeng, S. Liang, X. Hu, H. Wang, and Z. Xu, “FedLab: A flexible federated learning framework,” Journal of Machine Learning Research, vol. 24, no. 100, pp. 1–7, 2023.
  62. S. Caldas, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “LEAF: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097, 2018.
  63. Y. Chen, B. Zheng, Z. Zhang, Q. Wang, C. Shen, and Q. Zhang, “Deep learning on mobile and embedded devices: State-of-the-art, challenges, and future directions,” ACM Computing Surveys (CSUR), vol. 53, no. 4, pp. 1–37, 2020.
  64. L. Li, D. Li, T. F. Bissyandé, J. Klein, Y. Le Traon, D. Lo, and L. Cavallaro, “Understanding android app piggybacking: A systematic study of malicious code grafting,” IEEE Transactions on Information Forensics and Security (TIFS), vol. 12, no. 6, pp. 1269–1284, 2017.
  65. N. Bouacida and P. Mohapatra, “Vulnerabilities in federated learning,” IEEE Access, vol. 9, pp. 63 229–63 249, 2021.
  66. J. Park, D.-J. Han, M. Choi, and J. Moon, “Sageflow: Robust federated learning against both stragglers and adversaries,” in Advances in neural information processing systems (NeurIPS), vol. 34, 2021, pp. 840–851.
  67. M. Fang, X. Cao, J. Jia, and N. Z. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in Proceedings of the 29th USENIX Conference on Security Symposium, 2020, pp. 1623–1640.
  68. E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in International Conference on Artificial Intelligence and Statistics (AISTATS).   PMLR, 2020, pp. 2938–2948.
  69. H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, and D. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 16 070–16 084.
  70. A. Reisizadeh, H. Taheri, A. Mokhtari, H. Hassani, and R. Pedarsani, “Robust and communication-efficient collaborative learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
  71. H. Ludwig, N. Baracaldo, G. Thomas, Y. Zhou, A. Anwar, S. Rajamoni, Y. Ong, J. Radhakrishnan, A. Verma, M. Sinn et al., “Ibm federated learning: an enterprise framework white paper v0. 1,” arXiv preprint arXiv:2007.10987, 2020.
  72. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 17th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, pp. 4171–4186.
  73. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al., “BLOOM: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022.
  74. S. You, C. Xu, F. Wang, and C. Zhang, “Workshop on model mining,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 4177–4178.
  75. P. Brereton, D. Budgen, K. Bennnett, M. Munro, P. Layzell, L. MaCaulay, D. Griffiths, and C. Stannett, “The future of software,” Communications of the ACM, vol. 42, no. 12, pp. 78–84, 1999.
  76. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural computation, vol. 3, no. 1, pp. 79–87, 1991.
  77. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning and Representation Learning Workshop, 2014.
  78. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  79. Y. LeCun, C. Cortes, and C. Burges, “Mnist handwritten digit database,” ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, vol. 2, 2010.
  80. S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering (TKDE), vol. 22, no. 10, pp. 1345–1359, 2009.
  81. M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” Neurocomputing, vol. 312, pp. 135–153, 2018.
  82. L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 44, no. 6, pp. 3048–3068, 2021.
  83. H. Cao, C. Tan, Z. Gao, G. Chen, P.-A. Heng, and S. Z. Li, “A survey on generative diffusion model,” arXiv preprint arXiv:2209.02646, 2022.
  84. S. Ji, T. Saravirta, S. Pan, G. Long, and A. Walid, “Emerging trends in federated learning: From model fusion to federated x learning,” arXiv preprint arXiv:2102.12920, 2021.
  85. Y. Mirsky and W. Lee, “The creation and detection of deepfakes: A survey,” ACM Computing Surveys (CSUR), vol. 54, no. 1, pp. 1–41, 2021.
  86. D. Licari and G. Comandè, “ITALIAN-LEGAL-BERT: A Pre-trained Transformer Language Model for Italian Law,” in Companion Proceedings of the 23rd International Conference on Knowledge Engineering and Knowledge Management, ser. CEUR Workshop Proceedings, vol. 3256.   Bozen-Bolzano, Italy: CEUR, Sep. 2022.
  87. F. Tang, L. Zeng, F. Wang, and J. Zhou, “Persona authentication through generative dialogue,” arXiv preprint arXiv:2110.12949, 2021.
  88. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “CodeGen: An open large language model for code with multi-turn program synthesis,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  89. L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3836–3847.
  90. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10 684–10 695.
  91. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
  92. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “StarCoder: may the source be with you!” arXiv preprint arXiv:2305.06161, 2023.
  93. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “OPT: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022.
  94. P. Goyal, Q. Duval, I. Seessel, M. Caron, M. Singh, I. Misra, L. Sagun, A. Joulin, and P. Bojanowski, “Vision models are more robust and fair when pretrained on uncurated images without supervision,” arXiv preprint arXiv:2202.08360, 2022.
  95. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “GALACTICA: A large language model for science,” arXiv preprint arXiv:2211.09085, 2022.
  96. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016, pp. 2383–2392.
  97. Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, “LayoutLMv3: Pre-training for document ai with unified text and image masking,” in Proceedings of the 30th ACM International Conference on Multimedia (MM), 2022, pp. 4083–4091.
  98. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, “LEGAL-BERT: The muppets straight out of law school,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2898–2904.
  99. Y. Peng, S. Yan, and Z. Lu, “Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets,” in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 58–65.
  100. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015.
  101. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.
  102. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  103. A. Jobin, M. Ienca, and E. Vayena, “The global landscape of ai ethics guidelines,” Nature Machine Intelligence, vol. 1, no. 9, pp. 389–399, 2019.
  104. E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan, “The moral machine experiment,” Nature, vol. 563, no. 7729, pp. 59–64, 2018.
  105. R. Yuste, S. Goering, B. A. y. Arcas, G. Bi, J. M. Carmena, A. Carter, J. J. Fins, P. Friesen, J. Gallant, J. E. Huggins et al., “Four ethical priorities for neurotechnologies and ai,” Nature, vol. 551, no. 7679, pp. 159–163, 2017.
  106. D. Contractor, D. McDuff, J. K. Haines, J. Lee, C. Hines, B. Hecht, N. Vincent, and H. Li, “Behavioral use licensing for responsible ai,” in 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2022, pp. 778–788.
  107. L. Chen, P. Koutris, and A. Kumar, “Towards model-based pricing for machine learning in a data marketplace,” in Proceedings of the 2019 International Conference on Management of Data (COMAD), 2019, pp. 1535–1552.
  108. R. W. Gomulkiewicz, “Open source license proliferation: Helpful diversity or hopeless confusion?” Washington University Journal of Law & Policy, vol. 30, no. 1, 2009.
  109. H. R. Reddy, “Jacobsen v. katzer: The federal circuit weighs in on the enforceability of free and open source software licenses ii. copyright - note,” Berkeley Technology Law Journal, vol. 24, no. 1, pp. 299–320, 2009.
  110. D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2, pp. 241–259, 1992.
  111. M. P. Perrone and L. N. Cooper, “When networks disagree: Ensemble methods for hybrid neural networks,” in How We Learn; How We Remember: Toward An Understanding Of Brain And Neural Systems: Selected Papers of Leon N Cooper.   World Scientific, 1995, pp. 342–358.
  112. B. Clarke, “Comparing bayes model averaging and stacking when model approximation error cannot be ignored,” Journal of Machine Learning Research (JMLR), vol. 4, no. Oct, pp. 683–712, 2003.
  113. Z. Wu, Q. Li, and B. He, “Practical vertical federated learning with unsupervised representation learning,” IEEE Transactions on Big Data (TBD), 2022.
  114. L. Gao, X. Ma, J. Lin, and J. Callan, “Precise zero-shot dense retrieval without relevance labels,” arXiv preprint arXiv:2212.10496, 2022.
  115. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 27 730–27 744.
  116. G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave, “Unsupervised dense information retrieval with contrastive learning,” Transactions on Machine Learning Research (TMLR), 2022.
  117. A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher et al., “Large language models generate functional protein sequences across diverse families,” Nature Biotechnology, pp. 1–8, 2023.
  118. R. Maclin, J. W. Shavlik et al., “Combining the predictions of multiple classifiers: Using competitive learning to initialize neural networks,” in Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI), vol. 95, 1995, pp. 524–531.
  119. D. Opitz and J. Shavlik, “Generating accurate and diverse members of a neural-network ensemble,” Advances in Neural Information Processing Systems (NIPS), vol. 8, 1995.
  120. P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” Advances in Neural Information Processing Systems (NIPS), vol. 30, 2017.
  121. K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” IEEE Transactions on Signal Processing (TSP), vol. 70, pp. 1142–1154, 2022.
  122. Z. Sun, P. Kairouz, A. T. Suresh, and H. B. McMahan, “Can you really backdoor federated learning?” in NeurIPS 2019 Workshop on Federated Learning for User Privacy and Data Confidentiality, 2019.
  123. A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “FedQAP: A communication-efficient federated learning method with periodic averaging and quantization,” in International Conference on Artificial Intelligence and Statistics (AISTATS).   PMLR, 2020, pp. 2021–2031.
  124. H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni, “Federated learning with matched averaging,” in Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
  125. F. Yu, W. Zhang, Z. Qin, Z. Xu, D. Wang, C. Liu, Z. Tian, and X. Chen, “Fed2: Feature-aligned federated learning,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 2066–2074.
  126. G. K. Gudur, B. S. Balaji, and S. K. Perepu, “Resource-constrained federated learning with heterogeneous labels and models,” in KDD 2020 Workshop on Artificial Intelligence of Things, 2020.
  127. Z. Qu, X. Li, R. Duan, Y. Liu, B. Tang, and Z. Lu, “Generalized federated learning via sharpness aware minimization,” in Proceedings of the 39th International Conference on Machine Learning (ICML).   PMLR, 2022, pp. 18 250–18 280.
  128. E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” in NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices, 2018.
  129. C. Jin, X. Chen, Y. Gu, and Q. Li, “FedDyn: A dynamic and efficient federated distillation approach on recommender system,” in 2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS).   IEEE, 2023, pp. 786–793.
  130. M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML).   PMLR, 2019, pp. 7252–7261.
  131. M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, and N. Hoang, “Statistical model aggregation via parameter matching,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
  132. T. C. Lam, N. Hoang, B. K. H. Low, and P. Jaillet, “Model fusion for personalized learning,” in Proceedings of the 38th International Conference on Machine Learning (ICML).   PMLR, 2021, pp. 5948–5958.
  133. S. Su, B. Li, and X. Xue, “One-shot federated learning without server-side training,” Neural Networks, vol. 164, pp. 203–215, 2023.
  134. S. Wang, X. Li, J. Sun, and Z. Xu, “Training networks in null space of feature covariance for continual learning,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 184–193.
  135. Y. Kong, L. Liu, Z. Wang, and D. Tao, “Balancing stability and plasticity through advanced null space in continual learning,” in Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI.   Springer, 2022, pp. 219–236.
  136. C.-M. Feng, B. Li, X. Xu, Y. Liu, H. Fu, and W. Zuo, “Learning federated visual prompt in null space for mri reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  137. D. Sui, Y. Chen, J. Zhao, Y. Jia, Y. Xie, and W. Sun, “FedED: Federated learning via ensemble distillation for medical relation extraction,” in Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), 2020, pp. 2118–2128.
  138. T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 2351–2363.
  139. N. Guha, A. Talwalkar, and V. Smith, “One-shot federated learning,” in NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices, 2018.
  140. H. Chen and W. Chao, “FedBE: Making bayesian model ensemble applicable to federated learning,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
  141. C. Xie, D.-A. Huang, W. Chu, D. Xu, C. Xiao, B. Li, and A. Anandkumar, “PerAda: Parameter-efficient and generalizable federated learning personalization with guarantees,” arXiv preprint arXiv:2302.06637, 2023.
  142. Y. J. Cho, A. Manoel, G. Joshi, R. Sim, and D. Dimitriadis, “Heterogeneous ensemble knowledge transfer for training large models in federated learning,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 2881–2887.
  143. X. Yao, T. Huang, C. Wu, R. Zhang, and L. Sun, “Towards faster and better federated learning: A feature fusion approach,” in 2019 IEEE International Conference on Image Processing (ICIP).   IEEE, 2019, pp. 175–179.
  144. C. Wu, F. Wu, L. Lyu, Y. Huang, and X. Xie, “Communication-efficient federated learning via knowledge distillation,” Nature communications, vol. 13, no. 1, p. 2032, 2022.
  145. G. Lee, M. Jeong, Y. Shin, S. Bae, and S.-Y. Yun, “Preservation of the global knowledge by not-true distillation in federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
  146. J. Kim, G. Kim, and B. Han, “Multi-level branched regularization for federated learning,” in Proceedings of the 39th International Conference on Machine Learning (ICML).   PMLR, 2022, pp. 11 058–11 073.
  147. Y. He, Y. Chen, X. Yang, Y. Zhang, and B. Zeng, “Class-wise adaptive self distillation for heterogeneous federated learning,” in Proceedings of the 36th AAAI Conference on Artificial Intelligence, vol. 22, 2022, pp. 12 967–12 968.
  148. M. Mendieta, T. Yang, P. Wang, M. Lee, Z. Ding, and C. Chen, “Local learning matters: Rethinking data heterogeneity in federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 8397–8406.
  149. J. Zhang, X. Zhang, X. Zhang, D. Hong, R. K. Gupta, and J. Shang, “Navigating alignment for non-identical client class sets: A label name-anchored federated learning framework,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 3297–3308.
  150. W. Jeong, J. Yoon, E. Yang, and S. J. Hwang, “Federated semi-supervised learning with inter-client consistency & disjoint learning,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
  151. S. A. Aketi and K. Roy, “Cross-feature contrastive loss for decentralized deep learning on heterogeneous data,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 12–21.
  152. U. Michieli and M. Ozay, “Prototype guided federated learning of visual feature representations,” arXiv preprint arXiv:2105.08982, 2021.
  153. Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “FedProto: Federated prototype learning across heterogeneous clients,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8432–8440.
  154. D. Jiang, C. Shan, and Z. Zhang, “Federated learning algorithm based on knowledge distillation,” in 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE).   IEEE, 2020, pp. 163–167.
  155. W. Huang, M. Ye, and B. Du, “Learn from others and be yourself in heterogeneous federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10 143–10 153.
  156. H. Jin, D. Bai, D. Yao, Y. Dai, L. Gu, C. Yu, and L. Sun, “Personalized edge intelligence via federated self-knowledge distillation,” IEEE Transactions on Parallel and Distributed Systems (TPDS), vol. 34, no. 2, pp. 567–580, 2022.
  157. X. Liang, Y. Lin, H. Fu, L. Zhu, and X. Li, “RSCFed: random sampling consensus federated semi-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10 154–10 163.
  158. Q. Li, B. He, and D. Song, “Practical one-shot federated learning for cross-silo setting,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).   International Joint Conferences on Artificial Intelligence Organization, 2021, pp. 1484–1490.
  159. D. Li and J. Wang, “FedMD: Heterogenous federated learning via model distillation,” in NeurIPS 2019 Workshop on Federated Learning for User Privacy and Data Confidentiality, 2019.
  160. X. Gong, A. Sharma, S. Karanam, Z. Wu, T. Chen, D. Doermann, and A. Innanje, “Ensemble attention distillation for privacy-preserving federated learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15 076–15 086.
  161. L. Sun and L. Lyu, “Federated model distillation with noise-free differential privacy,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).   International Joint Conferences on Artificial Intelligence Organization, 8 2021, pp. 1563–1570.
  162. F. Sattler, T. Korjakow, R. Rischke, and W. Samek, “FedAUX: Leveraging unlabeled auxiliary data in federated learning,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2021.
  163. X. Fang and M. Ye, “Robust federated learning with noisy and heterogeneous clients,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10 072–10 081.
  164. X. Gong, A. Sharma, S. Karanam, Z. Wu, T. Chen, D. Doermann, and A. Innanje, “Preserving privacy in federated learning with ensemble cross-domain knowledge distillation,” in Proceedings of the 36th AAAI Conference on Artificial Intelligence, vol. 36, no. 11, 2022, pp. 11 891–11 899.
  165. H. Chang, V. Shejwalkar, R. Shokri, and A. Houmansadr, “Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer,” in NeurIPS 2021 Workshop on New Frontiers in Federated Learning: Privacy, Fairness, Robustness, Personalization and Data Ownership, 2021.
  166. J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu, “Parameterized knowledge transfer for personalized federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 10 092–10 104.
  167. S. Itahara, T. Nishio, Y. Koda, M. Morikura, and K. Yamamoto, “Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,” IEEE Transactions on Mobile Computing (TMC), vol. 22, no. 1, pp. 191–205, 2021.
  168. J. Zhang, C. Chen, B. Li, L. Lyu, S. Wu, S. Ding, C. Shen, and C. Wu, “DENSE: Data-free one-shot federated learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
  169. C. E. Heinbaugh, E. Luz-Ricca, and H. Shao, “Data-free one-shot federated learning under very high statistical heterogeneity,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  170. Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in Proceedings of the 38th International Conference on Machine Learning (ICML).   PMLR, 2021, pp. 12 878–12 889.
  171. L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan, “Fine-tuning global model via data-free knowledge distillation for non-iid federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10 174–10 183.
  172. H. Wang, H. Zhao, Y. Wang, T. Yu, J. Gu, and J. Gao, “FedKC: Federated knowledge composition for multilingual natural language understanding,” in Proceedings of the ACM Web Conference 2022, 2022, pp. 1839–1850.
  173. C. Fan, J. Hu, and J. Huang, “Private semi-supervised federated learning,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI), 2022, pp. 2009–2015.
  174. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker et al., “Large scale distributed deep networks,” in Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS), 2012, pp. 1223–1231.
  175. Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE).   IEEE, 2022, pp. 965–978.
  176. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “FitNets: Hints for thin deep nets,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
  177. W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao, “Federated learning in mobile edge networks: A comprehensive survey,” IEEE Communications Surveys & Tutorials (COMST), vol. 22, no. 3, pp. 2031–2063, 2020.
  178. Z. Luo, Y. Wang, Z. Wang, Z. Sun, and T. Tan, “FedIris: Towards more accurate and privacy-preserving iris recognition via federated template communication,” in CVPR 2022 Workshop on Federated Learning for Computer Vision, 2022, pp. 3357–3366.
  179. T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” in Proceedings of the 35th International Conference on Machine Learning (ICML).   PMLR, 2018, pp. 1607–1616.
  180. N. Dvornik, C. Schmid, and J. Mairal, “Diversity with cooperation: Ensemble methods for few-shot classification,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3723–3731.
  181. P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split learning for health: Distributed deep learning without sharing raw patient data,” in ICLR 2019 Workshop on AI for Social Good, 2019.
  182. C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When federated learning meets split learning,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 36, no. 8, 2022, pp. 8485–8493.
  183. J. Kim, S. Shin, Y. Yu, J. Lee, and K. Lee, “Multiple classification with split learning,” in The 9th International Conference on Smart Media and Applications, 2020, pp. 358–363.
  184. M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS), 2015, pp. 1322–1333.
  185. R. G. Lopes, S. Fenu, and T. Starner, “Data-free knowledge distillation for deep neural networks,” NIPS 2017 Workshop on Learning with Limited Labeled Data: Weak Supervision and Beyond, 2017.
  186. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014.
  187. N. Shi, F. Lai, R. A. Kontar, and M. Chowdhury, “Fed-ensemble: Ensemble models in federated learning for improved generalization and uncertainty quantification,” IEEE Transactions on Automation Science and Engineering (T-ASE), 2023.
  188. N. Wang, W. Feng, M. Duan, F. Liu, S.-K. Ng et al., “Data-free diversity-based ensemble selection for one-shot federated learning,” Transactions on Machine Learning Research, 2023.
  189. C. J. Geyer, “Practical markov chain monte carlo,” Statistical Science, vol. 7, no. 4, pp. 473–483, 1992.
  190. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  191. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” Journal of artificial intelligence research (JAIR), vol. 16, no. 1, pp. 321–357, 2002.
  192. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
  193. S. C. Wong, A. Gatt, V. Stamatescu, and M. D. McDonnell, “Understanding data augmentation for classification: when to warp?” in Proceedings of the 2016 international conference on digital image computing: techniques and applications (DICTA).   IEEE, 2016, pp. 1–6.
  194. T. Yoon, S. Shin, S. J. Hwang, and E. Yang, “FedMix: Approximation of mixup under mean augmented federated learning,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
  195. K. Zhang, C. Yang, X. Li, L. Sun, and S. M. Yiu, “Subgraph federated learning with missing neighbor generation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 6671–6682.
  196. Y. Cheng, L. Zhang, and A. Li, “GFL: Federated learning on non-iid data via privacy-preserving synthetic data,” in 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom).   IEEE, 2023, pp. 61–70.
  197. W. Hao, M. El-Khamy, J. Lee, J. Zhang, K. J. Liang, C. Chen, and L. C. Duke, “Towards fair federated learning with zero-shot data augmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021, pp. 3310–3319.
  198. H. Cha, J. Park, H. Kim, S.-L. Kim, and M. Bennis, “Federated reinforcement distillation with proxy experience memory,” in IJCAI 2019 Workshop on Federated Learning for User Privacy and Data Confidentiality, 2019.
  199. S. Yu, J. Hong, H. Wang, Z. Wang, and J. Zhou, “Turning the curse of heterogeneity in federated learning into a blessing for out-of-distribution detection,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  200. M. Yang, S. Su, B. Li, and X. Xue, “Exploring one-shot semi-supervised federated learning with a pre-trained diffusion model,” arXiv preprint arXiv:2305.04063, 2023.
  201. Q. Liu, C. Chen, J. Qin, Q. Dou, and P.-A. Heng, “FedDG: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 1013–1023.
  202. R. Pi, W. Zhang, Y. Xie, J. Gao, X. Wang, S. Kim, and Q. Chen, “DynaFed: Tackling client data heterogeneity with global dynamics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 12 177–12 186.
  203. Z. Li, J. Shao, Y. Mao, J. H. Wang, and J. Zhang, “Federated learning with GAN-based data synthesis for non-IID clients,” in IJCAI 2022 Workshop on Trustworthy Federated Learning.   Springer, 2022, pp. 17–32.
  204. E. Diao, J. Ding, and V. Tarokh, “SemiFL: Semi-supervised federated learning for unlabeled clients with alternate training,” vol. 35, 2022, pp. 17 871–17 884.
  205. L. Zhang, D. Wu, and X. Yuan, “FedZKT: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models,” in Proceedings of the IEEE 42nd International Conference on Distributed Computing Systems (ICDCS).   IEEE, 2022, pp. 928–938.
  206. T. Che, Z. Zhang, Y. Zhou, X. Zhao, J. Liu, Z. Jiang, D. Yan, R. Jin, and D. Dou, “Federated fingerprint learning with heterogeneous architectures,” in 2022 IEEE International Conference on Data Mining (ICDM).   IEEE, 2022, pp. 31–40.
  207. Y. Shang, B. Duan, Z. Zong, L. Nie, and Y. Yan, “Lipschitz continuity guided knowledge distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 10 675–10 684.
  208. C. He, M. Annavaram, and S. Avestimehr, “Group knowledge transfer: Federated learning of large cnns at the edge,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33.   Curran Associates, Inc., 2020, pp. 14 068–14 080.
  209. J. Chen, J. Wang, T. Peng, Y. Sun, P. Cheng, S. Ji, X. Ma, B. Li, and D. Song, “Copy, right? a testing framework for copyright protection of deep learning models,” in IEEE Symposium on Security and Privacy (SP).   IEEE, 2022, pp. 824–841.
  210. H. Jia, H. Chen, J. Guan, A. S. Shamsabadi, and N. Papernot, “A zest of LIME: Towards architecture-independent model distances,” in Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
  211. J. W. Vaughan, “Making better use of the crowd: How crowdsourcing can advance machine learning research,” The Journal of Machine Learning Research (JMLR), vol. 18, no. 1, pp. 7026–7071, 2018.
  212. B. Yuan, Y. He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 25 464–25 477.
  213. H. Dias and N. Meratnia, “Blocklearning: A modular framework for blockchain-based vertical federated learning,” in Inernational Conference on Ubiquitous Security (UbiSec).   Springer, 2022, pp. 319–333.
  214. R. Blythman, M. Arshath, S. Vivona, J. Smékal, and H. Shaji, “Opportunities for decentralized technologies within ai hubs,” in NeurIPS 2022 Workshop on Decentralization and Trustworthy Machine Learning in Web3: Methodologies, Platforms, and Applications, 2022.
  215. Y. Deng, T. Han, and N. Zhang, “FLeX: Trading edge computing resources for federated learning via blockchain,” in IEEE Conference on Computer Communications Workshops.   IEEE, 2021, pp. 1–2.
  216. S. Guo, F. Zhang, S. Guo, S. Xu, and F. Qi, “Blockchain-assisted privacy-preserving data computing architecture for web3,” IEEE Communications Magazine, vol. 61, no. 8, pp. 28–34, 2023.
  217. Z. Batool, K. Zhang, and M. Toews, “FL-MAB: client selection and monetization for blockchain-based federated learning,” in Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing (SAC), 2022, pp. 299–307.
  218. J. Steeves, A. Shaabana, Y. Hu, F. Luus, S. T. Liu, and J. D. Tasker-Steeves, “Incentivizing intelligence: The bittensor approach,” in NeurIPS 2022 Workshop on Decentralization and Trustworthy Machine Learning in Web3: Methodologies, Platforms, and Applications, 2022.
  219. T. McConaghy, “Ocean protocol: Tools for the web3 data economy,” in Handbook on Blockchain.   Springer, 2022, pp. 505–539.
  220. M. Ryabinin and A. Gusev, “Towards crowdsourced training of large neural networks using decentralized mixture-of-experts,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 3659–3672.
  221. A. Borzunov, M. Ryabinin, T. Dettmers, Q. Lhoest, L. Saulnier, M. Diskin, Y. Jernite, and T. Wolf, “Training transformers together,” in Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track.   PMLR, 2022, pp. 335–342.
  222. M. Ryabinin, E. Gorbunov, V. Plokhotnyuk, and G. Pekhimenko, “Moshpit SGD: Communication-efficient decentralized training on heterogeneous unreliable devices,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 18 195–18 211.
  223. M. Atre, B. Jha, and A. Rao, “Distributed deep learning using volunteer computing-like paradigm,” in 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW).   IEEE, 2021, pp. 933–942.
  224. W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning,” arXiv preprint arXiv:2309.00363, 2023.
  225. Z. Qin, D. Chen, B. Qian, B. Ding, Y. Li, and S. Deng, “Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes,” arXiv preprint arXiv:2312.06353, 2023.
  226. F. Berdoz, A. Singh, M. Jaggi, and R. Raskar, “Scalable collaborative learning via representation sharing,” in NeurIPS 2022 Workshop on Decentralization and Trustworthy Machine Learning in Web3: Methodologies, Platforms, and Applications, 2022.
  227. H. Woisetschläger, A. Isenko, S. Wang, R. Mayer, and H.-A. Jacobsen, “Federated fine-tuning of llms on the very edge: The good, the bad, the ugly,” arXiv preprint arXiv:2310.03150, 2023.
  228. T. Fan, Y. Kang, G. Ma, W. Chen, W. Wei, L. Fan, and Q. Yang, “FATE-LLM: A industrial grade federated learning framework for large language models,” arXiv preprint arXiv:2310.10049, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets