Papers
Topics
Authors
Recent
Search
2000 character limit reached

Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning

Published 3 Jul 2024 in cs.DC | (2407.03247v1)

Abstract: This paper presents FedType, a simple yet pioneering framework designed to fill research gaps in heterogeneous model aggregation within federated learning (FL). FedType introduces small identical proxy models for clients, serving as agents for information exchange, ensuring model security, and achieving efficient communication simultaneously. To transfer knowledge between large private and small proxy models on clients, we propose a novel uncertainty-based asymmetrical reciprocity learning method, eliminating the need for any public data. Comprehensive experiments conducted on benchmark datasets demonstrate the efficacy and generalization ability of FedType across diverse settings. Our approach redefines federated learning paradigms by bridging model heterogeneity, eliminating reliance on public data, prioritizing client privacy, and reducing communication costs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. Advances in Neural Information Processing Systems, 35:29677–29690, 2022.
  2. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193, 2020.
  3. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
  4. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022.
  5. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
  6. Optimizing the collaboration structure in cross-silo federated learning. arXiv preprint arXiv:2306.06508, 2023.
  7. Predictive inference with the jackknife+. 2021.
  8. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023.
  9. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp.  401–413, 2021.
  10. Vulnerabilities in federated learning. IEEE Access, 9:63229–63249, 2021.
  11. Efficient personalized federated learning via sparse model-adaptation. arXiv preprint arXiv:2305.02776, 2023.
  12. Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer. IEEE Journal of Selected Topics in Signal Processing, 17(1):234–247, 2023.
  13. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning, pp.  2611–2620. PMLR, 2021.
  14. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
  15. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning, pp.  3329–3339. PMLR, 2021.
  16. On calibration of modern neural networks. In International conference on machine learning, pp.  1321–1330. PMLR, 2017.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  18. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  19. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  20. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10143–10153, 2022.
  21. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp.  2796–2804. PMLR, 2018.
  22. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  23. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  24. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  25. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp.  22942–22964. PMLR, 2023.
  26. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems, 2022.
  27. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp.  116–131, 2018.
  28. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
  29. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp.  15070–15092. PMLR, 2022.
  30. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  31. Neal, R. M. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
  32. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. Advances in Neural Information Processing Systems, 35:5315–5334, 2022.
  33. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13, pp.  345–356. Springer, 2002.
  34. Conformal prediction for federated uncertainty quantification under label shift. arXiv preprint arXiv:2306.05131, 2023.
  35. Platt, J. et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
  36. Semantic uncertainty intervals for disentangled latent spaces. Advances in Neural Information Processing Systems, 35:6250–6263, 2022.
  37. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
  38. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pp.  9489–9502. PMLR, 2021.
  39. Pac-bayes generalization certificates for learned inductive conformal prediction. arXiv preprint arXiv:2312.04658, 2023.
  40. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives. Frontiers of Information Technology & Electronic Engineering, 24(10):1390–1402, 2023.
  41. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  42. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
  43. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp.  6105–6114. PMLR, 2019.
  44. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  8432–8440, 2022.
  45. Data poisoning attacks against federated learning systems. In Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14–18, 2020, Proceedings, Part I 25, pp.  480–501. Springer, 2020.
  46. Vovk, V. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74:9–28, 2015.
  47. Machine-learning applications of algorithmic randomness. 1999.
  48. Algorithmic learning in a random world, volume 29. Springer, 2005.
  49. Towards personalized federated learning via heterogeneous model reassembly. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  50. Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization. arXiv preprint arXiv:2111.14655, 2021.
  51. Fedgh: Heterogeneous federated learning with generalized global header. arXiv preprint arXiv:2303.13137, 2023.
  52. Resource-aware federated learning using knowledge extraction and multi-model fusion. arXiv preprint arXiv:2208.07978, 2022.
  53. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp.  7252–7261. PMLR, 2019.
  54. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  6848–6856, 2018.
  55. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pp.  26293–26310. PMLR, 2022.
  56. Adversarial robustness through bias variance decomposition: A new perspective for federated learning. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pp.  2753–2762. ACM, 2022.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up