Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Parametric Feature Transfer: One-shot Federated Learning with Foundation Models (2402.01862v1)

Published 2 Feb 2024 in cs.LG and cs.AI

Abstract: In one-shot federated learning (FL), clients collaboratively train a global model in a single round of communication. Existing approaches for one-shot FL enhance communication efficiency at the expense of diminished accuracy. This paper introduces FedPFT (Federated Learning with Parametric Feature Transfer), a methodology that harnesses the transferability of foundation models to enhance both accuracy and communication efficiency in one-shot FL. The approach involves transferring per-client parametric models (specifically, Gaussian mixtures) of features extracted from foundation models. Subsequently, each parametric model is employed to generate synthetic features for training a classifier head. Experimental results on eight datasets demonstrate that FedPFT enhances the communication-accuracy frontier in both centralized and decentralized FL scenarios, as well as across diverse data-heterogeneity settings such as covariate shift and task shift, with improvements of up to 20.6%. Additionally, FedPFT adheres to the data minimization principle of FL, as clients do not send real features. We demonstrate that sending real features is vulnerable to potent reconstruction attacks. Moreover, we show that FedPFT is amenable to formal privacy guarantees via differential privacy, demonstrating favourable privacy-accuracy tradeoffs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Federated learning over wireless networks: Challenges and solutions. IEEE Internet of Things Journal, 2023.
  2. Dsfl: Dynamic sparsification for federated learning. In 2022 5th International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pp.  1–6. IEEE, 2022.
  3. A theory of learning from different domains. Machine learning, 79:151–175, 2010.
  4. On the opportunities and risks of foundation models. ArXiv, 2021. URL https://crfm.stanford.edu/assets/report.pdf.
  5. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp.  446–461. Springer, 2014.
  6. Vulnerabilities in federated learning. IEEE Access, 9:63229–63249, 2021.
  7. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. Membership inference attacks from first principles. In 43rd IEEE Symposium on Security and Privacy (S&P’22), 2022.
  9. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974, 2020.
  10. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
  11. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1–22, 1977.
  12. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning, pp.  2611–2620. PMLR, 2021.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. Calibrating noise to sensitivity in private data analysis. In TCC’06: Proceedings of the Third conference on Theory of Cryptography, 2006.
  15. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  16. One-shot federated learning. arXiv preprint arXiv:1902.11175, 2019.
  17. Ganspace: Discovering interpretable GAN controls. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  18. Calibrated one round federated learning with bayesian inference in the predictive space. In Proceedings of the AAAI conference on artificial intelligence, 2024.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  20. Cross domain generative augmentation: Domain generalization with latent diffusion models. arXiv preprint arXiv:2312.05387, 2023.
  21. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  22. A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428, 2022.
  23. Osgan: One-shot distributed learning using generative adversarial networks. The Journal of Supercomputing, pp.  1–21, 2023.
  24. Fusion learning: A one shot federated learning. In Computational Science–ICCS 2020: 20th International Conference, Amsterdam, The Netherlands, June 3–5, 2020, Proceedings, Part III 20, pp.  424–436. Springer, 2020.
  25. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
  26. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp.  554–561, 2013.
  27. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 55(5), 2014.
  28. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp.  5542–5550, 2017.
  29. Caltech 101, Apr 2022.
  30. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01017, 2020a.
  31. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020b.
  32. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  33. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  746–751, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
  34. DP-EM: Differentially Private Expectation Maximization. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp.  896–904. PMLR, 20–22 Apr 2017. URL https://proceedings.mlr.press/v54/park17c.html.
  35. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp.  3498–3505. IEEE, 2012.
  36. Language models are unsupervised multitask learners. 2019.
  37. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  38. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
  39. Scott, D. W. Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015.
  40. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5018–5027, 2017.
  41. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5542–5551, 2018.
  42. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101.06395, 2021.
  43. Normalization is all you need: Understanding layer-normalized federated learning under extreme label shift. arXiv preprint arXiv:2308.09565, 2023.
  44. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022.
  45. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999, 2020.
  46. Zhou, Z.-H. Open-environment machine learning. National Science Review, 9(8):nwac123, 2022.
  47. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets