Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Few-shot Learning (2306.10234v3)

Published 17 Jun 2023 in cs.LG and cs.DC

Abstract: Federated Learning (FL) enables multiple clients to collaboratively learn a machine learning model without exchanging their own local data. In this way, the server can exploit the computational power of all clients and train the model on a larger set of data samples among all clients. Although such a mechanism is proven to be effective in various fields, existing works generally assume that each client preserves sufficient data for training. In practice, however, certain clients may only contain a limited number of samples (i.e., few-shot samples). For example, the available photo data taken by a specific user with a new mobile device is relatively rare. In this scenario, existing FL efforts typically encounter a significant performance drop on these clients. Therefore, it is urgent to develop a few-shot model that can generalize to clients with limited data under the FL scenario. In this paper, we refer to this novel problem as federated few-shot learning. Nevertheless, the problem remains challenging due to two major reasons: the global data variance among clients (i.e., the difference in data distributions among clients) and the local data insufficiency in each client (i.e., the lack of adequate local data for training). To overcome these two challenges, we propose a novel federated few-shot learning framework with two separately updated models and dedicated training strategies to reduce the adverse impact of global data variance and local data insufficiency. Extensive experiments on four prevalent datasets that cover news articles and images validate the effectiveness of our framework compared with the state-of-the-art baselines. Our code is provided at https://github.com/SongW-SW/F2L.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Federated learning with personalization layers. arXiv:1912.00818 (2019).
  2. Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In IJCNN.
  3. Federated user representation learning. arXiv:1909.12535 (2019).
  4. Soumen Chakrabarti. 2002. Mining the Web: Discovering knowledge from hypertext data. Morgan Kaufmann.
  5. Michael Chau and Hsinchun Chen. 2008. A machine learning approach to web page filtering using content and structure analysis. Decision Support Systems 44, 2 (2008), 482–494.
  6. Federated meta-learning with fast convergence and efficient communication. arXiv:1802.07876 (2018).
  7. Exploiting shared representations for personalized federated learning. In ICLR.
  8. Imagenet: A large-scale hierarchical image database. In CVPR.
  9. Graph prototypical networks for few-shot learning on attributed networks. In CIKM.
  10. Few-shot network anomaly detection via cross-network meta-learning. In TheWebConf.
  11. Few-shot learning via learning the representation, provably. arXiv:2002.09434 (2020).
  12. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems (2020).
  13. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In NeurIPS.
  14. Chenyou Fan and Jianwei Huang. 2021. Federated Few-Shot Learning with Adversarial Learning. In WiOpt.
  15. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML.
  16. Federated graph machine learning: A survey of concepts, techniques, and applications. ACM SIGKDD Explorations Newsletter (2022).
  17. Generative Adversarial Nets. In NIPS.
  18. Momentum contrast for unsupervised visual representation learning. In CVPR.
  19. Distilling the knowledge in a neural network. arXiv:1503.02531 2, 7 (2015).
  20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation (1997).
  21. Measuring the effects of non-identical data distribution for federated visual classification. arXiv:1909.06335 (2019).
  22. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning (2021).
  23. Scaffold: Stochastic controlled averaging for federated learning. In ICML.
  24. Adaptive gradient-based meta-learning methods. NeurIPS (2019).
  25. Supervised contrastive learning. In NeurIPS.
  26. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  27. Learning multiple layers of features from tiny images. (2009).
  28. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995.
  29. Meta-learning with differentiable convex optimization. In CVPR.
  30. Daliang Li and Junpu Wang. 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv:1910.03581 (2019).
  31. A survey on federated learning systems: vision, hype and reality for data privacy and protection. TKDE (2021).
  32. Federated optimization in heterogeneous networks. In MLSys.
  33. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials (2020).
  34. Bing Liu. 2011. Web data mining: exploring hyperlinks, contents, and usage data. Vol. 1. Springer.
  35. Learning to propagate for graph meta-learning. In NeurIPS.
  36. Communication-efficient learning of deep networks from decentralized data. In AISTATS.
  37. A Simple Neural Attentive Meta-Learner. In ICLR.
  38. Rishabh Misra. 2018. News category dataset. DOI: DOI: https://doi. org/10.13140/RG 2, 20331.18729 (2018).
  39. Rishabh Misra and Jigyasa Grover. 2021. Sculpting Data for ML: The first act of Machine Learning.
  40. Representation learning with contrastive predictive coding. In arXiv:1807.03748.
  41. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS.
  42. Sachin Ravi and Hugo Larochelle. 2016. Optimization as a model for few-shot learning. In ICLR.
  43. Few-shot learning with embedded class models and shot-free meta training. In CVPR.
  44. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR.
  45. Prototypical networks for few-shot learning. In NeurIPS.
  46. Meta-transfer learning for few-shot learning. In CVPR.
  47. Learning to compare: relation network for few-shot learning. In CVPR.
  48. Personalized federated learning with moreau envelopes. NeurIPS (2020).
  49. Towards personalized federated learning. IEEE TNNLS (2022).
  50. Transductive Linear Probing: A Novel Framework for Few-Shot Node Classification. arXiv:2212.05606 (2022).
  51. Rethinking few-shot image classification: a good embedding is all you need? ECCV (2020).
  52. Attention is all you need. In NeurIPS.
  53. Matching networks for one shot learning. In NeurIPS.
  54. Graph Few-shot Learning with Task-specific Structures. In NeurIPS.
  55. Task-adaptive few-shot node classification. In SIGKDD.
  56. FAITH: Few-Shot Graph Classification with Hierarchical Task Graphs. In IJCAI.
  57. REFORM: Error-Aware Few-Shot Knowledge Graph Completion. In CIKM.
  58. Web mining and social networking: techniques and applications. Vol. 6. Springer Science & Business Media.
  59. Federated machine learning: Concept and applications. ACM TIST (2019).
  60. Meta-Learning with Fewer Tasks through Task Interpolation. In ICLR.
  61. Salvaging federated learning by local adaptation. arXiv:2002.04758 (2020).
  62. Qingyu Zhang and Richard S Segall. 2008. Web mining: a survey of current research, techniques, and software. International Journal of Information Technology & Decision Making (2008).
  63. Federated learning with non-iid data. In arXiv:1806.00582.
  64. Data-free knowledge distillation for heterogeneous federated learning. In ICLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Song Wang (313 papers)
  2. Xingbo Fu (15 papers)
  3. Kaize Ding (59 papers)
  4. Chen Chen (753 papers)
  5. Huiyuan Chen (43 papers)
  6. Jundong Li (126 papers)
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now