Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Relaxed Contrastive Learning for Federated Learning (2401.04928v2)

Published 10 Jan 2024 in cs.LG

Abstract: We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a na\"ive adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Federated learning based on dynamic regularization. In ICLR, 2021.
  2. Federated learning via posterior averaging: A new perspective and practical algorithms. In ICLR, 2021.
  3. A simple framework for contrastive learning of visual representations. In ICML, 2020.
  4. Discriminability-transferability trade-off: an information-theoretic perspective. In ECCV, 2022.
  5. Fedbr: Improving federated learning on heterogeneous data via local learning bias reduction. In ICML, 2023.
  6. Federated learning with compression: Unified analysis and sharp guarantees. In AISTATS, 2021.
  7. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  8. The non-iid data quagmire of decentralized machine learning. In ICML, 2020.
  9. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  10. Federated visual classification with real-world data distribution. In ECCV, 2020.
  11. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
  12. Scaffold: Stochastic controlled averaging for on-device federated learning. In ICML, 2020.
  13. Supervised contrastive learning. NeurIPS, 2020.
  14. Communication-efficient federated learning with acceleration of global momentum. arXiv preprint arXiv:2201.03172, 2022a.
  15. Multi-level branched regularization for federated learning. In ICML, 2022b.
  16. Learning multiple layers of features from tiny images. 2009.
  17. Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  18. Preservation of the global knowledge by not-true distillation in federated learning. In NeurIPS, 2022.
  19. Model-contrastive federated learning. In CVPR, 2021.
  20. Feddane: A federated newton-type method. In ACSCC, 2019.
  21. Federated optimization in heterogeneous networks. In MLSys, 2020.
  22. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.
  23. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, 143:93–104, 2023.
  24. Fedpara: Low-rank hadamard product for communication-efficient federated learning. In ICLR, 2022.
  25. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  26. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  27. Adaptive federated optimization. In ICLR, 2021.
  28. FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization. In AISTATS, 2020.
  29. Fitnets: Hints for thin deep nets. In ICLR, 2015.
  30. The effective rank: A measure of effective dimensionality. In EUSIPCO, 2007.
  31. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
  32. Improving the generalization of supervised models. arXiv preprint arXiv:2206.15369, 2022.
  33. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. In ICLR, 2023.
  34. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.
  35. Communication-efficient adaptive federated learning. In ICML, 2022.
  36. Group normalization. In ECCV, 2018.
  37. Acceleration of federated learning with alleviated forgetting in local training. In ICLR, 2022.
  38. Quantifying the variability collapse of neural networks. In ICML, 2023.
  39. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874, 2021.
  40. Local-global knowledge distillation in heterogeneous federated learning with non-iid data. arXiv preprint arXiv:2107.00051, 2021.
  41. Fedmix: Approximation of mixup under mean augmented federated learning. In ICLR, 2021.
  42. Federated learning with label distribution skew via logits calibration. In ICML, 2022.
  43. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
  44. Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data. In arXiv preprint arXiv:2005.11418, 2020.
  45. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
  46. Divergence-aware federated self-supervised learning. In ICLR, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now