Papers
Topics
Authors
Recent
Search
2000 character limit reached

Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models

Published 26 Feb 2024 in cs.LG and cs.AI | (2402.16255v1)

Abstract: Federated learning encounters substantial challenges with heterogeneous data, leading to performance degradation and convergence issues. While considerable progress has been achieved in mitigating such an impact, the reliability aspect of federated models has been largely disregarded. In this study, we conduct extensive experiments to investigate the reliability of both generic and personalized federated models. Our exploration uncovers a significant finding: \textbf{federated models exhibit unreliability when faced with heterogeneous data}, demonstrating poor calibration on in-distribution test data and low uncertainty levels on out-of-distribution data. This unreliability is primarily attributed to the presence of biased projection heads, which introduce miscalibration into the federated models. Inspired by this observation, we propose the "Assembled Projection Heads" (APH) method for enhancing the reliability of federated models. By treating the existing projection head parameters as priors, APH randomly samples multiple initialized parameters of projection heads from the prior and further performs targeted fine-tuning on locally available data under varying learning rates. Such a head ensemble introduces parameter diversity into the deterministic model, eliminating the bias and producing reliable predictions via head averaging. We evaluate the effectiveness of the proposed APH method across three prominent federated benchmarks. Experimental results validate the efficacy of APH in model calibration and uncertainty estimation. Notably, APH can be seamlessly integrated into various federated approaches but only requires less than 30\% additional computation cost for 100$\times$ inferences within large models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263.
  2. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818.
  3. On Bridging Generic and Personalized Federated Learning for Image Classification. In International Conference on Learning Representations.
  4. Exploiting shared representations for personalized federated learning. In International conference on machine learning, 2089–2099. PMLR.
  5. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33: 3557–3568.
  6. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, 1050–1059. PMLR.
  7. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10112–10121.
  8. Graves, A. 2011. Practical variational inference for neural networks. Advances in neural information processing systems, 24.
  9. On calibration of modern neural networks. In International conference on machine learning, 1321–1330. PMLR.
  10. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.
  11. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15262–15271.
  12. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.
  13. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 7865–7873.
  14. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.
  15. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2): 1–210.
  16. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, 5132–5143. PMLR.
  17. Learning multiple layers of features from tiny images.
  18. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
  19. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3.
  20. Preservation of the global knowledge by not-true distillation in federated learning. Advances in Neural Information Processing Systems, 35: 38461–38474.
  21. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV), 163–168. IEEE.
  22. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), 965–978. IEEE.
  23. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10713–10722.
  24. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, 6357–6368. PMLR.
  25. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3): 50–60.
  26. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2: 429–450.
  27. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 429–450.
  28. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2682–2690.
  29. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351–2363.
  30. Structured and efficient variational deep learning with matrix gaussian posteriors. In International conference on machine learning, 1708–1716. PMLR.
  31. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34: 5972–5984.
  32. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32.
  33. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273–1282. PMLR.
  34. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific reports, 6(1): 1–10.
  35. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.
  36. Reading digits in natural images with unsupervised feature learning.
  37. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32.
  38. A Consistent and Differentiable Lpsubscript𝐿𝑝L_{p}italic_L start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT Canonical Calibration Error Estimator. In Advances in Neural Information Processing Systems.
  39. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127.
  40. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31.
  41. Federated multi-task learning. Advances in neural information processing systems, 30.
  42. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33: 21394–21405.
  43. Single model uncertainty estimation via stochastic data centering. Advances in Neural Information Processing Systems, 35: 8662–8674.
  44. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, 9690–9700. PMLR.
  45. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15: 3454–3469.
  46. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), 681–688.
  47. Personalized Federated Learning with Feature Alignment and Classifier Collaboration. In The Eleventh International Conference on Learning Representations.
  48. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2): 1–19.
  49. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 11237–11244.
  50. Personalized Federated Learning with First Order Model Optimization. In International Conference on Learning Representations.
  51. Federated learning on non-IID data: A survey. Neurocomputing, 465: 371–390.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up