Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning (2403.04207v2)

Published 7 Mar 2024 in cs.LG and cs.DC

Abstract: Federated Learning (FL) is a practical approach to train deep learning models collaboratively across user-end devices, protecting user privacy by retaining raw data on-device. In FL, participating user-end devices are highly fragmented in terms of hardware and software configurations. Such fragmentation introduces a new type of data heterogeneity in FL, namely \textit{system-induced data heterogeneity}, as each device generates distinct data depending on its hardware and software configurations. In this paper, we first characterize the impact of system-induced data heterogeneity on FL model performance. We collect a dataset using heterogeneous devices with variations across vendors and performance tiers. By using this dataset, we demonstrate that \textit{system-induced data heterogeneity} negatively impacts accuracy, and deteriorates fairness and domain generalization problems in FL. To address these challenges, we propose HeteroSwitch, which adaptively adopts generalization techniques (i.e., ISP transformation and SWAD) depending on the level of bias caused by varying HW and SW configurations. In our evaluation with a realistic FL dataset (FLAIR), HeteroSwitch reduces the variance of averaged precision by 6.3\% across device types.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Arena Com, Ltd. Gsmarena. https://www.gsmarena.com/, 2022. Accessed: 2023-07-11.
  2. Federated learning of predictive models from federated electronic health records. International journal of medical informatics, 112:59–67, 2018.
  3. Reconfiguring the imaging pipeline for computer vision. In Proceedings of the IEEE International Conference on Computer Vision, pp.  975–984, 2017.
  4. Cannistra, S. Pixel binning. http://www.starrywonders.com/binning.html. Accessed: 2023-08-06.
  5. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  6. Adaptive bayesian wavelet shrinkage. Journal of the American Statistical Association, 92(440):1413–1421, 1997.
  7. Characterizing and taming model instability across edge devices. Proceedings of Machine Learning and Systems, 3:624–636, 2021.
  8. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pp.  2921–2926. IEEE, 2017.
  9. Environment inference for invariant learning. In International Conference on Machine Learning, pp.  2189–2200. PMLR, 2021.
  10. A Modern Introduction to Probability and Statistics: Understanding why and how, volume 488. Springer, 2005.
  11. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  12. Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (QoMEX), pp.  1–6. IEEE, 2016.
  13. Domain generalization via model-agnostic learning of semantic features. Advances in neural information processing systems, 32, 2019.
  14. Ebner, M. Color constancy, volume 7. John Wiley & Sons, 2007.
  15. Gozdz, J. Fbdd denoising. https://valelab4.ucsf.edu/svn/"#"micromanager2/trunk/DeviceAdapters/"#"TetheredCam/LibRaw/internal/dcb_demosaicing.c, 2010. Accessed: 2023-08-06.
  16. Training speech recognition models with federated learning: A quality/cost framework. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  3080–3084. IEEE, 2021.
  17. Isp4ml: The role of image signal processing in efficient deep learning vision systems. In 2020 25th International Conference on Pattern Recognition (ICPR), pp.  2438–2445. IEEE, 2021.
  18. Fel: High capacity learning for recommendation and ranking via federated ensemble learning. arXiv preprint arXiv:2206.03852, 2022.
  19. Adaptive homogeneity-directed demosaicing algorithm. Ieee transactions on image processing, 14(3):360–369, 2005.
  20. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  1314–1324, 2019.
  21. Papaya: Practical, private, and scalable federated learning. Proceedings of Machine Learning and Systems, 4:814–832, 2022.
  22. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
  23. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
  24. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  25. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.
  26. Autoscale: Energy efficiency optimization for stochastic edge inference using reinforcement learning. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp.  1082–1096. IEEE, 2020.
  27. Autofl: Enabling heterogeneity-aware energy efficient federated learning. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pp.  183–198, 2021.
  28. Learning multiple layers of features from tiny images. 2009.
  29. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp.  5815–5826. PMLR, 2021.
  30. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  31. LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
  32. Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497, 2019.
  33. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  34. Integrated cnn and federated learning for covid-19 detection on chest x-ray images. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2022.
  35. Lin, C.-k. Pixel grouping for color filter array demosaicing, 2003.
  36. Neural network generalization: The impact of camera parameters. IEEE Access, 8:10443–10454, 2020.
  37. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp.  116–131, 2018.
  38. Towards fair federated recommendation learning: Characterizing the inter-dependence of system and data heterogeneity. In Proceedings of the 16th ACM Conference on Recommender Systems, pp.  156–167, 2022.
  39. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
  40. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  41. Agnostic federated learning. In International Conference on Machine Learning, pp.  4615–4625. PMLR, 2019.
  42. Morovič, J. Color gamut mapping. John Wiley & Sons, 2008.
  43. A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3):1–44, 2022.
  44. Mobilenetv3 for image classification. In 2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), pp.  490–497, 2021. doi: 10.1109/ICBAIE52039.2021.9389905.
  45. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019.
  46. Do datasets have politics? disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–37, 2021.
  47. Batch normalization embeddings for deep domain generalization. Pattern Recognition, 135:109115, 2023.
  48. Towards fairness-aware federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  49. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1–48, 2019.
  50. Flair: Federated learning annotated image repository. Advances in Neural Information Processing Systems, 35:37792–37805, 2022.
  51. StatCounter. Mobile vendor market share. https://gs.statcounter.com/vendor-market-share/mobile/united-states-of-america/2021, 2022. Accessed: 2023-07-11.
  52. Stokes, M. A standard default color space for the internet-srgb. http://www. w3. org/Graphics/Color/sRGB. html, 1996.
  53. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
  54. Machine learning at facebook: Understanding inference at the edge. In 2019 IEEE international symposium on high performance computer architecture (HPCA), pp.  331–344. IEEE, 2019.
  55. Node selection toward faster convergence for federated learning on non-iid data. IEEE Transactions on Network Science and Engineering, 9(5):3099–3111, 2022.
  56. Color science: concepts and methods, quantitative data and formulae, volume 40. John wiley & sons, 2000.
  57. Image prediction for limited-angle tomography via deep learning with convolutional neural network. arXiv preprint arXiv:1607.08707, 2016.
  58. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Gyudong Kim (1 paper)
  2. Mehdi Ghasemi (20 papers)
  3. Soroush Heidari (1 paper)
  4. Seungryong Kim (103 papers)
  5. Young Geun Kim (7 papers)
  6. Sarma Vrudhula (11 papers)
  7. Carole-Jean Wu (62 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets