Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 134 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 23 tok/s Pro
GPT-5 High 30 tok/s Pro
GPT-4o 70 tok/s Pro
Kimi K2 200 tok/s Pro
GPT OSS 120B 441 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning (2401.03230v1)

Published 6 Jan 2024 in cs.LG, cs.CR, and cs.DC

Abstract: Recently, Heterogeneous Federated Learning (HtFL) has attracted attention due to its ability to support heterogeneous models and data. To reduce the high communication cost of transmitting model parameters, a major challenge in HtFL, prototype-based HtFL methods are proposed to solely share class representatives, a.k.a, prototypes, among heterogeneous clients while maintaining the privacy of clients' models. However, these prototypes are naively aggregated into global prototypes on the server using weighted averaging, resulting in suboptimal global knowledge which negatively impacts the performance of clients. To overcome this challenge, we introduce a novel HtFL approach called FedTGP, which leverages our Adaptive-margin-enhanced Contrastive Learning (ACL) to learn Trainable Global Prototypes (TGP) on the server. By incorporating ACL, our approach enhances prototype separability while preserving semantic meaning. Extensive experiments with twelve heterogeneous models demonstrate that our FedTGP surpasses state-of-the-art methods by up to 9.08% in accuracy while maintaining the communication and privacy advantages of prototype-based HtFL. Our code is available at https://github.com/TsingZ0/FedTGP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Single-layer vision transformers for more accurate early exits with less overhead. Neural Networks, 153: 461–473.
  2. Iterative clustering pruning for convolutional neural networks. Knowledge-Based Systems, 265: 110386.
  3. On Bridging Generic and Personalized Federated Learning for Image Classification. In ICLR.
  4. MobileTL: On-Device Transfer Learning with Inverted Residual Blocks. In Proceedings of the AAAI Conference on Artificial Intelligence.
  5. AMC-loss: Angular margin contrastive loss for improved explainability in image classification. In CVPR Workshop.
  6. A Downsampled Variant of Imagenet as an Alternative to the Cifar Datasets. arXiv preprint arXiv:1707.08819.
  7. Exploiting Shared Representations for Personalized Federated Learning. In ICML.
  8. Arcface: Additive angular margin loss for deep face recognition. In CVPR.
  9. HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. In ICLR.
  10. Gaussian affinity for max-margin class imbalanced learning. In ICCV.
  11. Deep Residual Learning for Image Recognition. In CVPR.
  12. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  13. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. NeurIPS.
  14. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479.
  15. Regularized margin-based conditional log-likelihood loss for prototype learning. Pattern Recognition, 43(7): 2428–2438.
  16. Advances and Open Problems in Federated Learning. arXiv preprint arXiv:1912.04977.
  17. Attract, perturb, and explore: Learning a feature alignment network for semi-supervised domain adaptation. In ECCV.
  18. Learning Multiple Layers of Features From Tiny Images. Technical Report.
  19. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems.
  20. Iamnn: Iterative and adaptive mobile neural network for efficient image classification. arXiv preprint arXiv:1804.10123.
  21. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581.
  22. Optimizing the deep neural networks by layer-wise refined pruning and the acceleration on FPGA. Computational Intelligence and Neuroscience, 2022.
  23. Federated Learning on Non-IID Data Silos: An Experimental Study. In ICDE.
  24. Model-Contrastive Federated Learning. In CVPR.
  25. A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection. IEEE Transactions on Knowledge and Data Engineering.
  26. Ditto: Fair and Robust Federated Learning Through Personalization. In ICML.
  27. Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine, 37(3): 50–60.
  28. No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier. arXiv preprint arXiv:2303.10058.
  29. SMKD: Selective Mutual Knowledge Distillation. In IJCNN.
  30. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523.
  31. DraftFed: A Draft-Based Personalized Federated Learning Approach for Heterogeneous Convolutional Neural Networks. IEEE Transactions on Mobile Computing.
  32. A closer look at branch classifiers of multi-exit architectures. arXiv preprint arXiv:2204.13347.
  33. Ensemble distillation for robust model fusion in federated learning. NeurIPS.
  34. No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID data. In NeurIPS.
  35. Layer-wised model aggregation for personalized federated learning. In CVPR.
  36. Communication-Efficient Learning of Deep Networks from Decentralized Data. In AISTATS.
  37. Stable and efficient spectral divide and conquer algorithms for the symmetric eigenvalue decomposition and the SVD. SIAM Journal on Scientific Computing, 35(3): A1325–A1349.
  38. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, 722–729. IEEE.
  39. Pinheiro, P. O. 2018. Unsupervised domain adaptation with similarity learning. In CVPR.
  40. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR.
  41. Facenet: A unified embedding for face recognition and clustering. In CVPR.
  42. Personalized federated learning using hypernetworks. In ICML.
  43. Federated mutual learning. arXiv preprint arXiv:2006.16765.
  44. Scaling law for recommendation models: Towards general-purpose user representations. In AAAI.
  45. Going deeper with convolutions. In CVPR.
  46. Personalized Federated Learning with Moreau Envelopes. In NeurIPS.
  47. Towards Personalized Federated Learning. IEEE Transactions on Neural Networks and Learning Systems. Early Access.
  48. Fedproto: Federated Prototype Learning across Heterogeneous Clients. In AAAI.
  49. Federated Learning from Pre-Trained Models: A Contrastive Learning Approach. arXiv preprint arXiv:2209.10083.
  50. A prototype-oriented framework for unsupervised domain adaptation. NeurIPS.
  51. Visualizing Data Using T-SNE. Journal of Machine Learning Research, 9(11).
  52. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440.
  53. Model Barrier: A Compact Un-Transferable Isolation Domain for Model Intellectual Property Protection. In CVPR.
  54. Federated dropout—A simple approach for enabling federated learning on resource constrained devices. IEEE wireless communications letters, 11(5): 923–927.
  55. Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1): 2032.
  56. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747.
  57. Attribute prototype network for zero-shot learning. NeurIPS.
  58. Seizing critical learning periods in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
  59. Robust classification with convolutional prototype learning. In CVPR.
  60. Dynamic Personalized Federated Learning with Adaptive Differential Privacy. In NeurIPS.
  61. FedGH: Heterogeneous Federated Learning with Generalized Global Header. arXiv preprint arXiv:2303.13137.
  62. Multimodal Federated Learning via Contrastive Representation Ensemble. In ICLR.
  63. Protecting intellectual property of deep neural networks with watermarking. In ASIA-CCS.
  64. Towards Data-Independent Knowledge Transfer in Model-Heterogeneous Federated Learning. IEEE Transactions on Computers.
  65. Parameterized Knowledge Transfer for Personalized Federated Learning. In NeurIPS.
  66. Eliminating Domain Bias for Federated Learning in Representation Space. In NeurIPS.
  67. GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning. In ICCV.
  68. FedALA: Adaptive Local Aggregation for Personalized Federated Learning. In AAAI.
  69. FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy. In KDD.
  70. Semantic Image Segmentation by Dynamic Discriminative Prototypes. IEEE Transactions on Multimedia.
  71. Fine-Tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning. In CVPR.
  72. Deep mutual learning. In CVPR.
  73. A new lightweight network based on MobileNetV3. KSII Transactions on Internet & Information Systems, 16(1).
  74. A lightweight deep neural network with higher accuracy. Plos one, 17(8): e0271225.
  75. Deep residual networks for hyperspectral image classification. In IEEE international geoscience and remote sensing symposium (IGARSS).
  76. Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In ICML.
  77. When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions. arXiv preprint arXiv:2306.15546.
Citations (14)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up