Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Text-Enhanced Data-free Approach for Federated Class-Incremental Learning (2403.14101v1)

Published 21 Mar 2024 in cs.CV, cs.LG, and cs.CL

Abstract: Federated Class-Incremental Learning (FCIL) is an underexplored yet pivotal issue, involving the dynamic addition of new classes in the context of federated learning. In this field, Data-Free Knowledge Transfer (DFKT) plays a crucial role in addressing catastrophic forgetting and data privacy problems. However, prior approaches lack the crucial synergy between DFKT and the model training phases, causing DFKT to encounter difficulties in generating high-quality data from a non-anchored latent space of the old task model. In this paper, we introduce LANDER (Label Text Centered Data-Free Knowledge Transfer) to address this issue by utilizing label text embeddings (LTE) produced by pretrained LLMs. Specifically, during the model training phase, our approach treats LTE as anchor points and constrains the feature embeddings of corresponding training samples around them, enriching the surrounding area with more meaningful information. In the DFKT phase, by using these LTE anchors, LANDER can synthesize more meaningful samples, thereby effectively addressing the forgetting problem. Additionally, instead of tightly constraining embeddings toward the anchor, the Bounding Loss is introduced to encourage sample embeddings to remain flexible within a defined radius. This approach preserves the natural differences in sample embeddings and mitigates the embedding overlap caused by heterogeneous federated settings. Extensive experiments conducted on CIFAR100, Tiny-ImageNet, and ImageNet demonstrate that LANDER significantly outperforms previous methods and achieves state-of-the-art performance in FCIL. The code is available at https://github.com/tmtuan1307/lander.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Uncertainty-based continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
  2. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018.
  3. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254–11263, 2019.
  4. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
  5. A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  6. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8218–8227, 2021.
  7. Il2m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision, pages 583–592, 2019.
  8. Practical secure aggregation for federated learning on user-held data. arxiv 2016. arXiv preprint arXiv:1611.04482.
  9. A unified wasserstein distributional robustness framework for adversarial training. arXiv preprint arXiv:2202.13437, 2022.
  10. Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv preprint arXiv:1811.11206, 2018.
  11. Data-free learning of student networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3514–3522, 2019.
  12. Class-prototype conditional diffusion model for continual learning with generative replay. arXiv preprint arXiv:2312.06710, 2023.
  13. Federated class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10164–10173, 2022.
  14. Adversarial continual learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 386–402. Springer, 2020.
  15. Federated learning in vehicular networks. In 2022 IEEE International Mediterranean Conference on Communications and Networking (MeditCom), pages 72–77. IEEE, 2022.
  16. Up to 100x faster data-free knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6597–6604, 2022.
  17. Contrastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.08584, 2021.
  18. Local model poisoning attacks to {{\{{Byzantine-Robust}}\}} federated learning. In 29th USENIX security symposium (USENIX Security 20), pages 1605–1622, 2020.
  19. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.
  20. Vulexplainer: A transformer-based hierarchical distillation for explaining vulnerability types. IEEE Transactions on Software Engineering, 2023.
  21. R-dfcil: Relation-guided representation learning for data-free class incremental learning. In European Conference on Computer Vision, pages 423–439. Springer, 2022.
  22. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  23. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  24. Fedspeech: Federated text-to-speech with continual learning. arXiv preprint arXiv:2110.07216, 2021.
  25. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  26. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  27. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  28. Learning multiple layers of features from tiny images. 2009.
  29. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  30. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188–1196. PMLR, 2014.
  31. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  32. Generative models from the perspective of continual learning. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019.
  33. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020.
  34. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  35. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017.
  36. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  37. DFRD: Data-free robustness distillation for heterogeneous federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  38. Threats to federated learning: A survey. arXiv preprint arXiv:2003.02133, 2020.
  39. Continual federated learning based on knowledge distillation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, volume 3, 2022.
  40. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  41. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  42. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  43. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems, 32, 2019.
  44. Inceptionism: Going deeper into neural networks. 2015.
  45. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials, 23(3):1622–1658, 2021.
  46. Cycle class consistency with distributional optimal transport and knowledge distillation for unsupervised domain adaptation. In Uncertainty in Artificial Intelligence, pages 1519–1529. PMLR, 2022.
  47. Adversarial local distribution regularization for knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4681–4690, 2023.
  48. Continual deep learning by functional regularisation of memorable past. Advances in Neural Information Processing Systems, 33:4453–4464, 2020.
  49. Frequency attention for knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2277–2286, 2024.
  50. Continual distributed learning for crisis management. arXiv preprint arXiv:2104.12876, 2021.
  51. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  52. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  53. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  54. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9630–9638, 2021.
  55. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  56. Nayer: Noisy layer data generation for efficient and effective data-free knowledge distillation. arXiv preprint arXiv:2310.00258, 2023.
  57. Koppa: Improving prompt-based continual learning with key-query orthogonal projection and prototype-based one-versus-all. arXiv preprint arXiv:2311.15414, 2023.
  58. A distillation-based approach integrating continual learning and federated learning for pervasive services. arXiv preprint arXiv:2109.04197, 2021.
  59. Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
  60. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454–3469, 2020.
  61. Incremental classifier learning with generative adversarial networks. arXiv preprint arXiv:1802.00853, 2018.
  62. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5:1–19, 2021.
  63. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8724, 2020.
  64. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073–12086. PMLR, 2021.
  65. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017.
  66. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022.
  67. Target: Federated class-continual learning via exemplar-free distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4782–4793, 2023.
  68. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13208–13217, 2020.
  69. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
  70. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In The Eleventh International Conference on Learning Representations, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube