Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PIP: Prototypes-Injected Prompt for Federated Class Incremental Learning (2407.20705v1)

Published 30 Jul 2024 in cs.LG, cs.AI, and cs.CV

Abstract: Federated Class Incremental Learning (FCIL) is a new direction in continual learning (CL) for addressing catastrophic forgetting and non-IID data distribution simultaneously. Existing FCIL methods call for high communication costs and exemplars from previous classes. We propose a novel rehearsal-free method for FCIL named prototypes-injected prompt (PIP) that involves 3 main ideas: a) prototype injection on prompt learning, b) prototype augmentation, and c) weighted Gaussian aggregation on the server side. Our experiment result shows that the proposed method outperforms the current state of the arts (SOTAs) with a significant improvement (up to 33%) in CIFAR100, MiniImageNet and TinyImageNet datasets. Our extensive analysis demonstrates the robustness of PIP in different task sizes, and the advantage of requiring smaller participating local clients, and smaller global rounds. For further study, source codes of PIP, baseline, and experimental logs are shared publicly in https://github.com/anwarmaxsum/PIP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Ss-il: Separated softmax for incremental learning. In Proceedings of the IEEE/CVF International conference on computer vision. 844–853.
  2. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV). 139–154.
  3. Fed-cprompt: Contrastive prompt for rehearsal-free federated continual learning. arXiv preprint arXiv:2307.04869 (2023).
  4. Federated learning and blockchain-enabled fog-IoT platform for wearables in predictive healthcare. IEEE Transactions on Computational Social Systems (2023).
  5. Optimization methods for large-scale machine learning. SIAM review 60, 2 (2018), 223–311.
  6. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems 33 (2020), 15920–15930.
  7. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV). 233–248.
  8. No One Left Behind: Real-World Federated Class-Incremental Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023), 1–17. https://doi.org/10.1109/TPAMI.2023.3334213
  9. Federated class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10164–10173.
  10. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. Springer, 86–102.
  11. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9285–9295.
  12. Three-stage Stackelberg game enabled clustered federated learning in heterogeneous UAV swarms. IEEE Transactions on Vehicular Technology (2023).
  13. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 831–839.
  14. Distilling causal effect of data in class-incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 3957–3966.
  15. Towards the practical utility of federated learning in the medical domain. In Conference on Health, Inference, and Learning. PMLR, 163–181.
  16. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16071–16080.
  17. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning. PMLR, 5132–5143.
  18. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521–3526.
  19. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
  20. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 (2020), 429–450.
  21. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 (2019).
  22. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021).
  23. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40, 12 (2017), 2935–2947.
  24. Federated Class-Incremental Learning with Prompting. arXiv preprint arXiv:2310.08948 (2023).
  25. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems 34 (2021), 5972–5984.
  26. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282.
  27. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning. PMLR, 2642–2651.
  28. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 524–540.
  29. Better generative replay for continual federated learning. arXiv preprint arXiv:2302.13001 (2023).
  30. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001–2010.
  31. Continual learning with deep generative replay. Advances in neural information processing systems 30 (2017).
  32. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07796 (2019).
  33. On learning the geodesic path for incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 1591–1600.
  34. CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11909–11919.
  35. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8432–8440.
  36. A distillation-based approach integrating continual learning and federated learning for pervasive services. arXiv preprint arXiv:2109.04197 (2021).
  37. Attention is all you need. Advances in neural information processing systems 30 (2017).
  38. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision. Springer, 631–648.
  39. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 139–149.
  40. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 374–382.
  41. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning. PMLR, 12073–12086.
  42. Addressing Catastrophic Forgetting in Federated Class-Continual Learning. arXiv preprint arXiv:2303.06937 (2023).
  43. TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4782–4793.
  44. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1131–1140.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - anwarmaxsum/PIP (2 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets