Papers
Topics
Authors
Recent
Search
2000 character limit reached

Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform

Published 11 Jul 2023 in cs.LG and cs.AI | (2307.05399v1)

Abstract: Production deployments in complex systems require ML architectures to be highly efficient and usable against multiple tasks. Particularly demanding are classification problems in which data arrives in a streaming fashion and each class is presented separately. Recent methods with stochastic gradient learning have been shown to struggle in such setups or have limitations like memory buffers, and being restricted to specific domains that disable its usage in real-world scenarios. For this reason, we present a fully differentiable architecture based on the Mixture of Experts model, that enables the training of high-performance classifiers when examples from each class are presented separately. We conducted exhaustive experiments that proved its applicability in various domains and ability to learn online in production environments. The proposed technique achieves SOTA results without a memory buffer and clearly outperforms the reference methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Development and deployment of a large-scale dialog-based intelligent tutoring system. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 114–121, Minneapolis, Minnesota. Association for Computational Linguistics.
  2. A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework. arXiv preprint arXiv:2204.03719.
  3. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.09823.
  4. pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain.
  5. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930.
  6. Ensemble deep learning in bioinformatics. Nature Machine Intelligence, 2(9):500–508.
  7. Efficient lifelong learning with a-gem. In ICLR.
  8. Continual learning with tiny episodic memories. arXiv preprint arXiv:1902.10486, 2019.
  9. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32.
  10. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  11. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  12. Efficient continual learning ensembles in neural network subspaces. arXiv preprint arXiv:2202.09826.
  13. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135.
  14. Parameter-efficient mixture-of-experts architecture for pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3263–3273, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  15. Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML’06), pages 377–384. ACM Press.
  16. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667–3684, Florence, Italy. Association for Computational Linguistics.
  17. The challenges of deploying artificial intelligence models in a rapidly evolving pandemic. Nature Machine Intelligence, 2(6):298–300.
  18. Adaptive mixtures of local experts. Neural computation, 3(1):79–87.
  19. Simultaneous speech translation for live subtitling: from delay to display. In Proceedings of the 1st Workshop on Automatic Spoken Language Translation in Real-World Settings (ASLTRW), pages 35–48, Virtual. Association for Machine Translation in the Americas.
  20. Achieving forgetting prevention and knowledge transfer in continual learning. Advances in Neural Information Processing Systems, 34:22443–22456.
  21. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796.
  22. Lang. 2008. 20 newsgroups dataset.
  23. A neural dirichlet process mixture model for task-free continual learning. CoRR, abs/2001.00689.
  24. A novel ensemble deep learning model for stock prediction based on stock prices and news. International Journal of Data Science and Analytics, 13(2):139–149.
  25. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947.
  26. Vincenzo Lomonaco and Davide Maltoni. 2017. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17–26. PMLR.
  27. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6470–6479, Red Hook, NY, USA. Curran Associates Inc.
  28. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28–51.
  29. Saeed Masoudnia and Reza Ebrahimpour. 2014. Mixture of experts: a literature survey. The Artificial Intelligence Review, 42(2):275.
  30. nEYron: Implementation and deployment of an MT system for a large audit & consulting corporation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 183–189, Ghent, Belgium. European Association for Machine Translation.
  31. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning, 11(5-6):355–607.
  32. SummaReranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4504–4524, Dublin, Ireland. Association for Computational Linguistics.
  33. Waseem Rawat and Zenghui Wang. 2017. Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 29(9):2352–2449.
  34. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. CoRR, abs/1806.01246.
  35. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  36. Iqbal H Sarker. 2022. Ai-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems. SN Computer Science, 3(2):158.
  37. Encoders and ensembles for task-free continual learning. CoRR, abs/2105.13327.
  38. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958.
  39. Gido M Van de Ven and Andreas S Tolias. 2019. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.
  40. Attention is all you need. Advances in neural information processing systems, 30.
  41. Pete Warden. 2018. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209.
  42. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715.
  43. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.05482.
  44. Differentiable top-k with optimal transport. In Advances in Neural Information Processing Systems, volume 33, pages 20520–20531. Curran Associates, Inc.
  45. A survey on ensemble learning under the era of deep learning. arXiv preprint arXiv:2101.08387.
  46. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995. PMLR.
  47. Cha Zhang and Yunqian Ma. 2012. Ensemble machine learning: methods and applications. Springer.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up