Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Machine Learning-oriented Survey on Tiny Machine Learning (2309.11932v2)

Published 21 Sep 2023 in cs.LG

Abstract: The emergence of Tiny Machine Learning (TinyML) has positively revolutionized the field of Artificial Intelligence by promoting the joint design of resource-constrained IoT hardware devices and their learning-based software architectures. TinyML carries an essential role within the fourth and fifth industrial revolutions in helping societies, economies, and individuals employ effective AI-infused computing technologies (e.g., smart cities, automotive, and medical robotics). Given its multidisciplinary nature, the field of TinyML has been approached from many different angles: this comprehensive survey wishes to provide an up-to-date overview focused on all the learning algorithms within TinyML-based solutions. The survey is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodological flow, allowing for a systematic and complete literature survey. In particular, firstly we will examine the three different workflows for implementing a TinyML-based system, i.e., ML-oriented, HW-oriented, and co-design. Secondly, we propose a taxonomy that covers the learning panorama under the TinyML lens, examining in detail the different families of model optimization and design, as well as the state-of-the-art learning techniques. Thirdly, this survey will present the distinct features of hardware devices and software tools that represent the current state-of-the-art for TinyML intelligent edge applications. Finally, we discuss the challenges and future directions.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (171)
  1. L. Dutta and S. Bharali, “Tinyml meets iot: A comprehensive survey,” Internet of Things, vol. 16, p. 100461, 2021.
  2. M. Capra, R. Peloso, G. Masera, M. Ruo Roch, and M. Martina, “Edge computing: A survey on the hardware requirements in the internet of things world,” Future Internet, vol. 11, no. 4, p. 100, 2019.
  3. S. Madakam, V. Lake, V. Lake, V. Lake et al., “Internet of things (iot): A literature review,” Journal of Computer and Communications, vol. 3, no. 05, p. 164, 2015.
  4. A. Daissaoui, A. Boulmakoul, L. Karim, and A. Lbath, “Iot and big data analytics for smart buildings: A survey,” Procedia computer science, vol. 170, pp. 161–168, 2020.
  5. H. Arasteh, V. Hosseinnezhad, V. Loia, A. Tommasetti, O. Troisi, M. Shafie-khah, and P. Siano, “Iot-based smart cities: A survey,” in 2016 IEEE 16th international conference on environment and electrical engineering (EEEIC).   IEEE, 2016, pp. 1–6.
  6. Y. Kabalci, “A survey on smart metering and smart grid communication,” Renewable and Sustainable Energy Reviews, vol. 57, pp. 302–318, 2016.
  7. B. B. Sinha and R. Dhanalakshmi, “Recent advancements and challenges of internet of things in smart agriculture: A survey,” Future Generation Computer Systems, vol. 126, pp. 169–184, 2022.
  8. N. Gondchawar, R. Kawitkar et al., “Iot based smart agriculture,” International Journal of advanced research in Computer and Communication Engineering, vol. 5, no. 6, pp. 838–842, 2016.
  9. F. Alshehri and G. Muhammad, “A comprehensive survey of the internet of things (iot) and ai-based smart healthcare,” IEEE Access, vol. 9, pp. 3660–3678, 2020.
  10. Y. Song, F. R. Yu, L. Zhou, X. Yang, and Z. He, “Applications of the internet of things (iot) in smart logistics: A comprehensive survey,” IEEE Internet of Things Journal, vol. 8, no. 6, pp. 4250–4274, 2020.
  11. A. Jayaram, “Smart retail 4.0 iot consumer retailer model for retail intelligence and strategic marketing of in-store products,” Proceedings of the 17th international business horizon-INBUSH ERA-2017, Noida, India, vol. 9, 2017.
  12. Y. Liao, E. d. F. R. Loures, and F. Deschamps, “Industrial internet of things: A systematic literature review and insights,” IEEE Internet of Things Journal, vol. 5, no. 6, pp. 4515–4525, 2018.
  13. E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58 443–58 469, 2020.
  14. E. Hozdić, “Smart factory for industry 4.0: A review,” International Journal of Modern Manufacturing Technologies, vol. 7, no. 1, pp. 28–35, 2015.
  15. V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM computing surveys (CSUR), vol. 41, no. 3, pp. 1–58, 2009.
  16. W. Zhang, D. Yang, and H. Wang, “Data-driven methods for predictive maintenance of industrial equipment: A survey,” IEEE systems journal, vol. 13, no. 3, pp. 2213–2227, 2019.
  17. S. Branco, A. G. Ferreira, and J. Cabral, “Machine learning in resource-scarce embedded systems, fpgas, and end-devices: A survey,” Electronics, vol. 8, no. 11, p. 1289, 2019.
  18. J. Lin, W.-M. Chen, Y. Lin, C. Gan, S. Han et al., “Mcunet: Tiny deep learning on iot devices,” Advances in Neural Information Processing Systems, vol. 33, pp. 11 711–11 722, 2020.
  19. A. Pramod, H. S. Naicker, and A. K. Tyagi, “Machine learning and deep learning: Open issues and future research directions for the next 10 years,” Computational analysis and deep learning for medical care: Principles, methods, and applications, pp. 463–490, 2021.
  20. N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, “The computational limits of deep learning,” arXiv preprint arXiv:2007.05558, 2020.
  21. V. Rajapakse, I. Karunanayake, and N. Ahmed, “Intelligence at the extreme edge: A survey on reformable tinyml,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–30, 2023.
  22. O. Bringmann, W. Ecker, I. Feldner, A. Frischknecht, C. Gerum, T. Hämäläinen, M. A. Hanif, M. J. Klaiber, D. Mueller-Gritschneder, P. P. Bernardo et al., “Automated hw/sw co-design for edge ai: state, challenges and steps ahead,” in Proceedings of the 2021 International Conference on Hardware/Software Codesign and System Synthesis, 2021, pp. 11–20.
  23. T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 9, pp. 5149–5169, 2021.
  24. M. Trimmel, M. Zanfir, R. Hartley, and C. Sminchisescu, “Era: Enhanced rational activations,” in European Conference on Computer Vision.   Springer, 2022, pp. 722–738.
  25. L. Metz, J. Harrison, C. D. Freeman, A. Merchant, L. Beyer, J. Bradbury, N. Agrawal, B. Poole, I. Mordatch, A. Roberts et al., “Velo: Training versatile learned optimizers by scaling up,” arXiv preprint arXiv:2211.09760, 2022.
  26. C. R. Banbury, V. J. Reddi, M. Lam, W. Fu, A. Fazel, J. Holleman, X. Huang, R. Hurtado, D. Kanter, A. Lokhmotov et al., “Benchmarking tinyml systems: Challenges and direction,” arXiv preprint arXiv:2003.04821, 2020.
  27. R. Sanchez-Iborra and A. F. Skarmeta, “Tinyml-enabled frugal smart objects: Challenges and opportunities,” IEEE Circuits and Systems Magazine, vol. 20, no. 3, pp. 4–18, 2020.
  28. Y. Y. Siang, M. R. Ahamd, and M. S. Z. Abidin, “Anomaly detection based on tiny machine learning: A review,” Open International Journal of Informatics, vol. 9, no. Special Issue 2, pp. 67–78, 2021.
  29. V. Tsoukas, E. Boumpa, G. Giannakas, and A. Kakarountas, “A review of machine learning and tinyml in healthcare,” in Proceedings of the 25th Pan-Hellenic Conference on Informatics, 2021, pp. 69–73.
  30. P. P. Ray, “A review on tinyml: State-of-the-art and prospects,” Journal of King Saud University-Computer and Information Sciences, vol. 34, no. 4, pp. 1595–1623, 2022.
  31. E. Njor, J. Madsen, and X. Fafoutis, “A primer for tinyml predictive maintenance: Input and model optimisation,” in IFIP International Conference on Artificial Intelligence Applications and Innovations.   Springer, 2022, pp. 67–78.
  32. M. Giordano, L. Piccinelli, and M. Magno, “Survey and comparison of milliwatts micro controllers for tiny machine learning at the edge,” in 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS).   IEEE, 2022, pp. 94–97.
  33. R. Immonen, T. Hämäläinen et al., “Tiny machine learning for resource-constrained microcontrollers,” Journal of Sensors, vol. 2022, 2022.
  34. N. Schizas, A. Karras, C. Karras, and S. Sioutas, “Tinyml for ultra-low power ai and large scale iot deployments: A systematic review,” Future Internet, vol. 14, no. 12, p. 363, 2022.
  35. S. B. Lakshman and N. U. Eisty, “Software engineering approaches for tinyml based iot embedded vision: a systematic literature review,” in Proceedings of the 4th International Workshop on Software Engineering Research and Practice for the IoT, 2022, pp. 33–40.
  36. N. N. Alajlan and D. M. Ibrahim, “Tinyml: Enabling of inference deep learning models on ultra-low-power iot edge devices for ai applications,” Micromachines, vol. 13, no. 6, p. 851, 2022.
  37. H. Bamoumen, A. Temouden, N. Benamar, and Y. Chtouki, “How tinyml can be leveraged to solve environmental problems: A survey,” in 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT).   IEEE, 2022, pp. 338–343.
  38. W. Su, L. Li, F. Liu, M. He, and X. Liang, “Ai on the edge: a comprehensive review,” Artificial Intelligence Review, pp. 1–59, 2022.
  39. S. S. Saha, S. S. Sandha, and M. Srivastava, “Machine learning for microcontroller-class hardware-a review,” IEEE Sensors Journal, 2022.
  40. S. Prakash, T. Callahan, J. Bushagour, C. Banbury, A. V. Green, P. Warden, T. Ansell, and V. J. Reddi, “Cfu playground: Full-stack open-source framework for tiny machine learning (tinyml) acceleration on fpgas,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).   IEEE, 2023, pp. 157–167.
  41. S. Kalapothas, M. Galetakis, G. Flamis, F. Plessas, and P. Kitsos, “A survey on risc-v-based machine learning ecosystem,” Information, vol. 14, no. 2, p. 64, 2023.
  42. Y. Abadade, A. Temouden, H. Bamoumen, N. Benamar, Y. Chtouki, and A. S. Hafid, “A comprehensive survey on tinyml,” IEEE Access, 2023.
  43. J. Yepes-Nuñez, G. Urrutia, M. Romero-Garcia, and S. Alonso-Fernandez, “The prisma 2020 statement: an updated guideline for reporting systematic reviews.” Revista Espanola de Cardiologia (English ed.), vol. 74, no. 9, pp. 790–799, 2021.
  44. A. I. C. on Computer-Aided Design (ICCAD), “Tinyml design contest,” https://tinymlcontest.github.io/TinyML-Design-Contest/, 2023.
  45. tinyML Foundation, “tinyml challenge 2022: Smart weather station,” https://www.tinyml.org/event/tinyml-challenge-2022-smart-weather-station-2/, 2023.
  46. H. Ren, D. Anicic, and T. A. Runkler, “Tinyreptile: Tinyml with federated meta-learning,” arXiv preprint arXiv:2304.05201, 2023.
  47. L. Heim, A. Biri, Z. Qu, and L. Thiele, “Measuring what really matters: Optimizing neural networks for tinyml,” arXiv preprint arXiv:2104.10645, 2021.
  48. J. Chang, Y. Choi, T. Lee, and J. Cho, “Reducing mac operation in convolutional neural network with sign prediction,” in 2018 International Conference on Information and Communication Technology Convergence (ICTC).   IEEE, 2018, pp. 177–182.
  49. M. Olyaiy, C. Ng, and M. Lis, “Accelerating dnns inference with predictive layer fusion,” in Proceedings of the ACM International Conference on Supercomputing, 2021, pp. 291–303.
  50. W.-C. Lin, Y.-C. Chang, and J.-D. Huang, “An efficient and low-power mlp accelerator architecture supporting structured pruning, sparse activations and asymmetric quantization for edge computing,” in 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS).   IEEE, 2021, pp. 1–5.
  51. Y.-C. Zhou, M. Lei, Y.-L. Zhang, Q. Zhang, and J. Han, “An enhanced data cache with in-cache processing units for convolutional neural network accelerators,” in 2020 IEEE 15th International Conference on Solid-State & Integrated Circuit Technology (ICSICT).   IEEE, 2020, pp. 1–3.
  52. F. Conti, D. Rossi, A. Pullini, I. Loi, and L. Benini, “Energy-efficient vision on the pulp platform for ultra-low power parallel computing,” in 2014 IEEE Workshop on Signal Processing Systems (SiPS).   IEEE, 2014, pp. 1–6.
  53. A. Garofalo, M. Rusci, F. Conti, D. Rossi, and L. Benini, “Pulp-nn: accelerating quantized neural networks on parallel ultra-low-power risc-v processors,” Philosophical Transactions of the Royal Society A, vol. 378, no. 2164, p. 20190155, 2020.
  54. ——, “Pulp-nn: A computing library for quantized neural network inference at the edge on risc-v based parallel ultra low power clusters,” in 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS).   IEEE, 2019, pp. 33–36.
  55. C. Zhou, F. G. Redondo, J. Büchel, I. Boybat, X. T. Comas, S. Nandakumar, S. Das, A. Sebastian, M. Le Gallo, and P. N. Whatmough, “Ml-hw co-design of noise-robust tinyml models and always-on analog compute-in-memory edge accelerator,” IEEE Micro, vol. 42, no. 6, pp. 76–87, 2022.
  56. H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang, “Network trimming: A data-driven neuron pruning approach towards efficient deep architectures,” arXiv preprint arXiv:1607.03250, 2016.
  57. S. Anwar, K. Hwang, and W. Sung, “Structured pruning of deep convolutional neural networks,” ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 13, no. 3, pp. 1–18, 2017.
  58. M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” arXiv preprint arXiv:1710.01878, 2017.
  59. J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke, “Scalpel: Customizing dnn pruning to the underlying hardware parallelism,” ACM SIGARCH Computer Architecture News, vol. 45, no. 2, pp. 548–560, 2017.
  60. Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” arXiv preprint arXiv:1810.05270, 2018.
  61. D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag, “What is the state of neural network pruning?” Proceedings of machine learning and systems, vol. 2, pp. 129–146, 2020.
  62. J. D. De Leon and R. Atienza, “Depth pruning with auxiliary networks for tinyml,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2022, pp. 3963–3967.
  63. S. Vadera and S. Ameen, “Methods for pruning deep neural networks,” IEEE Access, vol. 10, pp. 63 280–63 300, 2022.
  64. B. Sun, S. Bayes, A. M. Abotaleb, and M. Hassan, “The case for tinyml in healthcare: Cnns for real-time on-edge blood pressure estimation,” in Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, 2023, pp. 629–638.
  65. M. Hashir, N. Khalid, N. Mahmood, M. A. Rehman, M. Asad, M. Q. Mehmood, M. Zubair, and Y. Massoud, “A tinyml based portable, low-cost microwave head imaging system for brain stroke detection,” in 2023 IEEE International Symposium on Circuits and Systems (ISCAS).   IEEE, 2023, pp. 1–4.
  66. Z. Cai, X. He, J. Sun, and N. Vasconcelos, “Deep learning with low precision by half-wave gaussian quantization,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5918–5926.
  67. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2704–2713.
  68. R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” arXiv preprint arXiv:1806.08342, 2018.
  69. Y. Mishchenko, Y. Goren, M. Sun, C. Beauchene, S. Matsoukas, O. Rybakov, and S. N. P. Vitaladevuni, “Low-bit quantization and quantization-aware training for small-footprint keyword spotting,” in 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA).   IEEE, 2019, pp. 706–711.
  70. M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, “Up or down? adaptive rounding for post-training quantization,” in International Conference on Machine Learning.   PMLR, 2020, pp. 7197–7206.
  71. P. Wang, Q. Chen, X. He, and J. Cheng, “Towards accurate post-training network quantization via bit-split and stitching,” in International Conference on Machine Learning.   PMLR, 2020, pp. 9847–9856.
  72. A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” in Low-Power Computer Vision.   Chapman and Hall/CRC, 2022, pp. 291–326.
  73. M. Nagel, M. Fournarakis, Y. Bondarenko, and T. Blankevoort, “Overcoming oscillations in quantization-aware training,” arXiv preprint arXiv:2203.11086, 2022.
  74. S. Zhuo, H. Chen, R. K. Ramakrishnan, T. Chen, C. Feng, Y. Lin, P. Zhang, and L. Shen, “An empirical study of low precision quantization for tinyml,” arXiv preprint arXiv:2203.05492, 2022.
  75. J. Moosmann, M. Giordano, C. Vogt, and M. Magno, “Tinyissimoyolo: A quantized, low-memory footprint, tinyml object detection network for low power microcontrollers,” arXiv preprint arXiv:2306.00001, 2023.
  76. Q. Lu and B. Murmann, “Enhancing the energy efficiency and robustness of tinyml computer vision using coarsely-quantized log-gradient input images,” ACM Transactions on Embedded Computing Systems, 2023.
  77. N. N. Alajlan and D. M. Ibrahim, “Ddd tinyml: A tinyml-based driver drowsiness detection model using deep learning,” Sensors, vol. 23, no. 12, p. 5696, 2023.
  78. S. Yun, J. Park, K. Lee, and J. Shin, “Regularizing class-wise predictions via self-knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 876–13 885.
  79. H. Zhao, X. Sun, J. Dong, C. Chen, and Z. Dong, “Highlight every step: Knowledge distillation via collaborative teaching,” IEEE Transactions on Cybernetics, 2020.
  80. L. Zhang and K. Ma, “Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors,” in International Conference on Learning Representations, 2020.
  81. N. Körber, A. Siebert, S. Hauke, and D. Mueller-Gritschneder, “Tiny generative image compression for bandwidth-constrained sensor applications,” in 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA).   IEEE, 2021, pp. 564–569.
  82. A. Ukil, I. Sahu, A. Majumdar, S. C. Racha, G. Kulkarni, A. D. Choudhury, S. Khandelwal, A. Ghose, and A. Pal, “Resource constrained cvd classification using single lead ecg on wearable and implantable devices,” in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).   IEEE, 2021, pp. 886–889.
  83. J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021.
  84. H. Zhang, Z. Hu, W. Qin, M. Xu, and M. Wang, “Adversarial co-distillation learning for image recognition,” Pattern Recognition, vol. 111, p. 107659, 2021.
  85. H. Cheng, L. Yang, and Z. Liu, “Relation-based knowledge distillation for anomaly detection,” in Chinese Conference on Pattern Recognition and Computer Vision (PRCV).   Springer, 2021, pp. 105–116.
  86. X. Dai, Z. Jiang, Z. Wu, Y. Bao, Z. Wang, S. Liu, and E. Zhou, “General instance distillation for object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7842–7851.
  87. H. Al Zein, M. Aoude, and Y. Harkous, “Implementation and optimization of neural networks for tiny hardware devices,” in 2022 International Conference on Smart Systems and Power Management (IC2SPM).   IEEE, 2022, pp. 191–196.
  88. A. Brutti, F. Paissan, A. Ancilotto, and E. Farella, “Optimizing phinet architectures for the detection of urban sounds on low-end devices,” in 2022 30th European Signal Processing Conference (EUSIPCO).   IEEE, 2022, pp. 1121–1125.
  89. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” Advances in neural information processing systems, vol. 24, 2011.
  90. J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research, vol. 13, no. 2, 2012.
  91. J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, “Hyperparameter optimization for machine learning models based on bayesian optimization,” Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, 2019.
  92. S. S. Sandha, M. Aggarwal, S. S. Saha, and M. Srivastava, “Enabling hyperparameter tuning of machine learning classifiers in production,” in 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI).   IEEE, 2021, pp. 262–271.
  93. C. Tang, K. Ouyang, Z. Wang, Y. Zhu, W. Ji, Y. Wang, and W. Zhu, “Mixed-precision neural network quantization via learned layer-wise importance,” in European Conference on Computer Vision.   Springer, 2022, pp. 259–275.
  94. J. Youn, J. Song, H.-S. Kim, and S. Bahk, “Bitwidth-adaptive quantization-aware neural network training: A meta-learning approach,” in European Conference on Computer Vision.   Springer, 2022, pp. 208–224.
  95. H. R. Mendis, C.-K. Kang, and P.-c. Hsiu, “Intermittent-aware neural architecture search,” ACM Transactions on Embedded Computing Systems (TECS), vol. 20, no. 5s, pp. 1–27, 2021.
  96. C. Banbury, C. Zhou, I. Fedorov, R. Matas, U. Thakker, D. Gope, V. Janapa Reddi, M. Mattina, and P. Whatmough, “Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers,” Proceedings of Machine Learning and Systems, vol. 3, pp. 517–532, 2021.
  97. E. Liberis, Ł. Dudziak, and N. D. Lane, “μ𝜇\muitalic_μnas: Constrained neural architecture search for microcontrollers,” in Proceedings of the 1st Workshop on Machine Learning and Systems, 2021, pp. 70–79.
  98. P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, X. Chen, and X. Wang, “A comprehensive survey of neural architecture search: Challenges and solutions,” ACM Computing Surveys (CSUR), vol. 54, no. 4, pp. 1–34, 2021.
  99. D. Baymurzina, E. Golikov, and M. Burtsev, “A review of neural architecture search,” Neurocomputing, 2021.
  100. Y. Liu, Y. Sun, B. Xue, M. Zhang, G. G. Yen, and K. C. Tan, “A survey on evolutionary neural architecture search,” IEEE transactions on neural networks and learning systems, 2021.
  101. E. Njor, J. Madsen, and X. Fafoutis, “Data aware neural architecture search,” arXiv preprint arXiv:2304.01821, 2023.
  102. D. P. Pau, P. K. Ambrose, and F. M. Aymone, “A quantitative review of automated neural search and on-device learning for tiny devices,” Chips, vol. 2, no. 2, pp. 130–141, 2023.
  103. A. M. Garavagno, E. Ragusa, A. Frisoli, and P. Gastaldo, “A hardware-aware neural architecture search algorithm targeting low-end microcontrollers,” in 2023 18th Conference on Ph. D Research in Microelectronics and Electronics (PRIME).   IEEE, 2023, pp. 281–284.
  104. A. Molina, P. Schramowski, and K. Kersting, “Pad\\\backslash\’e activation units: End-to-end learning of flexible activation functions in deep networks,” arXiv preprint arXiv:1907.06732, 2019.
  105. A. Apicella, F. Donnarumma, F. Isgrò, and R. Prevete, “A survey on modern trainable activation functions,” Neural Networks, vol. 138, pp. 14–32, 2021.
  106. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  107. F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
  108. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520.
  109. J. Luo, J. Wang, N. Cheng, E. Xiao, X. Zhang, and J. Xiao, “Tiny-sepformer: A tiny time-domain transformer network for speech separation,” arXiv preprint arXiv:2206.13689, 2022.
  110. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” Advances in neural information processing systems, vol. 28, 2015.
  111. L. Huang, W. Wang, J. Chen, and X.-Y. Wei, “Attention on attention for image captioning,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4634–4643.
  112. A. Galassi, M. Lippi, and P. Torroni, “Attention in natural language processing,” IEEE transactions on neural networks and learning systems, vol. 32, no. 10, pp. 4291–4308, 2020.
  113. A. Wong, M. Famouri, and M. J. Shafiee, “Attendnets: tiny deep image recognition neural networks for the edge via visual attention condensers,” arXiv preprint arXiv:2009.14385, 2020.
  114. G. Brauwers and F. Frasincar, “A general survey on attention mechanisms in deep learning,” IEEE Transactions on Knowledge and Data Engineering, 2021.
  115. A. Burrello, M. Scherer, M. Zanghieri, F. Conti, and L. Benini, “A microcontroller is all you need: Enabling transformer execution on low-power iot endnodes,” in 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS).   IEEE, 2021, pp. 1–6.
  116. S. Mehta and M. Rastegari, “Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,” arXiv preprint arXiv:2110.02178, 2021.
  117. S. Abbasi Koohpayegani, A. Tejankar, and H. Pirsiavash, “Compress: Self-supervised learning by compressing representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 12 980–12 992, 2020.
  118. M. de Prado, M. Rusci, A. Capotondi, R. Donze, L. Benini, and N. Pazos, “Robustifying the deployment of tinyml models for autonomous mini-vehicles,” Sensors, vol. 21, no. 4, p. 1339, 2021.
  119. D. Hussein and G. Bhat, “Sensorgan: A novel data recovery approach for wearable human activity recognition,” ACM Transactions on Embedded Computing Systems, 2023.
  120. N. Yamin and G. Bhat, “Uncertainty-aware energy harvest prediction and management for iot devices,” ACM Transactions on Design Automation of Electronic Systems, 2023.
  121. S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain, “Drocc: Deep robust one-class classification,” in International conference on machine learning.   PMLR, 2020, pp. 3711–3721.
  122. S. Abbasi, M. Famouri, M. J. Shafiee, and A. Wong, “Outliernets: highly compact deep autoencoder network architectures for on-device acoustic anomaly detection,” Sensors, vol. 21, no. 14, p. 4805, 2021.
  123. H. Kayan, Y. Majib, W. Alsafery, M. Barhamgi, and C. Perera, “Anoml-iot: An end to end re-configurable multi-protocol anomaly detection pipeline for internet of things,” Internet of Things, vol. 16, p. 100437, 2021.
  124. M. Lord and A. Kaplan, “Mechanical anomaly detection on an embedded microcontroller,” in 2021 International Conference on Computational Science and Computational Intelligence (CSCI).   IEEE, 2021, pp. 562–568.
  125. P. Andrade, I. Silva, G. Signoretti, M. Silva, J. Dias, L. Marques, and D. G. Costa, “An unsupervised tinyml approach applied for pavement anomalies detection under the internet of intelligent vehicles,” in 2021 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4. 0&IoT).   IEEE, 2021, pp. 642–647.
  126. P. Andrade, I. Silva, M. Silva, T. Flores, J. Cassiano, and D. G. Costa, “A tinyml soft-sensor approach for low-cost detection and monitoring of vehicular emissions,” Sensors, vol. 22, no. 10, p. 3838, 2022.
  127. X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” IEEE transactions on knowledge and data engineering, vol. 35, no. 1, pp. 857–876, 2021.
  128. M. Abououf, R. Mizouni, S. Singh, H. Otrok, and E. Damiani, “Self-supervised online and lightweight anomaly and event detection for iot devices,” IEEE Internet of Things Journal, vol. 9, no. 24, pp. 25 285–25 299, 2022.
  129. V. Rani, S. T. Nabi, M. Kumar, A. Mittal, and K. Kumar, “Self-supervised learning: A succinct review,” Archives of Computational Methods in Engineering, vol. 30, no. 4, pp. 2761–2775, 2023.
  130. L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of artificial intelligence research, vol. 4, pp. 237–285, 1996.
  131. F. Svoboda, D. Nunes, M. Alizadeh, R. Daries, R. Luo, A. Mathur, S. Bhattacharya, J. S. Silva, and N. D. Lane, “Resource efficient deep reinforcement learning for acutely constrained tinyml devices,” in Research Symposium on Tiny Machine Learning, 2020.
  132. T. Szydlo, P. P. Jayaraman, Y. Li, G. Morgan, and R. Ranjan, “Tinyrl: Towards reinforcement learning on tiny embedded devices,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 4985–4988.
  133. X. Kong, G. Duan, M. Hou, G. Shen, H. Wang, X. Yan, and M. Collotta, “Deep reinforcement learning-based energy-efficient edge computing for internet of vehicles,” IEEE Transactions on Industrial Informatics, vol. 18, no. 9, pp. 6308–6316, 2022.
  134. D. Pau, S. Colella, and C. Marchisio, “End to end optimized tiny learning for repositionable walls in maze topologies,” in 2023 IEEE International Conference on Consumer Electronics (ICCE).   IEEE, 2023, pp. 1–7.
  135. T. Barbariol and G. A. Susto, “Tiws-iforest: Isolation forest in weakly supervised and tiny ml scenarios,” Information Sciences, vol. 610, pp. 126–143, 2022.
  136. M. Antonini, M. Pincheira, M. Vecchio, and F. Antonelli, “An adaptable and unsupervised tinyml anomaly detection system for extreme industrial environments,” Sensors, vol. 23, no. 4, p. 2344, 2023.
  137. C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning.   PMLR, 2017, pp. 1126–1135.
  138. S. Belkhale, R. Li, G. Kahn, R. McAllister, R. Calandra, and S. Levine, “Model-based meta-reinforcement learning for flight with suspended payloads,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1471–1478, 2021.
  139. H. Cho, Y. Cho, J. Yu, and J. Kim, “Camera distortion-aware 3d human pose estimation in video with optimization-based meta-learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 11 169–11 178.
  140. D. Gao, X. He, Z. Zhou, Y. Tong, and L. Thiele, “Pruning meta-trained networks for on-device adaptation,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021, pp. 514–523.
  141. D. Gao, Y. Xie, Z. Zhou, Z. Wang, Y. Li, and B. Ding, “Finding meta winning ticket to train your maml,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 411–420.
  142. Y. Liu, S. Duan, X. Xu, and S. Ren, “Metaldc: Meta learning of low-dimensional computing classifiers for fast on-device adaption,” arXiv preprint arXiv:2302.12347, 2023.
  143. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
  144. H. Cai, C. Gan, L. Zhu, and S. Han, “Tinytl: Reduce memory, not parameters for efficient on-device learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 11 285–11 297, 2020.
  145. H. Ren, D. Anicic, and T. A. Runkler, “Tinyol: Tinyml with online-learning on microcontrollers,” in 2021 international joint conference on neural networks (IJCNN).   IEEE, 2021, pp. 1–8.
  146. L. Ravaglia, M. Rusci, D. Nadalini, A. Capotondi, F. Conti, and L. Benini, “A tinyml platform for on-device continual learning with quantized latent replays,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 11, no. 4, pp. 789–802, 2021.
  147. B. Sudharsan, P. Yadav, J. G. Breslin, and M. I. Ali, “Train++: An incremental ml model training algorithm to create self-learning iot devices,” in 2021 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/IOP/SCI).   IEEE, 2021, pp. 97–106.
  148. A. Avi, A. Albanese, and D. Brunelli, “Incremental online learning algorithms comparison for gesture and visual smart sensors,” in 2022 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2022, pp. 1–8.
  149. M. Pavan, E. Ostrovan, A. Caltabiano, and M. Roveri, “Tybox: an automatic design and code-generation toolbox for tinyml incremental on-device learning,” ACM Transactions on Embedded Computing Systems, 2023.
  150. K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26–38, 2017.
  151. N. Suda and D. Loh, “Machine learning on arm cortex-m microcontrollers,” Arm Ltd.: Cambridge, UK, 2019.
  152. L. Lai, N. Suda, and V. Chandra, “Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus,” arXiv preprint arXiv:1801.06601, 2018.
  153. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
  154. S. Markidis, S. W. Der Chien, E. Laure, I. B. Peng, and J. S. Vetter, “Nvidia tensor core programmability, performance & precision,” in 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW).   IEEE, 2018, pp. 522–531.
  155. Y. E. Wang, G.-Y. Wei, and D. Brooks, “Benchmarking tpu, gpu, and cpu platforms for deep learning,” arXiv preprint arXiv:1907.10701, 2019.
  156. A. Shawahna, S. M. Sait, and A. El-Maleh, “Fpga-based accelerators of deep learning networks for learning and classification: A review,” ieee Access, vol. 7, pp. 7823–7859, 2018.
  157. Z. A. O. Nasri Sulaiman, M. Marhaban, and M. Hamidon, “Design and implementation of fpga-based systems-a review,” Australian Journal of Basic and Applied Sciences, vol. 3, no. 4, pp. 3575–3596, 2009.
  158. T. Wang, C. Wang, X. Zhou, and H. Chen, “A survey of fpga based deep learning accelerators: Challenges and opportunities,” arXiv preprint arXiv:1901.04988, 2018.
  159. N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers et al., “In-datacenter performance analysis of a tensor processing unit,” in Proceedings of the 44th annual international symposium on computer architecture, 2017, pp. 1–12.
  160. K. Seshadri, B. Akin, J. Laudon, R. Narayanaswami, and A. Yazdanbakhsh, “An evaluation of edge tpu accelerators for convolutional neural networks,” in 2022 IEEE International Symposium on Workload Characterization (IISWC).   IEEE, 2022, pp. 79–91.
  161. R. David, J. Duke, A. Jain, V. Janapa Reddi, N. Jeffries, J. Li, N. Kreeger, I. Nappier, M. Natraj, T. Wang et al., “Tensorflow lite micro: Embedded machine learning for tinyml systems,” Proceedings of Machine Learning and Systems, vol. 3, pp. 800–811, 2021.
  162. uTensor, “utensor: Tinyml ai inference library,” https://utensor.github.io/website/, 2023.
  163. V. Janapa Reddi, A. Elium, S. Hymel, D. Tischler, D. Situnayake, C. Ward, L. Moreau, J. Plunkett, M. Kelcey, M. Baaijens et al., “Edge impulse: An mlops platform for tiny machine learning,” Proceedings of Machine Learning and Systems, vol. 5, 2023.
  164. STMicroelectronics, “X-cube-ai: Ai expansion pack for stm32cubemx,” https://www.st.com/en/embedded-software/x-cube-ai.html, 2023.
  165. Microsoft, “Embedded learning library: An open source library for embedded ai and machine learning from microsoft,” https://microsoft.github.io/ELL/, 2023.
  166. T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze et al., “{{\{{TVM}}\}}: An automated {{\{{End-to-End}}\}} optimizing compiler for deep learning,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. 578–594.
  167. S. Jaiswal, R. K. K. Goli, A. Kumar, V. Seshadri, and R. Sharma, “Minun: Accurate ml inference on microcontrollers,” in Proceedings of the 24th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems, 2023, pp. 26–39.
  168. P. Mattson, C. Cheng, G. Diamos, C. Coleman, P. Micikevicius, D. Patterson, H. Tang, G.-Y. Wei, P. Bailis, V. Bittorf et al., “Mlperf training benchmark,” Proceedings of Machine Learning and Systems, vol. 2, pp. 336–349, 2020.
  169. V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou et al., “Mlperf inference benchmark,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA).   IEEE, 2020, pp. 446–459.
  170. C. Banbury, V. J. Reddi, P. Torelli, J. Holleman, N. Jeffries, C. Kiraly, P. Montino, D. Kanter, S. Ahmed, D. Pau et al., “Mlperf tiny benchmark,” arXiv preprint arXiv:2106.07597, 2021.
  171. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Luigi Capogrosso (16 papers)
  2. Federico Cunico (13 papers)
  3. Dong Seon Cheng (1 paper)
  4. Franco Fummi (18 papers)
  5. Marco Cristani (64 papers)
Citations (22)
View on Semantic Scholar