Papers
Topics
Authors
Recent
Search
2000 character limit reached

Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing

Published 7 Mar 2024 in cs.LG and cs.NE | (2403.04759v1)

Abstract: On-device learning has emerged as a prevailing trend that avoids the slow response time and costly communication of cloud-based learning. The ability to learn continuously and indefinitely in a changing environment, and with resource constraints, is critical for real sensor deployments. However, existing designs are inadequate for practical scenarios with (i) streaming data input, (ii) lack of supervision and (iii) limited on-board resources. In this paper, we design and deploy the first on-device lifelong learning system called LifeHD for general IoT applications with limited supervision. LifeHD is designed based on a novel neurally-inspired and lightweight learning paradigm called Hyperdimensional Computing (HDC). We utilize a two-tier associative memory organization to intelligently store and manage high-dimensional, low-precision vectors, which represent the historical patterns as cluster centroids. We additionally propose two variants of LifeHD to cope with scarce labeled inputs and power constraints. We implement LifeHD on off-the-shelf edge platforms and perform extensive evaluations across three scenarios. Our measurements show that LifeHD improves the unsupervised clustering accuracy by up to 74.8% compared to the state-of-the-art NN-based unsupervised lifelong learning baselines with as much as 34.3x better energy efficiency. Our code is available at https://github.com/Orienfish/LifeHD.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. 2023. Jetson TX2 Module. https://developer.nvidia.com/embedded/jetson-tx2. [Online].
  2. 2023a. Raspberry Pi 4B. https://www.raspberrypi.com/products/raspberry-pi-4-model-b/. [Online].
  3. 2023b. Raspberry Pi Zero 2 W. https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/. [Online].
  4. Aurore Avarguès-Weber et al. 2012. Simultaneous mastering of two abstract concepts by the miniature brain of bees. Proceedings of the National Academy of Sciences 109, 19 (2012), 7481–7486.
  5. Alan Baddeley. 1992. Working memory. Science 255, 5044 (1992), 556–559.
  6. Garcia Rafael Banos, Oresti and Alejandro Saez. 2014. MHEALTH Dataset. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5TW22.
  7. Bernd Bischl et al. 2023. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 13, 2 (2023), e1484.
  8. Trenton Bricken et al. 2023. Sparse Distributed Memory is a Continual Learner. In International Conference on Learning Representations.
  9. Han Cai et al. 2020. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems 33 (2020), 11285–11297.
  10. Ning Chen et al. 2016. Smart urban surveillance using fog computing. In 2016 IEEE/ACM Symposium on Edge Computing (SEC). IEEE, 95–96.
  11. Arpan Dutta et al. 2022. Hdnn-pim: Efficient in memory design of hyperdimensional computing with feature extraction. In Proceedings of the Great Lakes Symposium on VLSI 2022. 281–286.
  12. Ehab Essa and Islam R Abdelmaksoud. 2023. Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors. Knowledge-Based Systems 278 (2023), 110867.
  13. Representational Continuity for Unsupervised Continual Learning. In International Conference on Learning Representations.
  14. Enrico Fini et al. 2022. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  15. In Gim and JeongGil Ko. 2022. Memory-efficient DNN training on mobile devices. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 464–476.
  16. Jean-Bastien Grill et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33 (2020), 21271–21284.
  17. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review 53, 2 (2011), 217–288.
  18. Michael Hersche et al. 2022. Constrained few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9057–9067.
  19. Hioki. 2023. Hioki3334 Powermeter. https://www.hioki.com/en/products/detail/?product_key=5812.
  20. Andrew Howard et al. 2019. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1314–1324.
  21. Mohsen Imani et al. 2019a. Hdcluster: An accurate clustering using brain-inspired high-dimensional computing. In Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 1591–1594.
  22. Mohsen Imani et al. 2019b. Semihd: Semi-supervised learning using hyperdimensional computing. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 1–8.
  23. Voicehd: Hyperdimensional computing for efficient speech recognition. In IEEE International Conference on Rebooting Computing (ICRC). IEEE, 1–8.
  24. Pentti Kanerva. 2009. Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors. Cognitive Computation 1 (2009), 139–159.
  25. Prive-hd: Privacy-preserved hyperdimensional computing. In ACM/IEEE Design Automation Conference (DAC). IEEE, 1–6.
  26. Efficient neural network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12569–12577.
  27. Efficient human activity recognition using hyperdimensional computing. In Proceedings of the 8th International Conference on the Internet of Things. 1–6.
  28. James Kirkpatrick et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences (2017).
  29. Learning multiple layers of features from tiny images. (2009).
  30. LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms. In Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems.
  31. Soochan Lee et al. 2020. A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning. In International Conference on Learning Representations.
  32. Ji Lin et al. 2020. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems 33 (2020), 11711–11722.
  33. Ji Lin et al. 2021. Memory-efficient patch-based inference for tiny deep learning. Advances in Neural Information Processing Systems 34 (2021), 2346–2358.
  34. Ji Lin et al. 2022. On-device training under 256kb memory. Advances in Neural Information Processing Systems 35 (2022), 22941–22954.
  35. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems 30 (2017).
  36. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Vol. 24. Elsevier, 109–165.
  37. Md Mohaimenuzzaman et al. 2023. Environmental Sound Classification on the Edge: A Pipeline for Deep Acoustic Networks on Extremely Resource-Constrained Devices. Pattern Recognition 133 (2023), 109025.
  38. Ali Moin et al. 2021. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nature Electronics 4, 1 (2021), 54–63.
  39. James O’ Neill. 2020. An overview of neural network compression. arXiv preprint arXiv:2006.03669 (2020).
  40. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems 14 (2001).
  41. Evgeny Osipov et al. 2022. Hyperseed: Unsupervised learning with vector symbolic architectures. IEEE Transactions on Neural Networks and Learning Systems (2022).
  42. Continual lifelong learning with neural networks: A review. Neural networks 113 (2019), 54–71.
  43. Adam Paszke et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  44. Karol J Piczak. 2015. ESC: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia. 1015–1018.
  45. MiniLearn: On-Device Learning for Low-Power IoT Devices. In International Conference on Embedded Wireless Systems and Networks.
  46. Continual unsupervised representation learning. Advances in neural information processing systems 32 (2019).
  47. Tinyol: Tinyml with online-learning on microcontrollers. In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.
  48. Olga Russakovsky et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115 (2015), 211–252.
  49. Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016).
  50. Swapnil Sayan Saha et al. 2023. TinyNS: Platform-Aware Neurosymbolic Auto Tiny Machine Learning. ACM Transactions on Embedded Computing Systems (2023).
  51. Mark Sandler et al. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510–4520.
  52. Algorithmic insights on continual learning from fruit flies. arXiv preprint arXiv:2107.07617 (2021).
  53. Shun Shunhou and Yang Peng. 2022. AIoT on Cloud. In Digital Transformation in Cloud Computing. CRC Press, 629–732.
  54. James Smith et al. 2021. Unsupervised Progressive Learning and the STAM Architecture. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 2979–2987.
  55. ” Alexa, stop spying on me!” speech privacy protection against voice assistants. In Proceedings of the 18th conference on Embedded Networked Sensor Systems. 298–311.
  56. A theoretical perspective on hyperdimensional computing. Journal of Artificial Intelligence Research 72 (2021), 215–249.
  57. Matteo Tiezzi et al. 2022. Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. 3480–3486.
  58. Rishabh Tiwari et al. 2022. Gcr: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 99–108.
  59. Ulrike Von Luxburg. 2007. A tutorial on spectral clustering. Statistics and computing 17 (2007), 395–416.
  60. Erwei Wang et al. 2019. Deep neural network approximation for custom hardware: Where we’ve been, where we’re going. ACM Computing Surveys (CSUR) 52, 2 (2019), 1–39.
  61. Qipeng Wang et al. 2022. Melon: Breaking the memory wall for resource-efficient on-device machine learning. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 450–463.
  62. Gary M Weiss et al. 2016. Smartwatch-based activity recognition: A machine learning approach. In 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 426–429.
  63. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning. PMLR, 478–487.
  64. Daliang Xu et al. 2022. Mandheling: Mixed-precision on-device dnn training with dsp offloading. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 214–227.
  65. FSL-HD: Accelerating Few-Shot Learning on ReRAM using Hyperdimensional Computing. In 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 1–6.
  66. Junting Zhang et al. 2020a. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1131–1140.
  67. MDLdroidLite: A release-and-inhibit control approach to resource-efficient deep neural networks on mobile devices. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 463–475.

Summary

  • The paper presents LifeHD as the first implementation of lifelong learning on edge devices using hyperdimensional computing, effectively addressing streaming data and resource constraints.
  • It introduces distinct LifeHD variants that optimize learning with minimal supervision and conserve power through dynamic model pruning.
  • Evaluations demonstrate up to 74.8% improvement in unsupervised clustering accuracy and 34.3x energy efficiency over conventional neural network-based methods.

Lifelong Intelligence Beyond the Edge with Hyperdimensional Computing

Introduction

The proliferation of IoT devices across various sectors necessitates on-device learning capabilities that can adapt to changing environments without the latency and privacy concerns associated with cloud computing. This paper introduces LifeHD, a novel system for on-device lifelong learning tailored for IoT applications that face dynamic data streams, minimal supervision, and stringent resource constraints.

LifeHD at a Glance

LifeHD leverages Hyperdimensional Computing (HDC) to achieve efficient and continuous learning directly on edge devices. Unlike traditional neural network approaches that struggle with the constraints of edge computing, LifeHD's HDC-based framework allows it to efficiently process and learn from streaming data in an unsupervised manner. Key to LifeHD's design is its two-tier memory system, which simulates short-term and long-term memories, enabling the system to adaptively retain and forget information over time.

Technical Contributions

The primary contributions of this work include:

  1. Design and Implementation: LifeHD is the first implementation of a lifelong learning system on edge devices using HDC. It addresses critical challenges such as streaming data, minimal supervision, and resource limitations through its innovative design incorporating a novel two-tier memory system.
  2. LifeHD Variants: The introduction of semi and a variants of LifeHD caters to scenarios with limited labeled inputs and power constraints, respectively. semi leverages any available labels to enhance learning efficiency, while a dynamically prunes HDC models to conserve energy.
  3. Evaluation: Extensive evaluations showcase LifeHD's superior performance across several IoT scenarios, including activity recognition, sound classification, and object recognition. LifeHD demonstrates up to 74.8\% improvement in unsupervised clustering accuracy and 34.3x better energy efficiency compared to state-of-the-art neural network-based baselines.

Theoretical and Practical Implications

From a theoretical perspective, LifeHD's success underscores the potential of HDC in addressing the challenges of lifelong learning on resource-constrained devices. Practically, LifeHD's efficiency and adaptability make it an attractive solution for a wide range of IoT applications requiring on-device intelligence.

Future Directions

While LifeHD marks a significant advancement in lifelong learning for edge computing, several avenues for future research are identified. These include scaling LifeHD for more complex applications, exploring the integration of LifeHD with federated learning frameworks, and extending its capabilities to handle a wider variety of tasks beyond classification.

Conclusion

LifeHD represents a pivotal step toward realizing the full potential of edge computing in IoT systems. By leveraging the lightweight and neurally-inspired paradigm of HDC, LifeHD offers a scalable and efficient solution for on-device lifelong learning, paving the way for smarter and more autonomous IoT devices.

File Document Download Save Streamline Icon: https://streamlinehq.com Markdown

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free