Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 73 tok/s
Gemini 2.5 Pro 51 tok/s Pro
GPT-5 Medium 31 tok/s Pro
GPT-5 High 32 tok/s Pro
GPT-4o 103 tok/s Pro
Kimi K2 218 tok/s Pro
GPT OSS 120B 460 tok/s Pro
Claude Sonnet 4.5 35 tok/s Pro
2000 character limit reached

Hardware-Aware Static Optimization of Hyperdimensional Computations (2304.03335v3)

Published 6 Apr 2023 in cs.PL, cs.IT, and math.IT

Abstract: Binary spatter code (BSC)-based hyperdimensional computing (HDC) is a highly error-resilient approximate computational paradigm suited for error-prone, emerging hardware platforms. In BSC HDC, the basic datatype is a hypervector, a typically large binary vector, where the size of the hypervector has a significant impact on the fidelity and resource usage of the computation. Typically, the hypervector size is dynamically tuned to deliver the desired accuracy; this process is time-consuming and often produces hypervector sizes that lack accuracy guarantees and produce poor results when reused for very similar workloads. We present Heim, a hardware-aware static analysis and optimization framework for BSC HD computations. Heim analytically derives the minimum hypervector size that minimizes resource usage and meets the target accuracy requirement. Heim guarantees the optimized computation converges to the user-provided accuracy target on expectation, even in the presence of hardware error. Heim deploys a novel static analysis procedure that unifies theoretical results from the neuroscience community to systematically optimize HD computations. We evaluate Heim against dynamic tuning-based optimization on 25 benchmark data structures. Given a 99% accuracy requirement, Heim-optimized computations achieve a 99.2%-100.0% median accuracy, up to 49.5% higher than dynamic tuning-based optimization, while achieving 1.15x-7.14x reductions in hypervector size compared to HD computations that achieve comparable query accuracy and finding parametrizations 30.0x-100167.4x faster than dynamic tuning-based approaches. We also use Heim to systematically evaluate the performance benefits of using analog CAMs and multiple-bit-per-cell ReRAM over conventional hardware, while maintaining iso-accuracy -- for both emerging technologies, we find usages where the emerging hardware imparts significant benefits.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Sara Achour and Martin C Rinard. 2015. Approximate computation with outlier detection in topaz. Acm Sigplan Notices 50, 10 (2015), 711–730. https://doi.org/10.1145/2858965.2814314
  2. Hypervector design for efficient hyperdimensional computing on edge devices. arXiv preprint arXiv:2103.06709 (2021). https://doi.org/10.48550/arXiv.2103.06709
  3. The gem5 simulator. ACM SIGARCH computer architecture news 39, 2 (2011), 1–7. https://doi.org/10.1145/2024716.2024718
  4. Capacity Analysis of Vector Symbolic Architectures. arXiv preprint arXiv:2301.10352 (2023). https://doi.org/10.48550/arXiv.2301.10352
  5. A 5 μ𝜇\muitalic_μw standard cell memory-based configurable hyperdimensional computing accelerator for always-on smart sensing. IEEE Transactions on Circuits and Systems I: Regular Papers 68, 10 (2021), 4116–4128. https://doi.org/10.1109/TCSI.2021.3100266
  6. A theory of sequence indexing and working memory in recurrent neural networks. Neural Computation 30, 6 (2018), 1449–1513. https://doi.org/10.1162/neco_a_01084
  7. Stephen I Gallant and T Wendy Okaywe. 2013. Representing objects, relations, and sequences. Neural computation 25, 8 (2013), 2038–2078. https://doi.org/10.1162/NECO_a_00467
  8. Ross W Gayler and Simon D Levy. 2009. A distributed basis for analogical mapping. In New Frontiers in Analogy Research; Proc. of 2nd Intern. Analogy Conf, Vol. 9.
  9. Resistive RAM endurance: Array-level characterization and correction techniques targeting deep learning applications. IEEE Transactions on Electron Devices 66, 3 (2019), 1281–1288. https://doi.org/10.1109/TED.2019.2894387
  10. Fused RRAM-based shift-add architecture for efficient hyperdimensional computing paradigm. In 2021 IEEE International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE, 179–182. https://doi.org/10.1109/MWSCAS47672.2021.9531748
  11. Hyperdimensional hashing: a robust and efficient dynamic hash table. In Proceedings of the 59th ACM/IEEE Design Automation Conference. 907–912. https://doi.org/10.1145/3489517.3530553
  12. High-density multiple bits-per-cell 1T4R RRAM array with gradual SET/RESET and its effectiveness for deep learning. In 2019 IEEE International Electron Devices Meeting (IEDM). IEEE, 35–6. https://doi.org/10.1109/IEDM19573.2019.8993514
  13. Quanthd: A quantization framework for hyperdimensional computing. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 39, 10 (2019), 2268–2278. https://doi.org/10.1109/TCAD.2019.2954472
  14. Hierarchical hyperdimensional computing for energy efficient classification. In Proceedings of the 55th Annual Design Automation Conference. 1–6. https://doi.org/10.1145/3195970.3196060
  15. Voicehd: Hyperdimensional computing for efficient speech recognition. In 2017 IEEE international conference on rebooting computing (ICRC). IEEE, 1–8. https://doi.org/10.1109/ICRC.2017.8123650
  16. Exploring hyperdimensional associative memory. In 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 445–456. https://doi.org/10.1109/HPCA.2017.28
  17. Fach: Fpga-based acceleration of hyperdimensional computing by reducing computational complexity. In Proceedings of the 24th Asia and South Pacific Design Automation Conference. 493–498. https://doi.org/10.1145/3287624.3287667
  18. Sparsehd: Algorithm-hardware co-optimization for efficient high-dimensional computing. In 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE, 190–198. https://doi.org/10.1109/FCCM.2019.00034
  19. Michael N Jones and Douglas JK Mewhort. 2007. Representing word meaning and order information in a composite holographic lexicon. Psychological review 114, 1 (2007), 1. https://doi.org/10.1037/0033-295X.114.1.1
  20. Pentti Kanerva. 2009. Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors. Cognitive computation 1, 2 (2009), 139–159.
  21. Pentti Kanerva. 2010. What we mean when we say” What’s the dollar of Mexico?”: Prototypes and mapping in concept space. In 2010 AAAI fall symposium series.
  22. Pentti Kanerva. 2014. Computing with 10,000-bit words. In 2014 52nd annual Allerton conference on communication, control, and computing (Allerton). IEEE, 304–310. https://doi.org/10.1109/ALLERTON.2014.7028470
  23. Pentti Kanerva. 2018. Computing with high-dimensional vectors. IEEE Design & Test 36, 3 (2018), 7–14. https://doi.org/10.1109/MDAT.2018.2890221
  24. Pentti Kanerva et al. 1997. Fully distributed representation. PAT 1, 5 (1997), 10000.
  25. In-memory hyperdimensional computing. Nature Electronics 3, 6 (2020), 327–337. https://doi.org/10.1038/s41565-023-01357-8
  26. Geniehd: Efficient dna pattern matching accelerator using hyperdimensional computing. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 115–120. https://doi.org/10.23919/DATE48585.2020.9116397
  27. Efficient decoding of compositional structure in holistic representations. Neural Computation 35, 7 (2023), 1159–1186. https://doi.org/10.1162/neco_a_01590
  28. Vector Symbolic Architectures as a Computing Framework for Emerging Hardware. Proc. IEEE 110, 10 (2022), 1538–1571. https://doi.org/10.1109/JPROC.2022.3209104
  29. Holographic graph neuron: A bioinspired architecture for pattern processing. IEEE transactions on neural networks and learning systems 28, 6 (2016), 1250–1262. https://doi.org/10.1109/TNNLS.2016.2535338
  30. A survey on hyperdimensional computing aka vector symbolic architectures, part ii: Applications, cognitive models, and challenges. Comput. Surveys 55, 9 (2023), 1–52. https://doi.org/10.1145/3558000
  31. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Computing Surveys (CSUR) (2021). https://doi.org/ASurveyonHyperdimensionalComputingakaVectorSymbolicArchitectures
  32. Autoscaling bloom filter: controlling trade-off between true and false positives. Neural Computing and Applications 32 (2020), 3675–3684. https://doi.org/10.1007/s00521-019-04397-1
  33. Perceptron Theory Can Predict the Accuracy of Neural Networks. IEEE Transactions on Neural Networks and Learning Systems (2023), 1–15. https://doi.org/10.1109/TNNLS.2023.3237381
  34. In-memory factorization of holographic perceptual representations. Nature Nanotechnology (2023), 1–7. https://doi.org/10.1038/s41565-023-01357-8
  35. RADAR: A fast and energy-efficient programming technique for multiple bits-per-cell RRAM arrays. IEEE Transactions on Electron Devices 68, 9 (2021), 4397–4403. https://doi.org/10.1109/TED.2021.3097975
  36. Hyperdimensional computing with 3D VRRAM in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition. In 2016 IEEE International Electron Devices Meeting (IEDM). IEEE, 16–1. https://doi.org/10.1109/IEDM.2016.7838428
  37. Chisel: Reliability-and accuracy-aware optimization of approximate computational kernels. ACM Sigplan Notices 49, 10 (2014), 309–328. https://doi.org/10.1145/2714064.2660231
  38. PULP-HD: Accelerating brain-inspired high-dimensional computing on a parallel ultra-low power platform. In 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC). IEEE, 1–6. https://doi.org/10.1145/3195970.3196096
  39. CompHD: Efficient hyperdimensional computing using model compression. In 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED). IEEE, 1–6. https://doi.org/10.1109/ISLPED.2019.8824908
  40. SV Nagaev and VI Chebotarev. 2011. On the bound of proximity of the binomial distribution to the normal one. In Doklady Mathematics, Vol. 83. Springer, 19–21. https://doi.org/10.1134/S1064562411010030
  41. Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing. In IECON 2017-43rd Annual Conference of the IEEE Industrial Electronics Society. IEEE, 3276–3281. https://doi.org/10.1109/IECON.2017.8216554
  42. Search for a substring of characters using the theory of non-deterministic finite automata and vector-character architecture. Bulletin of Electrical Engineering and Informatics 9, 3 (2020), 1238–1250. https://doi.org/10.11591/eei.v9i3.1720
  43. Tony A Plate. 1994. Distributed representations and nested compositional structure. Citeseer.
  44. Tony A Plate. 2000. Analogy retrieval and processing with distributed vector representations. Expert systems 17, 1 (2000), 29–40. https://doi.org/10.1111/1468-0394.00125
  45. Tony A Plate. 2003. Holographic Reduced Representation: Distributed representation for cognitive structures. (2003).
  46. Stochd: Stochastic hyperdimensional system for efficient and robust learning from raw data. In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 1195–1200. https://doi.org/10.1109/DAC18074.2021.9586166
  47. Dmitri A Rachkovskij and Serge V Slipchenko. 2012. Similarity-based retrieval with structure-sensitive sparse binary distributed representations. Computational Intelligence 28, 1 (2012), 106–129. https://doi.org/10.1111/j.1467-8640.2011.00423.x
  48. Hyperdimensional biosignal processing: A case study for EMG-based hand gesture recognition. In 2016 IEEE International Conference on Rebooting Computing (ICRC). IEEE, 1–8. https://doi.org/10.1109/ICRC.2016.7738683
  49. High-dimensional computing as a nanoscalable paradigm. IEEE Transactions on Circuits and Systems I: Regular Papers 64, 9 (2017), 2508–2521. https://doi.org/10.1109/TCSI.2017.2705051
  50. Efficient biosignal processing using hyperdimensional computing: Network templates for combined learning and classification of exg signals. Proc. IEEE 107, 1 (2018), 123–143. https://doi.org/10.1109/JPROC.2018.2871163
  51. Multivariate time series analysis for driving style classification using neural networks and hyperdimensional computing. In 2021 IEEE Intelligent Vehicles Symposium (IV). IEEE, 602–609. https://doi.org/10.1109/IV48863.2021.9576028
  52. HDC-MiniROCKET: Explicit time encoding in time series classification with hyperdimensional computing. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8. https://doi.org/10.1109/IJCNN55064.2022.9892158
  53. ApproxTuner: a compiler and runtime system for adaptive approximations. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 262–277. https://doi.org/10.1145/3437801.3446108
  54. Monolithic 3D integration of logic and memory: Carbon nanotube FETs, resistive RAM, and silicon FETs. In 2014 IEEE International Electron Devices Meeting. IEEE, 27–4. https://doi.org/10.1109/IEDM.2014.7047120
  55. Constructing distributed time-critical applications using cognitive enabled services. Future Generation Computer Systems 100 (2019), 70–85. https://doi.org/10.1016/j.future.2019.04.010
  56. Unpaired Image Translation via Vector Symbolic Architectures. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXI. Springer, 17–32. https://doi.org/10.1007/978-3-031-19803-8_2
  57. Theoretical Foundations of Hyperdimensional Computing. Journal of Artificial Intelligence Research 72 (2021), 215–249. https://doi.org/10.48550/arXiv.2010.07426
  58. PBA: Percentile-Based Level Allocation for Multiple-Bits-Per-Cell RRAM. In ICCAD.
  59. Hyperdimensional computing exploiting carbon nanotube FETs, resistive RAM, and their monolithic 3D integration. IEEE Journal of Solid-State Circuits 53, 11 (2018), 3183–3196. https://doi.org/10.1109/JSSC.2018.2870560
  60. The hyperdimensional stack machine. Cognitive Computing (2018), 1–2.
  61. Pu (Luke) Yi and Sara Achour. 2023. Artifact for the OOPSLA 2023 Article ”Hardware-Aware Static Optimization of Hyperdimensional Computations”. https://doi.org/10.5281/zenodo.8329813
  62. Understanding hyperdimensional computing for parallel single-pass learning. Advances in Neural Information Processing Systems 35 (2022), 1157–1169.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (3)

  1. Pu
  2. Yi
  3. Sara Achour
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 post and received 0 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View