Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 97 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 21 tok/s Pro
GPT-5 High 18 tok/s Pro
GPT-4o 92 tok/s Pro
GPT OSS 120B 468 tok/s Pro
Kimi K2 175 tok/s Pro
2000 character limit reached

BOLD: Boolean Logic Deep Learning (2405.16339v2)

Published 25 May 2024 in stat.ML and cs.LG

Abstract: Deep learning is computationally intensive, with significant efforts focused on reducing arithmetic complexity, particularly regarding energy consumption dominated by data movement. While existing literature emphasizes inference, training is considerably more resource-intensive. This paper proposes a novel mathematical principle by introducing the notion of Boolean variation such that neurons made of Boolean weights and inputs can be trained -- for the first time -- efficiently in Boolean domain using Boolean logic instead of gradient descent and real arithmetic. We explore its convergence, conduct extensively experimental benchmarking, and provide consistent complexity evaluation by considering chip architecture, memory hierarchy, dataflow, and arithmetic precision. Our approach achieves baseline full-precision accuracy in ImageNet classification and surpasses state-of-the-art results in semantic segmentation, with notable performance in image super-resolution, and natural language understanding with transformer-based models. Moreover, it significantly reduces energy consumption during both training and inference.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (114)
  1. E. Agustsson and R. Timofte. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  2. Proximal Mean-field for Neural Network Quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  3. Mirror Descent View for Neural Network Quantization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 2809–2817. PMLR, 13–15 Apr 2021.
  4. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  5. BinaryBERT: Pushing the Limit of BERT Quantization. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4334–4348. Association for Computational Linguistics, 2021.
  6. ProxQuant: Quantized Neural Networks via Proximal Operators. In International Conference on Learning Representations, 2018.
  7. C. Baldassi. Generalization Learning in a Perceptron with Binary Synapses. Journal of Statistical Physics, 136(5):902–916, Sep 2009.
  8. C. Baldassi and A. Braunstein. A Max-Sum Algorithm for Training Discrete Neural Networks. Journal of Statistical Mechanics: Theory and Experiment, 2015(8):P08008, 2015.
  9. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12):128101, 2015.
  10. MeliusNet: An Improved Network Architecture for Binary Neural Networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1439–1448, January 2021.
  11. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In Proceedings of the British Machine Vision Conference, pages 135.1–135.10. BMVA Press, 2012.
  12. Benchmark Analysis of Representative Deep Neural Network Architectures. IEEE Access, 6:64270–64277, 2018.
  13. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  14. An Analysis of Deep Neural Network Models for Practical Applications. arXiv preprint arXiv:1605.07678, 2016.
  15. AdderNet: Do We Really Need Multiplications in Deep Learning? In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  16. A Statistical Framework for Low-bitwidth Training of Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 883–894. Curran Associates, Inc., 2020.
  17. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint arXiv:1706.05587, 2017.
  18. Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks. SIGARCH Computer Architecture News, 44(3):367–379, jun 2016.
  19. Eyeriss: An Energy-efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-state Circuits, 52(1):127–138, 2016.
  20. Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges. IEEE Signal Processing Magazine, 35(1):126–136, 2018.
  21. Logarithmic Unbiased Quantization: Simple 4-bit Training in Deep Learning. arXiv:2112.10769, 2021.
  22. Bridging the Accuracy Gap for 2-Bit Quantized Neural Networks (QNN). arXiv preprint arXiv:1807.06964, 2018.
  23. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  24. BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  25. High-Accuracy Low-Precision Training. arXiv preprint arXiv:1803.03383, 2018.
  26. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019.
  27. Regularizing Activation Distribution for Training Binarized Deep Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  28. ShiDianNao: Shifting Vision Processing Closer to the Sensor. In Proceedings of the 42nd Annual International Symposium on Computer Architecture, pages 92–104, 2015.
  29. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  30. Binary DAD-Net: Binarized Driveable Area Detection Network for Autonomous Driving. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2295–2301. IEEE, 2020.
  31. Adult Neuroplasticity: More than 40 Years of Research. Neural plasticity, 2014, 2014.
  32. Estimating GPU Memory Consumption of Deep Learning Models. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1342–1352, 2020.
  33. Estimation of Energy Consumption in Machine Learning. Journal of Parallel and Distributed Computing, 134:75–88, 2019.
  34. A Survey of Quantization Methods for Efficient Neural Network Inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC, 2022.
  35. C. Grimm and N. Verma. Neural Network Training on In-Memory-Computing Hardware With Radix-4 Gradients. IEEE Transactions on Circuits and Systems I: Regular Papers I, 69(10):4056–4068, 2022.
  36. Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket. arXiv preprint arXiv:2211.12933, 2022.
  37. Deep Learning with Limited Numerical Precision. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1737–1746, Lille, France, 07–09 Jul 2015. PMLR.
  38. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In International Conference on Learning Representations, 2015.
  39. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  40. D. O. Hebb. The Organization of Behavior: A Neuropsychological Theory. Psychology press, 2005.
  41. Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  42. M. Horowitz. 1.1 Computing’s Energy Problem (and What We Can Do about It). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14, 2014.
  43. Loss-aware Binarization of Deep Networks. In International Conference on Learning Representations, 2016.
  44. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint arXiv:1704.04861, 2017.
  45. DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9924–9935, 2022.
  46. Single Image Super-Resolution from Transformed Self-Exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  47. Binarized Neural Networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  48. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. The Journal of Machine Learning Research, 18(1):6869–6898, 2017.
  49. S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 448–456, Lille, France, 07–09 Jul 2015. PMLR.
  50. R. Ito and T. Saito. Dynamic Binary Neural Networks and Evolutionary Learning. In The 2010 International Joint Conference on Neural Networks, pages 1–5. IEEE, 2010.
  51. F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization. In International Conference on Learning Representations, 2021.
  52. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 3252–3261. PMLR, 09–15 Jun 2019.
  53. D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
  54. A. Krizhevsky and G. Hinton. Learning Multiple Layers of Features from Tiny Images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  55. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  56. Understanding Reuse, Performance, and Hardware Cost of DNN Dataflow: A Data-centric Approach. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 754–768, 2019.
  57. Proximity Preserving Binary Code Using Signed Graph-Cut. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4535–4544, April 2020.
  58. Deep Learning. Nature, 521(7553):436–444, 2015.
  59. INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17325–17334, October 2023.
  60. Training Quantized Nets: A Deeper Understanding. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  61. Z. Li and C. M. De Sa. Dimension-Free Bounds for Low-Precision Training. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  62. Ascend: a Scalable and Unified Architecture for Ubiquitous Deep Neural Network Computing : Industry Track Paper. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 789–801, 2021.
  63. Enhanced Deep Residual Networks for Single Image Super-Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017.
  64. Rotated Binary Neural Network. In Advances in Neural Information Processing Systems, volume 33, pages 7474–7485. Curran Associates, Inc., 2020.
  65. RB-Net: Training Highly Accurate and Efficient Binary Neural Networks with Reshaped Point-wise Convolution and Balanced activation. IEEE Transactions on Circuits and Systems for Video Technology, 32(9):6414–6424, 2022.
  66. BiT: Robustly Binarized Multi-distilled Transformer. In Advances in Neural Information Processing Systems, volume 35, pages 14303–14316. Curran Associates, Inc., 2022.
  67. ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions. In Proceedings of the European Conference on Computer Vision (ECCV), August 2020.
  68. Bi-Real Net: Enhancing the Performance of 1-bit CNNs with Improved Representational Capability and Advanced Training Algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  69. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  70. I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2017.
  71. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), July 2001.
  72. Training Binary Neural Networks with Real-to-binary Convolutions. In International Conference on Learning Representations, 2020.
  73. Benchmarking the Memory Hierarchy of Modern GPUs. In IFIP International Conference on Network and Parallel Computing, pages 144–156. Springer, 2014.
  74. G. Morse and K. O. Stanley. Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 477–484, 2016.
  75. Binary Neural Networks as a General-propose Compute Paradigm for On-device Computer Vision. arXiv preprint arXiv:2202.03716, 2022.
  76. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  77. C. Philippenko and A. Dieuleveut. Bidirectional Compression in Heterogeneous Settings for Distributed or Federated Learning with Partial Participation: Tight Convergence Guarantees. arXiv preprint arXiv:2006.14591, 2020.
  78. BiBERT: Accurate Fully Binarized BERT. In International Conference on Learning Representations, 2022.
  79. Binary Neural Networks: A Survey. Pattern Recognition, 105:107281, 2020.
  80. Forward and Backward Information Retention for Accurate Binary Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  81. BiBench: Benchmarking and Analyzing Network Binarization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28351–28388. PMLR, 23–29 Jul 2023.
  82. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In Proceedings of the European Conference on Computer Vision (ECCV), October 2016.
  83. Memory Devices and Applications for in-memory Computing. Nature Nanotechnology, 15(7):529–544, 2020.
  84. Y. S. Shao and D. Brooks. Energy Characterization and Instruction-Level Energy Model of Intelś Xeon Phi Processor. In International Symposium on Low Power Electronics and Design (ISLPED), pages 389–394, 2013.
  85. An Energy-Efficient Deep Convolutional Neural Network Inference Processor With Enhanced Output Stationary Dataflow in 65-nm CMOS. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 28(1):87–100, 2019.
  86. K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations, 2015.
  87. Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  88. Energy and Policy Considerations for Deep Learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, Jul 2019. Association for Computational Linguistics.
  89. Ultra-Low Precision 4-bit Training of Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 1796–1807. Curran Associates, Inc., 2020.
  90. Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proceedings of the IEEE, 105(12):2295–2329, 2017.
  91. How to Evaluate Deep Neural Network Processors: Tops/w (Alone) Considered Harmful. IEEE Solid-State Circuits Magazine, 12(3):28–41, 2020.
  92. M. Tan and Q. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 09–15 Jun 2019.
  93. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  94. Fixing the Train-Test Resolution Discrepancy. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  95. AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets. In Proceedings of the European Conference on Computer Vision (ECCV), October 2022.
  96. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  97. In-Memory Computing: Advances and Prospects. IEEE Solid-State Circuits Magazine, 11(3):43–55, 2019.
  98. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations, 2019.
  99. Enabling Binary Neural Network Training on the Edge. In Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning, pages 37–38, 2021.
  100. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4):65–76, apr 2009.
  101. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 38087–38099. PMLR, 23–29 Jul 2023.
  102. Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies. In Proceedings of the European Conference on Computer Vision (ECCV), October 2022.
  103. A Method to Estimate the Energy Consumption of Deep Neural Networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pages 1916–1920, October 2017.
  104. Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  105. Interstellar: Using Halide’s Scheduling Language to Analyze DNN Accelerators. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 369–383, 2020.
  106. Searching for Low-Bit Weights in Quantized Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 4091–4102. Curran Associates, Inc., 2020.
  107. S. Yu and P.-Y. Chen. Emerging Memory Technologies: Recent Trends and Prospects. IEEE Solid-State Circuits Magazine, 8(2):43–56, 2016.
  108. On Single Image Scale-up using Sparse-Representations. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pages 711–730. Springer, 2012.
  109. LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  110. Mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations, 2018.
  111. Low-Precision Stochastic Gradient Langevin Dynamics. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 26624–26644. PMLR, 17–23 Jul 2022.
  112. PokeBNN: A Binary Pursuit of Lightweight Accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12475–12485, June 2022.
  113. Z. Zhang. Derivation of Backpropagation in Convolutional Neural Network (CNN). University of Tennessee, Knoxville, TN, 22:23, 2016.
  114. Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

HackerNews

  1. Bold: Boolean Logic Deep Learning (2 points, 1 comment)