Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

Efficient Model Compression for Bayesian Neural Networks (2411.00273v1)

Published 1 Nov 2024 in cs.LG, stat.AP, and stat.ML

Abstract: Model Compression has drawn much attention within the deep learning community recently. Compressing a dense neural network offers many advantages including lower computation cost, deployability to devices of limited storage and memories, and resistance to adversarial attacks. This may be achieved via weight pruning or fully discarding certain input features. Here we demonstrate a novel strategy to emulate principles of Bayesian model selection in a deep learning setup. Given a fully connected Bayesian neural network with spike-and-slab priors trained via a variational algorithm, we obtain the posterior inclusion probability for every node that typically gets lost. We employ these probabilities for pruning and feature selection on a host of simulated and real-world benchmark data and find evidence of better generalizability of the pruned model in all our experiments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. “Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee.”
  2. “Ensemble Learning in Bayesian Neural Networks.” In Generalization in Neural Networks and Machine Learning, 215–237. Springer Verlag.
  3. “The generalization-stability tradeoff in neural network pruning.” Advances in Neural Information Processing Systems, 33: 20852–20864.
  4. “Reconciling modern machine-learning practice and the classical bias–variance trade-off.” Proceedings of the National Academy of Sciences, 116(32): 15849–15854.
  5. Bertin-Mahieux, T. (2011). “Year Prediction MSD.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C50K61.
  6. “What is the State of Neural Network Pruning?” In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org.
  7. “Weight Uncertainty in Neural Networks.” In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 1613–1622. JMLR.org.
  8. “Explaining bayesian neural networks.” arXiv preprint arXiv:2108.10346.
  9. Statistical Inference. Thomson Learning.
  10. “Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges.” IEEE Signal Processing Magazine, 35(1): 126–136.
  11. “Prediction Via Orthogonalized Model Mixing.” Journal of the American Statistical Association, 91(435): 1197–1208.
  12. “Model Uncertainty.” Statistical Science, 19(1): 81 – 94.
  13. “Memory Bounded Deep Convolutional Networks.” arXiv:1412.1442.
  14. “Condition Based Maintenance of Naval Propulsion Plants.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5K31K.
  15. “Wine Quality.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C56S3T.
  16. “Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1.”
  17. “Consistent feature selection for analytic deep neural networks.” In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, 2420–2431. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1959eb9d5a0f7ebc58ebde81d5df400d-Paper.pdf
  18. “Bayesian Neural Networks for Selection of Drug Sensitive Genes.” Journal of the American Statistical Association, 113(523): 955–972. PMID: 31354179.
  19. Gal, Y. (2016). “Uncertainty in Deep Learning.” Ph.D. thesis, University of Cambridge.
  20. “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, 1050–1059. JMLR.org.
  21. “Variable Selection via Gibbs Sampling.” Journal of the American Statistical Association, 88(423): 881–889.
  22. — (1997). “Approaches for Bayesian Variable Selection.” Statistica Sinica, 7(2): 339–373.
  23. “Yacht Hydrodynamics.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5XG7R.
  24. “Model Selection in Bayesian Neural Networks via Horseshoe Priors.”
  25. “Bayesian Neural Networks: An Introduction and Survey.” In Case Studies in Applied Bayesian Data Science, 45–87. Springer.
  26. “Pruning Neural Networks During Training by Backpropagation.” In Proceedings of TENCON’94 - 1994 IEEE Region 10’s 9th Annual International Conference on: ’Frontiers of Computer Technology’, 805–808 vol.2.
  27. Deep Learning. MIT Press. http://www.deeplearningbook.org.
  28. “Dynamic Network Surgery for Efficient DNNs.”
  29. “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman Coding.” arXiv:1510.00149.
  30. “Hedonic housing prices and the demand for clean air.” Journal of Environmental Economics and Management, 5: 81–102. URL https://api.semanticscholar.org/CorpusID:55571328
  31. “Probabilistic backpropagation for scalable learning of Bayesian neural networks.” In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 1861–1869. JMLR.org.
  32. “Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights.” In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT ’93, 5–13. New York, NY, USA: Association for Computing Machinery.
  33. “Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks.” The Journal of Machine Learning Research, 22(1): 10882–11005.
  34. Hornik, K. (1991). “Approximation Capabilities of Multilayer Feedforward Networks.” Neural Networks, 4(2): 251 – 257.
  35. “Spike and Slab Variable Selection: Frequentist and Bayesian Strategies.” The Annals of Statistics, 33(2): 730–773.
  36. Janowsky, S. A. (1989). “Pruning versus Clipping in Neural Networks.” Physical Review A, 39: 6600–6603.
  37. “Variational Dropout and the Local Reparameterization Trick.” In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  38. “Auto-Encoding Variational Bayes.” In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
  39. “Learning Multiple Layers of Features from Tiny Images.”
  40. “Deep Learning.” Nature, 521: 436–44.
  41. “Gradient-based learning applied to document recognition.” Proceedings of the IEEE, 86(11): 2278–2324.
  42. “Optimal Brain Damage.” In Advances in Neural Information Processing Systems, 598–605. Morgan Kaufmann.
  43. “Deep Feature Selection: Theory and Application to Identify Enhancers and Promoters.” In Przytycka, T. M. (ed.), Research in Computational Molecular Biology, 205–217. Cham: Springer International Publishing.
  44. Lindley, D. V. (1957). “A Statistical Paradox.” Biometrika, 44(1-2): 187–192.
  45. Liu, J. (2021). “Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon.” In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, 3124–3132. PMLR.
  46. “Implicit Generative Prior for Bayesian Neural Networks.”
  47. “Bayesian Compression for Deep Learning.”
  48. MacKay, D. J. C. (1992). “A Practical Bayesian Framework for Backpropagation Networks.” Neural Computation, 4(3): 448–472.
  49. “Bayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association, 83(404): 1023–1032.
  50. “Variational Dropout Sparsifies Deep Neural Networks.” In International Conference on Machine Learning, 2498–2507. PMLR.
  51. “Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment.” In Proceedings of the 1st International Conference on Neural Information Processing Systems, 107–115. Cambridge, MA, USA: MIT Press.
  52. Neal, R. M. (1995). “Bayesian Learning for Neural Networks.” Ph.D. thesis, University of Toronto.
  53. “A Review of Bayesian Variable Selection Methods: What, How and Which.” Bayesian Analysis, 4(1): 85 – 117.
  54. “An Accurate Comparison of Methods for Quantifying Variable Importance in Artificial Neural Networks using Simulated Data.” Ecological Modelling, 3(178): 389–397.
  55. “Robust Sparse Regularization: Defending Adversarial Attacks Via Regularized Sparse Network.” In Proceedings of the 2020 on Great Lakes Symposium on VLSI, GLSVLSI ’20, 125–130. New York, NY, USA: Association for Computing Machinery. URL https://doi.org/10.1145/3386263.3407651
  56. Rana, P. (2013). “Physicochemical Properties of Protein Tertiary Structure.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5QW3H.
  57. “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks.”
  58. “Hierarchical Bayesian formulations for selecting variables in regression models.” Statistics in medicine, 31(11-12): 1221–1237.
  59. “Learning Discrete Weights Using the Local Reparameterization Trick.” In International Conference on Learning Representations.
  60. “Nonparametric Regression Using Bayesian Variable Selection.” Journal of Econometrics, 75(2): 317–343.
  61. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” J. Mach. Learn. Res., 15(1): 1929–1958.
  62. Ström, N. (1997). “Sparse Connection and Pruning in Large Dynamic Artificial Neural Networks.” In Proceedings of 5th European Conference on Speech Communication and Technology (Eurospeech 1997), 2807–2810.
  63. “Combined Cycle Power Plant.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5002N.
  64. “Reparameterization trick for discrete variables.”
  65. “Energy Efficiency.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C51307.
  66. “Soft Weight-Sharing for Neural Network Compression.”
  67. “Fast Dropout Training.” In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, 118–126. PMLR.
  68. Williams, R. J. (1992). “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.” Machince Learning, 8(3–4): 229–256.
  69. Yeh, I.-C. (2007). “Concrete Compressive Strength.” UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5PK67.
  70. Zagoruyko, S. (2015). “92.45 on cifar-10 in torch.” http://torch.ch/blog/2015/07/30/cifar.html.
  71. “Understanding deep learning (still) requires rethinking generalization.” Communications of the ACM, 64(3): 107–115.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets