Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 79 tok/s
Gemini 2.5 Pro 60 tok/s Pro
GPT-5 Medium 25 tok/s Pro
GPT-5 High 29 tok/s Pro
GPT-4o 117 tok/s Pro
Kimi K2 201 tok/s Pro
GPT OSS 120B 466 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Bayesian Semi-structured Subspace Inference (2401.12950v1)

Published 23 Jan 2024 in cs.LG and stat.ML

Abstract: Semi-structured regression models enable the joint modeling of interpretable structured and complex unstructured feature effects. The structured model part is inspired by statistical models and can be used to infer the input-output relationship for features of particular importance. The complex unstructured part defines an arbitrary deep neural network and thereby provides enough flexibility to achieve competitive prediction performance. While these models can also account for aleatoric uncertainty, there is still a lack of work on accounting for epistemic uncertainty. In this paper, we address this problem by presenting a Bayesian approximation for semi-structured regression models using subspace inference. To this end, we extend subspace inference for joint posterior sampling from a full parameter space for structured effects and a subspace for unstructured effects. Apart from this hybrid sampling scheme, our method allows for tunable complexity of the subspace and can capture multiple minima in the loss landscape. Numerical experiments validate our approach's efficacy in recovering structured effect parameter posteriors in semi-structured models and approaching the full-space posterior distribution of MCMC for increasing subspace dimension. Further, our approach exhibits competitive predictive performance across simulated and real-world datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. A state-space neural network for modeling dynamical nonlinear systems. In IJCCI (NCTA), pages 369–376.
  2. Deep conditional transformation models. In Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), pages 3–18. Springer International Publishing.
  3. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7–10. ACM.
  4. Designing neural networks from statistical models: A new approach to data exploration. In KDD, pages 45–50.
  5. Statistical models as building blocks of neural networks. Communications in statistics-theory and methods, 26(4):991–1009.
  6. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089–20103.
  7. Generalized additive models from a neural network perspective. In Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007), pages 265–270. IEEE.
  8. Automation of generalized additive neural networks for predictive data mining. Applied Artificial Intelligence, 25(5):380–425.
  9. Frequentist Uncertainty Quantification in Semi-Structured Neural Networks. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pages 1924–1941. PMLR.
  10. Uci machine learning repository.
  11. Bernstein Flows for Flexible Posteriors in Variational Bayes. arXiv preprint arXiv:2202.05650.
  12. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
  13. Deep Bayesian regression models.
  14. International Skin Imaging Collaboration (2020). Siim-isic 2020 challenge dataset. Accessed on September 28, 2023.
  15. Subspace Inference for Bayesian Deep Learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 1169–1179. PMLR.
  16. Learning active subspaces for effective and scalable uncertainty quantification in deep neural networks. arXiv preprint arXiv:2309.03061.
  17. Deep interpretable ensembles.
  18. Deep and interpretable regression models for ordinal outcomes. Pattern Recognition, 122:108263.
  19. DeepPAMM: Deep Piecewise Exponential Additive Mixed Models for Complex Hazard Structures in Survival Analysis. In Advances in Knowledge Discovery and Data Mining (PAKDD), pages 249–261. Springer International Publishing.
  20. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
  21. Elliptical slice sampling. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 541–548. JMLR Workshop and Conference Proceedings.
  22. Generalized linear models. Journal of the Royal Statistical Society Series A: Statistics in Society, 135(3):370–384.
  23. Potts, W. J. (1999). Generalized additive neural networks. In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 194–200.
  24. A wide and deep neural network for survival analysis from anatomical shape and tabular clinical data. Communications in Computer and Information Science, page 453–464.
  25. Rügamer, D. (2023). A new PHO-rmula for improved performance of semi-structured networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29291–29305. PMLR.
  26. Semi-structured distributional regression. The American Statistician, 0(0):1–12.
  27. Deep transformation models: Tackling complex regression problems with neural network based transformation models. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2476–2481. IEEE.
  28. Bayesian deep net GLM and GLMM. Journal of Computational and Graphical Statistics, 29(1):97–113.
  29. Wahba, G. (1990). Spline models for observational data. SIAM.
  30. Towards efficient posterior sampling in deep neural networks via symmetry removal. In Machine Learning and Knowledge Discovery in Databases (ECML-PKDD). Springer International Publishing.
  31. Wood, S. N. (2017). Generalized additive models: an introduction with R. CRC press.
  32. Learning neural network subspaces. In International Conference on Machine Learning, pages 11217–11227. PMLR.
  33. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations.
  34. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up