Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

On the Independence Assumption in Neurosymbolic Learning (2404.08458v2)

Published 12 Apr 2024 in stat.ML, cs.AI, and cs.LG

Abstract: State-of-the-art neurosymbolic learning systems use probabilistic reasoning to guide neural networks towards predictions that conform to logical constraints over symbols. Many such systems assume that the probabilities of the considered symbols are conditionally independent given the input to simplify learning and reasoning. We study and criticise this assumption, highlighting how it can hinder optimisation and prevent uncertainty quantification. We prove that loss functions bias conditionally independent neural networks to become overconfident in their predictions. As a result, they are unable to represent uncertainty over multiple valid options. Furthermore, we prove that these loss functions are difficult to optimise: they are non-convex, and their minima are usually highly disconnected. Our theoretical analysis gives the foundation for replacing the conditional independence assumption and designing more expressive neurosymbolic probabilistic models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Semantic probabilistic layers for neuro-symbolic learning. Advances in Neural Information Processing Systems, 35:29944–29959, 2022a.
  2. Neuro-Symbolic Entropy Regularization. In The 38th Conference on Uncertainty in Artificial Intelligence, June 2022b.
  3. A pseudo-semantic loss for autoregressive models with logical constraints. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023.
  4. Logic Tensor Networks. Artificial Intelligence, 303:103649, February 2022. ISSN 0004-3702. doi: 10.1016/j.artint.2021.103649.
  5. Interpretable Neural-Symbolic Concept Reasoning. In Proceedings of the 40th International Conference on Machine Learning, pages 1801–1825. PMLR, July 2023.
  6. Handling epistemic and aleatory uncertainties in probabilistic circuits. Machine Learning, 111(4):1259–1301, April 2022. ISSN 1573-0565. doi: 10.1007/s10994-021-06086-4.
  7. On the Number of Prime Implicants. Discrete Mathematics, 24(1):7–11, January 1978. doi: 10.1016/0012-365X(78)90168-1.
  8. On probabilistic inference by weighted model counting. Artificial Intelligence, 172(6):772–799, April 2008. ISSN 0004-3702. doi: 10.1016/j.artint.2007.11.002.
  9. Deep Learning with Logical Constraints. In Luc De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5478–5485. ijcai.org, 2022. doi: 10.24963/ijcai.2022/767.
  10. ROAD-R: The autonomous driving dataset with logical requirements. Machine Learning, 112(9):3261–3291, September 2023. ISSN 0885-6125, 1573-0565. doi: 10.1007/s10994-023-06322-z.
  11. Taming the sigmoid bottleneck: Provably argmaxable sparse multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12208–12216, 2024.
  12. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/guo17a.html.
  13. Allen Hatcher. Algebraic Topology. Cambridge University Press, 2002.
  14. Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning. In Advances in Neural Information Processing Systems, May 2021.
  15. Computational Homology, volume 157 of Applied Mathematical Sciences. Springer, New York, NY, 2004. ISBN 978-1-4419-2354-7 978-0-387-21597-6. doi: 10.1007/b97315.
  16. Learning with logical constraints but without shortcut satisfaction. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=M2unceRvqhh.
  17. Softened Symbol Grounding for Neuro-symbolic Systems. In The Eleventh International Conference on Learning Representations, February 2023b.
  18. DeepProbLog: Neural probabilistic logic programming. In Samy Bengio, Hanna M Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, 2018.
  19. Neural probabilistic logic programming in DeepProbLog. Artificial Intelligence, 298:103504, 2021. ISSN 0004-3702. doi: 10.1016/j.artint.2021.103504.
  20. Neuro-Symbolic Reasoning Shortcuts: Mitigation Strategies and their Limitations, March 2023a.
  21. Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts. In Thirty-Seventh Conference on Neural Information Processing Systems, May 2023b.
  22. Bears make neuro-symbolic models aware of their reasoning shortcuts, 2024.
  23. Jiří Matoušek. Using the Borsuk–Ulam Theorem. Springer, Berlin, Heidelberg, 2008. ISBN 978-3-540-00362-5 978-3-540-76649-0. doi: 10.1007/978-3-540-76649-0.
  24. Edward J McCluskey. Minimization of boolean functions. The Bell System Technical Journal, 35(6):1417–1444, 1956.
  25. NeuPSL: Neural Probabilistic Soft Logic. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 4145–4153, Macau, SAR China, August 2023. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-956792-03-4. doi: 10.24963/ijcai.2023/461.
  26. W. V. Quine. The Problem of Simplifying Truth Functions. The American Mathematical Monthly, 59(8):521–531, 1952. ISSN 0002-9890. doi: 10.2307/2308219.
  27. Willard V Quine. On cores and prime implicants of truth functions. The American Mathematical Monthly, 66(9):755–760, 1959.
  28. J. Paul Roth. Algebraic Topological Methods for the Synthesis of Switching Systems. I. Transactions of the American Mathematical Society, 88(2):301–326, 1958. ISSN 0002-9947. doi: 10.2307/1993216.
  29. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. CEUR Workshop Proceedings, 1768, 2016. ISSN 16130073.
  30. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302:103602, 2022. ISSN 0004-3702. doi: 10.1016/j.artint.2021.103602.
  31. A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inference. In Thirty-Seventh Conference on Neural Information Processing Systems. arXiv, May 2023.
  32. A compositional atlas of tractable circuit operations for probabilistic inference. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13189–13201. Curran Associates, Inc., 2021.
  33. On regularization and inference with label constraints. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 35740–35762. PMLR, 2023a.
  34. On Learning Latent Models with Multi-Instance Weak Supervision. In Thirty-Seventh Conference on Neural Information Processing Systems, June 2023b.
  35. Deepstochlog: Neural stochastic logic programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10090–10100, 2022.
  36. A semantic loss function for deep learning with symbolic knowledge. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5502–5511, Stockholmsmässan, Stockholm Sweden, 2018. PMLR.
  37. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HkwZSG-CZ.
  38. NeurASP: Embracing neural networks into answer set programming. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1755–1762. International Joint Conferences on Artificial Intelligence Organization, July 2020. doi: 10.24963/ijcai.2020/243.
  39. Günter M. Ziegler. Lectures on polytopes. Springer-Verlag, New York, 1995.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now