Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning with Logical Constraints but without Shortcut Satisfaction (2403.00329v1)

Published 1 Mar 2024 in cs.AI and cs.LG

Abstract: Recent studies in neuro-symbolic learning have explored the integration of logical knowledge into deep learning via encoding logical constraints as an additional loss function. However, existing approaches tend to vacuously satisfy logical constraints through shortcuts, failing to fully exploit the knowledge. In this paper, we present a new framework for learning with logical constraints. Specifically, we address the shortcut satisfaction issue by introducing dual variables for logical connectives, encoding how the constraint is satisfied. We further propose a variational framework where the encoded logical constraint is expressed as a distributional loss that is compatible with the model's original training loss. The theoretical analysis shows that the proposed approach bears salient properties, and the experimental evaluations demonstrate its superior performance in both model generalizability and constraint satisfaction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Leonard Adolphs. Non convex-concave saddle point optimization. Master’s thesis, ETH Zurich, 2018.
  2. Semantic probabilistic layers for neuro-symbolic learning. arXiv preprint arXiv:2206.00426, 2022.
  3. Learning from rules generalizing labeled exemplars. In International Conference on Learning Representations, 2020.
  4. Hinge-loss markov random fields and probabilistic soft logic. 2017.
  5. Logic tensor networks. Artificial Intelligence, 303:103649, 2022.
  6. A meta-transfer objective for learning to disentangle causal mechanisms. In International Conference on Learning Representations, 2020.
  7. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017.
  8. Convex optimization. Cambridge university press, 2004.
  9. Learning and inference with constraints. In Proceedings of the 23rd national conference on Artificial intelligence, pp.  1513–1518, 2008.
  10. Understanding deep architectures with reasoning layer. In Advances in Neural Information Processing Systems, 2020.
  11. Rizwan A Choudrey. Variational methods for Bayesian independent component analysis. PhD thesis, University of Oxford Oxford, UK, 2002.
  12. Inductive logic programming at 30: a new introduction. arXiv preprint arXiv:2008.07912, 2020.
  13. Bridging machine learning and logical reasoning by abductive learning. In Advances in Neural Information Processing Systems, pp. 2815–2826, 2019.
  14. A knowledge compilation map. Journal of Artificial Intelligence Research, 17:229–264, 2002.
  15. Stochastic subgradient method converges at the rate 𝒪⁢(k−1/4)𝒪superscript𝑘14\mathcal{O}(k^{-1/4})caligraphic_O ( italic_k start_POSTSUPERSCRIPT - 1 / 4 end_POSTSUPERSCRIPT ) on weakly convex functions. arXiv preprint arXiv:1802.02988, 2018.
  16. Paul Adrien Maurice Dirac. The principles of quantum mechanics. Number 27. Oxford university press, 1981.
  17. Neural logic machines. In International Conference on Learning Representations, 2019.
  18. Dc3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
  19. Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming, 178(1):503–558, 2019.
  20. The paradoxical success of fuzzy logic. IEEE expert, 9(4):3–49, 1994.
  21. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1–64, 2018.
  22. Dl2: training and querying neural networks with logic. In International Conference on Machine Learning, pp. 1931–1941. PMLR, 2019.
  23. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. Journal of Applied Logics, 6(4):611–632, 2019.
  24. On the relation between loss functions and t-norms. In International Conference on Inductive Logic Programming, pp.  36–45. Springer, 2019.
  25. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523, 2022.
  26. Petr Hájek. Metamathematics of fuzzy logic, volume 4. Springer Science & Business Media, 2013.
  27. Multiplexnet: Towards fully satisfied logical constraints in neural networks. arXiv preprint arXiv:2111.01564, 2021.
  28. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp.  2410–2420, August 2016.
  29. Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
  30. What is local optimality in nonconvex-nonconcave minimax optimization? In International Conference on Machine Learning, pp. 4880–4889. PMLR, 2020.
  31. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  7482–7491, 2018.
  32. A short introduction to probabilistic soft logic. In Proceedings of the NIPS Workshop on Probabilistic Programming: Foundations and Applications, pp.  1–4, 2012.
  33. Learning multiple layers of features from tiny images. 2009.
  34. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2, 1989.
  35. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning, pp. 5884–5894. PMLR, 2020.
  36. Augmenting neural networks with first-order logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.  292–302, July 2019.
  37. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning, pp. 6083–6093. PMLR, 2020.
  38. Pa-gd: On the convergence of perturbed alternating gradient descent to second-order stationary points for structured nonconvex optimization. In International Conference on Machine Learning, pp. 4134–4143. PMLR, 2019.
  39. Deepproblog: Neural probabilistic logic programming. Advances in Neural Information Processing Systems, 31:3749–3759, 2018.
  40. Survey of multi-objective optimization methods for engineering. Structural and multidisciplinary optimization, 26(6):369–395, 2004.
  41. From statistical relational to neural symbolic artificial intelligence: a survey. arXiv preprint arXiv:2108.11451, 2021.
  42. Antonio Martinón. Distance to the intersection of two sets. Bulletin of the Australian Mathematical Society, 70(2):329–341, 2004.
  43. Learning reasoning strategies in end-to-end differentiable proving. In International Conference on Machine Learning, pp. 6938–6949. PMLR, 2020.
  44. Methods of theoretical physics. American Journal of Physics, 22(6):410–413, 1954.
  45. Stephen Muggleton. Inductive logic programming. Number 38. Morgan Kaufmann, 1992.
  46. A primal dual formulation for deep learning with constraints. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  47. Subgradient methods for saddle-point problems. Journal of optimization theory and applications, 142(1):205–228, 2009.
  48. Michael JD Powell. On search directions for minimization algorithms. Mathematical programming, 4(1):193–201, 1973.
  49. Ralph Tyrell Rockafellar. Convex analysis. Princeton university press, 2015.
  50. End-to-end differentiable proving. In Advances in Neural Information Processing Systems, pp. 3791–3803, 2017.
  51. A linear programming formulation for global inference in natural language tasks. Technical report, Illinois Univ at Urbana-Champaign Dept of Computer Science, 2004.
  52. Tim Roughgarden. Algorithmic game theory. Communications of the ACM, 53(7):78–86, 2010.
  53. Competitive gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  54. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 525–536, 2018.
  55. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302:103602, 2022.
  56. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning, pp. 6545–6554. PMLR, 2019.
  57. Mark J Wierman. An introduction to the mathematics of uncertainty: including set theory, logic, probability, fuzzy sets, rough sets, and evidence theory. Creighton University. Retrieved, 16, 2016.
  58. Embedding symbolic knowledge into deep networks. In Advances in Neural Information Processing Systems, pp. 4233–4243, 2019.
  59. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pp. 5502–5511. PMLR, 2018.
  60. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information Processing Systems, pp. 2316–2325, 2017.
  61. Faster single-loop algorithms for minimax optimization without strong concavity. arXiv preprint arXiv:2112.05604, 2021.
  62. Neurasp: Embracing neural networks into answer set programming. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 1755–1762. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10.24963/ijcai.2020/243. URL https://doi.org/10.24963/ijcai.2020/243. Main track.
  63. Injecting logical constraints into neural networks via straight-through estimators. In International Conference on Machine Learning, pp. 25096–25122. PMLR, 2022.
  64. L.A. Zadeh. Fuzzy sets. Information and Control, 8(3):338–353, 1965.
  65. Zhi-Hua Zhou. Abductive learning: towards bridging machine learning and logical reasoning. Science China Information Sciences, 62:1–3, 2019.
Citations (14)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now