Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning (2402.01889v1)

Published 2 Feb 2024 in cs.AI and cs.LG

Abstract: Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems, as interpretable symbolic techniques provide formal behaviour guarantees. The challenge is how to effectively integrate neural and symbolic computation, to enable learning and reasoning from raw data. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling, whereas end-to-end approaches are limited in terms of scalability, due to the combinatorial explosion in the symbol grounding problem. In this paper, we leverage the implicit knowledge within foundation models to enhance the performance in NeSy tasks, whilst reducing the amount of data labelling and manual engineering. We introduce a new architecture, called NeSyGPT, which fine-tunes a vision-language foundation model to extract symbolic features from raw data, before learning a highly expressive answer set program to solve a downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has superior accuracy over various baselines, and can scale to complex NeSy tasks. Finally, we highlight the effective use of a LLM to generate the programmatic interface between the neural and symbolic components, significantly reducing the amount of manual engineering required.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Embed2sym-scalable neuro-symbolic reasoning via clustered embeddings. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, volume 19, pages 421–431, 2022.
  2. Logic tensor networks. Artificial Intelligence, 303:103649, 2022.
  3. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022.
  4. No routing needed between capsules. Neurocomputing, 463:545–553, 2021.
  5. Neuralfastlas: Fast logic-based learning from raw data. arXiv preprint arXiv:2310.05145, 2023.
  6. Ffnsl: feed-forward neural-symbolic learner. Machine Learning, 112(2):515–569, 2023.
  7. Neuro-symbolic learning of answer set programs from raw data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 3586–3596, 2023.
  8. Abductive knowledge induction from raw data. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1845–1851. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
  9. Bridging machine learning and logical reasoning by abductive learning. Advances in Neural Information Processing Systems, 32, 2019.
  10. Deep symbolic learning: Discovering symbols and rules from perceptions. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 3597–3605. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
  11. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. FLAP, 6(4):611–632, 2019.
  12. Scalable coupling of deep learning with logical reasoning. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 3615–3623. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
  13. A neuro-symbolic asp pipeline for visual question answering. Theory and Practice of Logic Programming, 22(5):739–754, 2022.
  14. Making sense of raw input. Artificial Intelligence, 299:103521, 2021.
  15. Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach. Cambridge University Press, Cambridge, UK, 2014.
  16. Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990.
  17. Block-recurrent transformers. Advances in Neural Information Processing Systems, 35:33248–33261, 2022.
  18. Leveraging Large Language Models to Generate Answer Set Programs. In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, pages 374–383, 8 2023.
  19. Richard M Karp. Reducibility among combinatorial problems. In Complexity of computer computations, pages 85–103. Springer, 1972.
  20. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv preprint arXiv:1911.03343, 2019.
  21. Logic-based learning of answer set programs. In Reasoning Web. Explainable Artificial Intelligence - 15th International Summer School 2019, Bolzano, Italy, September 20-24, 2019, Tutorial Lectures, pages 196–231, 2019.
  22. Fastlas: Scalable inductive logic programming incorporating domain-specific optimisation criteria. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2877–2885, 2020.
  23. The ilasp system for inductive learning of answer set programs. arXiv preprint arXiv:2005.00904, 2020.
  24. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019, 2022.
  25. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  26. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  27. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems, 31, 2018.
  28. Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. CoRR, abs/2305.19951, 2023.
  29. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192–25204, 2021.
  30. Gpt-4 technical report, 2023.
  31. OpenAI. Gpt-4 system card - openai, 2023.
  32. Neupsl: Neural probabilistic soft logic. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 4145–4153. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
  33. Logical neural networks. CoRR, abs/2006.13155, 2020.
  34. α𝛼\alphaitalic_α ilp: thinking visual scenes as differentiable logic programs. Machine Learning, 112(5):1465–1497, 2023.
  35. Plantdoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 249–253, 2020.
  36. Scalable neural-probabilistic answer set programming. arXiv preprint arXiv:2306.08397, 2023.
  37. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3619–3629, 2021.
  38. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023.
  39. Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660, 2023.
  40. Neurasp: Embracing neural networks into answer set programming. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1755–1762. International Joint Conferences on Artificial Intelligence Organization, 7 2020.
  41. Coupling large language models with logic programming for robust and general reasoning from text. arXiv preprint arXiv:2307.07696, 2023.
  42. Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 29(6):1091–1095, 2007.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Daniel Cunnington (9 papers)
  2. Mark Law (15 papers)
  3. Jorge Lobo (7 papers)
  4. Alessandra Russo (48 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets