Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Weakly Supervised Reasoning by Neuro-Symbolic Approaches (2309.13072v1)

Published 19 Sep 2023 in cs.CL, cs.AI, and cs.LG

Abstract: Deep learning has largely improved the performance of various NLP tasks. However, most deep learning models are black-box machinery, and lack explicit interpretation. In this chapter, we will introduce our recent progress on neuro-symbolic approaches to NLP, which combines different schools of AI, namely, symbolism and connectionism. Generally, we will design a neural system with symbolic latent structures for an NLP task, and apply reinforcement learning or its relaxation to perform weakly supervised reasoning in the downstream task. Our framework has been successfully applied to various tasks, including table query reasoning, syntactic structure reasoning, information extraction reasoning, and rule reasoning. For each application, we will introduce the background, our approach, and experimental results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Towards string-to-tree neural machine translation. In ACL, pages 132–140, 2017.
  2. The Description Logic Handbook: Theory, Implementation and Applications. Cambridge University Press, 2003.
  3. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
  4. Measuring abstract reasoning in neural networks. In ICML, pages 511–520, 2018.
  5. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In ACL, pages 16–23, 2003.
  6. Semantic parsing on freebase from question-answer pairs. In EMNLP, pages 1533–1544, 2013.
  7. Semantic parsing via paraphrasing. In ACL, pages 1415–1425, 2014.
  8. John A Bernard. Use of a rule-based system for process control. IEEE Control Systems Magazine, 8(5):3–13, 1988.
  9. Robust incremental neural semantic graph parsing. In NAACL, pages 1215–1226, 2017.
  10. Connectionism and information processing abstractions. AI Magazine, 9(4):24–24, 1988.
  11. Learning to compose task-specific tree structures. In AAAI, pages 5094–5101, 2018.
  12. What does BERT look at? An analysis of BERT attention. In BlackboxNLP, pages 276–286, 2019.
  13. Unsupervised chunking as syntactic structure induction with a knowledge-transfer approach. In Findings of EMNLP, pages 3626–3634, 2021.
  14. Li Dong and Mirella Lapata. Language to logical form with neural attention. In ACL, pages 33–43, 2016.
  15. Unsupervised latent tree induction with deep inside-outside recursive autoencoders. In NAACL-HLT, pages 1129–1141, 2019.
  16. Template filling with generative Transformers. In NAACL, pages 909–914, 2021.
  17. Herbert B Enderton. A Mathematical Introduction to Logic. Elsevier, 2001.
  18. Paraphrase generation with latent bag of words. In NeurIPS, pages 13645–13656, 2019.
  19. Ashok Goel. Looking back, looking ahead: Symbolic versus connectionist AI. AI Magazine, 42(4):83–85, 2022.
  20. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014.
  21. A deep generative framework for paraphrase generation. In AAAI, pages 5149–5156, 2018.
  22. DeBERTa: Decoding-enhanced BERT with disentangled attention. In ICLR, 2021.
  23. Factorising meaning and form for intent-preserving paraphrasing. In ACL-IJCNLP, pages 1405–1418, 2021.
  24. Grammar induction with neural language models: An unusual replication. In EMNLP, pages 4998–5003, 2018.
  25. Attention is not explanation. In NAACL-HLT, pages 3543–3556, 2019.
  26. Categorical reparameterization with Gumbel-softmax. In ICLR, 2017.
  27. Dynamic entity representations in neural language models. In EMNLP, pages 1830–1839, 2017.
  28. Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751, 2014.
  29. Corpus-based induction of syntactic structure: Models of dependency and constituency. In ACL, pages 479–486, 2014.
  30. Attention is not only a weight: Analyzing Transformers with vector norms. In EMNLP, pages 7057–7075, 2020.
  31. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:E253, 2017.
  32. Deep learning. Nature, 521(7553):436–444, 2015.
  33. An imitation learning approach to unsupervised parsing. In ACL, pages 3485–3492, 2019.
  34. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In ACL, pages 23–33, 2017.
  35. Abstract rule learning for paraphrase generation. In IJCAI, pages 4273–4279, 2022.
  36. Jumper: Learning when to make classification decision in reading. In IJCAI, pages 4237–4243, 2018.
  37. Finding decision jumps in text classification. Neurocomputing, 371:177–187, 2020.
  38. Unsupervised paraphrasing by simulated annealing. In ACL, pages 302–312, 2019.
  39. Object-oriented neural programming (OONP) for document understanding. In ACL, pages 2717–2726, 2018.
  40. Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs. arXiv preprint arXiv:1705.09189, 2017.
  41. Gary Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631, 2018.
  42. Rule learning by seven-month-old infants. Science, 283(5398):77–80, 1999.
  43. John McCarthy. Artificial intelligence, logic and formalizing common sense. In Philosophical Logic and Artificial Intelligence, pages 161–190, 1989.
  44. Distributed representations of words and phrases and their compositionality. In NeurIPS, 2013.
  45. Coupling distributed and symbolic execution for natural language queries. In ICML, pages 2518–2526, 2017.
  46. Hybrid simplification using deep semantics and machine translation. In ACL, pages 435–445, 2014.
  47. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2016.
  48. Thumbs up? Sentiment classification using machine learning techniques. In EMNLP, pages 79–86, 2002.
  49. Mathematical Methods in Linguistics. Springer Science & Business Media, 2012.
  50. Neural paraphrase generation with stacked residual LSTM networks. In COLING, pages 2923––2934, 2016.
  51. Monolingual machine translation for paraphrase generation. In EMNLP, pages 142–149, 2004.
  52. Bidirectional attention flow for machine comprehension. In ICLR, 2017.
  53. Neural language modeling by jointly learning syntax and lexicon. In ICLR, 2018.
  54. An expert system for oncology protocol management. Rule-Based Expert Systems, pages 653–65, 1984.
  55. Axiomatic attribution for deep networks. In ICML, pages 3319–3328, 2017.
  56. Reinforcement Learning: An Introduction. MIT Press, 2018.
  57. Attention is all you need. In NeurIPS, pages 5998–6008, 2017.
  58. Multilingual relation extraction using compositional universal schema. In NAACL-HLT, pages 886–896, 2016.
  59. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, pages 90–94, 2012.
  60. A task in a suit and a tie: Paraphrase generation with semantic augmentation. In AAAI, pages 7176–7183, 2019.
  61. Do latent tree learning models identify meaningful structure in sentences? TACL, 6:253–267, 2018.
  62. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.
  63. Weakly supervised explainable phrasal reasoning with neural fuzzy logic. In ICLR, 2023.
  64. Differentiable learning of logical rules for knowledge base reasoning. In NeurIPS, 2017.
  65. Neural enquirer: Learning to query tables with natural language. In IJCAI, pages 2308–2314, 2016.
  66. Learning to compose words into sentences with reinforcement learning. In ICLR, 2017.
  67. Lotfi A Zadeh. Fuzzy logic. Computer, 21(4):83–93, 1988.
  68. Kernel methods for relation extraction. Journal of Machine Learning Research, 3:1083–1106, 2003.
  69. Relation classification via convolutional deep neural network. In COLING, pages 2335–2344, 2014.
  70. Learning robust rule representations for abstract reasoning via internal inferences. In NeurIPS, 2022.
  71. Character-level convolutional networks for text classification. In NeurIPS, pages 649–657, 2015.
  72. Application-driven statistical paraphrase generation. In ACL, pages 834–842, 2009.
  73. Attention-based LSTM network for cross-lingual sentiment classification. In EMNLP, pages 247–256, 2016.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now