Logical Negation Augmenting and Debiasing for Prompt-based Methods (2405.04872v1)
Abstract: Prompt-based methods have gained increasing attention on NLP and shown validity on many downstream tasks. Many works have focused on mining these methods' potential for knowledge extraction, but few explore their ability to make logical reasoning. In this work, we focus on the effectiveness of the prompt-based methods on first-order logical reasoning and find that the bottleneck lies in logical negation. Based on our analysis, logical negation tends to result in spurious correlations to negative answers, while propositions without logical negation correlate to positive answers. To solve the problem, we propose a simple but effective method, Negation Augmenting and Negation Debiasing (NAND), which introduces negative propositions to prompt-based methods without updating parameters. Specifically, these negative propositions can counteract spurious correlations by providing "not" for all instances so that models cannot make decisions only by whether expressions contain a logical negation. Experiments on three datasets show that NAND not only solves the problem of calibrating logical negation but also significantly enhances prompt-based methods of logical reasoning without model retraining.
- Language models are few-shot learners. In NeurIPS.
- Implicit premise generation with discourse-aware commonsense knowledge models. In EMNLP.
- Transformers as soft reasoners over language. In IJCAI.
- Template-based named entity recognition using BART. In ACL/IJCNLP.
- Ernest Davis. 2017. Logical formalizations of commonsense reasoning: A survey. J. Artif. Intell. Res.
- BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
- Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Linguistics.
- Making pre-trained language models better few-shot learners. In ACL/IJCNLP.
- M. Gelfond and V. Lifschitz. 1991. Classical negation in logic pro-grams and deductive databases. New Generation Computing.
- Stack: Sentence ordering with temporal commonsense knowledge. In EMNLP.
- Annotation artifacts in natural language inference data. In NAACL-HLT.
- PTR: prompt tuning with rules for text classification. CoRR.
- Laurence R. Horn and Heinrich Wansing. 2020. Negation. In The Stanford Encyclopedia of Philosophy.
- Predicting the focus of negation: Model and error analysis. In ACL.
- Understanding by understanding not: Modeling negation in language models. In NAACL-HLT.
- What does BERT learn about the structure of language? In ACL.
- How can we know what language models know. Trans. Assoc. Comput. Linguistics.
- Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In ACL.
- Are prompt-based models clueless? In ACL.
- Aditya Khandelwal and Suraj Sawant. 2020. Negbert: A transfer learning approach for negation detection and scope resolution. In LREC.
- Generated knowledge prompting for commonsense reasoning. In ACL.
- Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR.
- Roberta: A robustly optimized BERT pretraining approach. CoRR.
- Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL.
- Language models as knowledge bases? In EMNLP-IJCNLP.
- Raymond Reiter. 1981. On closed world data bases. In Readings in Artificial Intelligence.
- Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In CHI.
- Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In AAAI.
- Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP.
- Proofwriter: Generating implications, proofs, and abductive statements over natural language. In ACL/IJCNLP.
- olmpics - on what language model pre-training captures. Trans. Assoc. Comput. Linguistics.
- Diagnosing the first-order logical reasoning ability through logicnli. In EMNLP.
- Learning disentangled representations of negation and uncertainty. In ACL.
- Universal adversarial triggers for attacking and analyzing NLP. In EMNLP-IJCNLP.
- Can generative pre-trained language models serve as knowledge bases for closed-book qa? In ACL/IJCNLP.
- How does BERT capture semantics? A closer look at polysemous words. In BlackboxNLP@EMNLP.
- Reclor: A reading comprehension dataset requiring logical reasoning. In ICLR.
- Differentiable prompt makes pre-trained language models better few-shot learners. CoRR.
- Calibrate before use: Improving few-shot performance of language models. In ICML.