2000 character limit reached
CASA: Causality-driven Argument Sufficiency Assessment (2401.05249v2)
Published 10 Jan 2024 in cs.CL
Abstract: The argument sufficiency assessment task aims to determine if the premises of a given argument support its conclusion. To tackle this task, existing works often train a classifier on data annotated by humans. However, annotating data is laborious, and annotations are often inconsistent due to subjective criteria. Motivated by the definition of probability of sufficiency (PS) in the causal literature, we proposeCASA, a zero-shot causality-driven argument sufficiency assessment framework. PS measures how likely introducing the premise event would lead to the conclusion when both the premise and conclusion events are absent. To estimate this probability, we propose to use LLMs to generate contexts that are inconsistent with the premise and conclusion and revise them by injecting the premise event. Experiments on two logical fallacy detection datasets demonstrate that CASA accurately identifies insufficient arguments. We further deploy CASA in a writing assistance application, and find that suggestions generated by CASA enhance the sufficiency of student-written arguments. Code and data are available at https://github.com/xxxiaol/CASA.
- Unit segmentation of argumentative texts. In Proceedings of the 4th Workshop on Argument Mining, pages 118–128, Copenhagen, Denmark. Association for Computational Linguistics.
- Multitask instruction-based prompting for fallacy recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8172–8187, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- This is not correct! negation-aware evaluation of language generation systems. arXiv preprint arXiv:2307.13989.
- Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12317–12325.
- J Anthony Blair. 2011. Groundwork in the theory of argumentation: Selected papers of J. Anthony Blair, volume 21. Springer Science & Business Media.
- Robert J Fogelin and Walter Sinnott-Armstrong. 2005. Understanding arguments. An introduction to informal logic.
- Assessing the sufficiency of arguments through conclusion generation. In Proceedings of the 8th Workshop on Argument Mining, pages 67–77, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Exploring methods for generating feedback comments for writing learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9719–9730, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Guido W Imbens and Donald B Rubin. 2015. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
- Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
- Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.
- OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
- Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19(2):3.
- Evaluation of argument search approaches in the context of argumentative dialogue systems. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 513–522, Marseille, France. European Language Resources Association.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
- Donald B Rubin. 1978. Bayesian inference for causal effects: The role of randomization. The Annals of statistics, pages 34–58.
- Graph-based argument quality assessment. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1268–1280, Held Online. INCOMA Ltd.
- Gabriella Skitalinskaya and Henning Wachsmuth. 2023. To revise or not to revise: Learning to detect improvable claims for argumentative writing support. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15799–15816, Toronto, Canada. Association for Computational Linguistics.
- Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
- Christian Stab and Iryna Gurevych. 2017a. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619–659.
- Christian Stab and Iryna Gurevych. 2017b. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619–659.
- Christian Stab and Iryna Gurevych. 2017c. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 980–990, Valencia, Spain. Association for Computational Linguistics.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 176–187, Valencia, Spain. Association for Computational Linguistics.
- Henning Wachsmuth and Till Werner. 2020. Intrinsic quality assessment of arguments. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6739–6745, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Thiemo Wambsganss and Christina Niklaus. 2022. Modeling persuasive discourse to adaptively support students’ argumentative writing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8748–8760, Dublin, Ireland. Association for Computational Linguistics.
- Smart word suggestions for writing assistance. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11212–11225, Toronto, Canada. Association for Computational Linguistics.
- How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751.
- Rock: Causal inference principles for reasoning about commonsense causality. In International Conference on Machine Learning, pages 26750–26771. PMLR.
- Automatic comment generation for Chinese student narrative essays. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 214–223, Abu Dhabi, UAE. Association for Computational Linguistics.
Collections
Sign up for free to add this paper to one or more collections.