Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement

Published 5 Apr 2024 in cs.CL | (2404.04035v1)

Abstract: We present a novel dataset for physical and abstract plausibility of events in English. Based on naturally occurring sentences extracted from Wikipedia, we infiltrate degrees of abstractness, and automatically generate perturbed pseudo-implausible events. We annotate a filtered and balanced subset for plausibility using crowd-sourcing, and perform extensive cleansing to ensure annotation quality. In-depth quantitative analyses indicate that annotators favor plausibility over implausibility and disagree more on implausible events. Furthermore, our plausibility dataset is the first to capture abstractness in events to the same extent as concreteness, and we find that event abstractness has an impact on plausibility ratings: more concrete event participants trigger a perception of implausibility.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Dina Almanea and Massimo Poesio. 2022. ArMIS - the Arabic misogyny and sexism corpus with annotator subjective disagreements. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2282–2291, Marseille, France. European Language Resources Association.
  2. Clarifying Implicit and Underspecified Phrases in Instructional Text. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3319–3330, Marseille, France. European Language Resources Association.
  3. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24.
  4. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  5. Lawrence W Barsalou and Katja Wiemer-Hastings. 2005. Situating Abstract Concepts. Grounding cognition: The role of perception and action in memory, language, and thought, pages 129–163.
  6. Toward a perspectivist turn in ground truthing for predictive computing.
  7. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics.
  8. A domain-specific dataset of difficulty ratings for German noun compounds in the domains DIY, cooking and automotive. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4359–4367, Marseille, France. European Language Resources Association.
  9. Adding part-of-speech information to the SUBTLEX-US word frequencies. Behavior research methods, 44(4):991–997.
  10. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior research methods, 46(3):904–911.
  11. Jean Charbonnier and Christian Wartena. 2019. Predicting word concreteness and imagery. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 176–187, Gothenburg, Sweden. Association for Computational Linguistics.
  12. Towards a distributional model of semantic complexity. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 12–22, Osaka, Japan. The COLING 2016 Organizing Committee.
  13. Guy Dove. 2009. Beyond perceptual symbols: A call for representational pluralism. Cognition, 110(3):412–431.
  14. ADEPT: An adjective-dependent plausibility task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7117–7128, Online. Association for Computational Linguistics.
  15. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723–763.
  16. Agnieszka Falenska and Özlem Çetinoğlu. 2021. Assessing gender bias in Wikipedia: Inequalities in article titles. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 75–85, Online. Association for Computational Linguistics.
  17. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591–2597, Online. Association for Computational Linguistics.
  18. Diego Frassinelli and Sabine Schulte im Walde. 2019. Distributional interaction of concreteness and abstractness in verb–noun subcategorisation. In Proceedings of the 13th International Conference on Computational Semantics - Short Papers, pages 38–43, Gothenburg, Sweden. Association for Computational Linguistics.
  19. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 42–47, Portland, Oregon, USA. Association for Computational Linguistics.
  20. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, page 25–30, New York, NY, USA. Association for Computing Machinery.
  21. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
  22. Experiments with crowdsourced re-annotation of a POS tagging data set. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 377–382, Baltimore, Maryland. Association for Computational Linguistics.
  23. Paul Jaccard. 1902. Lois de distribution florale dans la zone alpine. Bulletin de la Société vaudoise des sciences naturelles, 38:69–130.
  24. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297, Lisbon, Portugal. Association for Computational Linguistics.
  25. Sentiment analysis: It’s complicated! In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1886–1895, New Orleans, Louisiana. Association for Computational Linguistics.
  26. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  27. How relevant are selectional preferences for transformer-based language models? In Proceedings of the 28th International Conference on Computational Linguistics, pages 1266–1278, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  28. Quantitative semantic variation in the contexts of concrete and abstract words. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 76–85, New Orleans, Louisiana. Association for Computational Linguistics.
  29. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data, 2.
  30. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694.
  31. Did the cat drink the coffee? challenging transformers with generalized event knowledge. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 1–11, Online. Association for Computational Linguistics.
  32. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics.
  33. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778–1789, Minneapolis, Minnesota. Association for Computational Linguistics.
  34. Can a gorilla ride a camel? learning semantic plausibility from text. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 123–129, Hong Kong, China. Association for Computational Linguistics.
  35. Modeling event plausibility with consistent conceptual abstraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1732–1743, Online. Association for Computational Linguistics.
  36. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics.
  37. The possible, the plausible, and the desirable: Event-based modality detection for language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 953–965, Online. Association for Computational Linguistics.
  38. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
  39. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta. ELRA. http://is.muni.cz/publication/884893/en.
  40. Philip Stuart Resnik. 1993. Selection and information: A class-based approach to lexical relationships. Ph.D. thesis, University of Pennsylvania.
  41. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2409–2422.
  42. The (un)expected effects of applying standard cleansing models to human ratings on compositionality. In Proceedings of the 9th Workshop on Multiword Expressions, pages 32–41, Atlanta, Georgia, USA. Association for Computational Linguistics.
  43. Thematic fit evaluation: an aspect of selectional preferences. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 99–105, Berlin, Germany. Association for Computational Linguistics.
  44. Roland Schäfer. 2015. Processing and querying large web corpora with the COW14 architecture. Proceedings of the 3rd Workshop on Challenges in the Management of Large Corpora, Lancaster, 20 July 2015, pages 28 – 34, Mannheim.
  45. Is Wikipedia succeeding in reducing gender bias? assessing changes in gender bias in Wikipedia using word embeddings. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 94–103, Online. Association for Computational Linguistics.
  46. Sabine Schulte im Walde and Diego Frassinelli. 2022. Distributional Measures of Abstraction. Frontiers in Artificial Intelligence: Language and Computation 4:796756. Alessandro Lenci and Sebastian Pado (topic editors): "Perspectives for Natural Language Processing between AI, Linguistics and Cognitive Science".
  47. Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  48. Jiao Sun and Nanyun Peng. 2021. Men are elected, women are married: Events gender bias on Wikipedia. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 350–360, Online. Association for Computational Linguistics.
  49. Concreteness vs. abstractness: A selectional preference perspective. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 92–98, Online. Association for Computational Linguistics.
  50. Scaling and Disagreements: Bias, Noise, and Ambiguity. Frontiers in Artificial Intelligence, 5.
  51. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338–347, Online. Association for Computational Linguistics.
  52. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470.
  53. Context effects in embodied lexical-semantic processing. Frontiers in Psychology, 1:150.
  54. Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 26–35, Doha, Qatar. Association for Computational Linguistics.
  55. Modeling semantic plausibility by injecting world knowledge. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 303–308, New Orleans, Louisiana. Association for Computational Linguistics.
  56. Yorick Wilks. 1975. A preferential, pattern-seeking, semantics for natural language inference. Artificial Intelligence, 6(1):53–74.
  57. Unifying local and global agreement and disagreement classification in online debates. In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pages 61–69, Jeju, Korea. Association for Computational Linguistics.
  58. SP-10K: A large-scale evaluation set for selectional preference acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 722–731, Florence, Italy. Association for Computational Linguistics.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up