Papers
Topics
Authors
Recent
2000 character limit reached

Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity

Published 29 May 2023 in cs.CL | (2305.18185v2)

Abstract: Recent advances in LLMs have prompted researchers to examine their abilities across a variety of linguistic tasks, but little has been done to investigate how models handle the interactions in meaning across words and larger syntactic forms -- i.e. phenomena at the intersection of syntax and semantics. We present the semantic notion of agentivity as a case study for probing such interactions. We created a novel evaluation dataset by utilitizing the unique linguistic properties of a subset of optionally transitive English verbs. This dataset was used to prompt varying sizes of three model classes to see if they are sensitive to agentivity at the lexical level, and if they can appropriately employ these word-level priors given a specific syntactic context. Overall, GPT-3 text-davinci-003 performs extremely well across all experiments, outperforming all other models tested by far. In fact, the results are even better correlated with human judgements than both syntactic and semantic corpus statistics. This suggests that LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery than select corpora for certain tasks. Code is available at https://github.com/lindiatjuatja/lm_sem

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT press.
  3. Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology. University of Chicago press.
  4. Cleo Condoravdi. 1989. The middle: Where semantics and morphology meet. In MIT Working Papers in Linguistics 11, pages 16–31. MIT Press.
  5. William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective. Oxford University Press on Demand.
  6. David Dowty. 1991. Thematic proto-roles and argument selection. Language, 67(3):547–619.
  7. Thomas Ernst. 2001. The syntax of adjuncts, volume 96. Cambridge University Press.
  8. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48.
  9. Sarah M. B. Fagan. 1992. The Syntax and Semantics of Middle Constructions. Cambridge University Press, Cambridge.
  10. Charles J Fillmore. 1986. Pragmatically controlled zero anaphora. In Annual Meeting of the Berkeley Linguistics Society, volume 12, pages 95–107.
  11. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  12. Brendan S Gillon. 2012. Implicit complements: a dilemma for model theoretic semantics. Linguistics and Philosophy, 35:313–359.
  13. Adele E.. Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press.
  14. Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-ngrams over time from a very large corpus of English books. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 241–247, Atlanta, Georgia, USA. Association for Computational Linguistics.
  15. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  16. Ray S Jackendoff. 1972. Semantic interpretation in generative grammar.
  17. Osvaldo A. Jaeggli. 1986. Passive. Linguistic Inquiry, 17:587–622.
  18. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  19. Jerrold J Katz and Jerry A Fodor. 1963. The structure of a semantic theory. Language, 39(2):170–210.
  20. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics.
  21. Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In LREC, pages 1989–1993.
  22. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago press.
  23. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computational Linguistics.
  24. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195–212.
  25. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  26. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. CoRR, abs/2104.08786.
  27. Kyle Mahowald. 2023. A discerning several thousand judgments: GPT-3 rates the Article + Adjective + Numeral + Noun construction. arXiv preprint arXiv:2301.12564.
  28. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38(3):283–312.
  29. COMPS: Conceptual minimal pair sentences for testing property knowledge and inheritance in pre-trained language models. arXiv preprint arXiv:2210.01963.
  30. Ulrike Padó. 2007. The integration of syntax and semantic plausibility in a wide-coverage model of human sentence processing.
  31. When classifying arguments, BERT doesn’t care about word order…except when it matters. In Proceedings of the Society for Computation in Linguistics 2022, pages 203–205, online. Association for Computational Linguistics.
  32. Erika Petersen and Christopher Potts. 2022. Lexical semantics with large language models: A case study of English break. Ms., Stanford University.
  33. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  34. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194–209, Online. Association for Computational Linguistics.
  35. Philip Resnik. 1996. Selectional constraints: An information-theoretic model and its computational realization. Cognition, 61(1-2):127–159.
  36. Lilia Rissman and Asifa Majid. 2019. Thematic roles: Core knowledge or linguistic construct? Psychonomic bulletin & review, 26(6):1850–1869.
  37. David E Rumelhart and James L McClelland. 1986. On learning the past tenses of English verbs.
  38. Thematic fit evaluation: an aspect of selectional preferences. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 99–105, Berlin, Germany. Association for Computational Linguistics.
  39. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
  40. Michael Silverstein. 1976. Shifters, linguistic categories, and cultural description. Meaning in anthropology.
  41. The role of thematic structures in interpretation and parsing. Language and Cognitive Processes, 4(3-4):SI211–SI234.
  42. Carol Tenny. 1992. The aspectual interface hypothesis. pages 490–508. CSLI Publications, Stanford.
  43. Carol Tenny. 1994. Aspectual Roles and the Syntax-Semantic Interface. Kluwer, Dordrecht.
  44. Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33(3):285–318.
  45. Jeanne van Oosten. 1977. Subjects and agenthood in English. In CLS 13, pages 451–471.
  46. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational Linguistics.
  47. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.
  48. The better your syntax, the better your semantics? Probing pretrained language models for the English comparative correlative. arXiv preprint arXiv:2210.13181.
  49. What do RNN language models learn about filler-gap dependencies? arXiv preprint arXiv:1809.00042.
  50. Animacy encoding in English: Why and how. In Proceedings of the workshop on discourse annotation, pages 118–125.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up