Papers
Topics
Authors
Recent
Search
2000 character limit reached

Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives

Published 20 Oct 2023 in cs.CL | (2310.13676v1)

Abstract: We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (96)
  1. Prediction at the intersection of sentence context and word form: Evidence from eye-movements and self-paced reading. Psychonomic Bulletin & Review.
  2. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 301–313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Probabilistic language models in cognitive neuroscience: Promises and pitfalls. Neuroscience & Biobehavioral Reviews, 83:579–588.
  4. John Langshaw Austin. 1975. How to do things with words. Clarendon Press, Oxford.
  5. David I Beaver and Brady Z Clark. 2009. Sense and sensitivity: How focus determines meaning. John Wiley & Sons.
  6. The influence of context on sentence acceptability judgements. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461, Melbourne, Australia. Association for Computational Linguistics.
  7. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg.
  8. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  9. Conceptual alternatives: Competition in language and beyond. Linguistics and Philosophy, 45(2):265–291.
  10. Robyn Carston. 1998. Informativeness, relevance and scalar implicature. Pragmatics And Beyond New Series, pages 179–238.
  11. The effect of context on noisy-channel sentence comprehension. Cognition, 238:105503.
  12. Inquisitive semantics. Oxford University Press.
  13. Forrest Davis and Marten van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 396–407.
  14. Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non-native reading. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 138–144. Association for Computational Linguistics.
  15. Judith Degen and Michael K Tanenhaus. 2015. Processing scalar implicature: A constraint-based approach. Cognitive science, 39(4):667–710.
  16. Judith Degen and Michael K Tanenhaus. 2016. Availability of alternatives and the processing of scalar implicatures: A visual world eye-tracking study. Cognitive science, 40(1):172–201.
  17. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210.
  18. Gabriel Doyle and Michael Frank. 2015. Shared common ground influences information density in microblog texts. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1587–1596, Denver, Colorado. Association for Computational Linguistics.
  19. Bryan Eikema and Wilker Aziz. 2022. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10978–10993, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  20. Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: Computer applications, pages 231–243. Springer.
  21. Danny Fox and Roni Katzir. 2011. On the characterization of alternatives. Natural language semantics, 19:87–107.
  22. The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140:1–11.
  23. Richard Futrell and Roger Levy. 2017. Noisy-context surprisal as a human sentence processing cost model. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, long papers, pages 688–698.
  24. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  25. Gerald Gazdar. 1979. Pragmatics, implicature, presuposition and logical form. Critica, 12(35).
  26. Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 199–206.
  27. Edward Gibson and James Thomas. 1999. Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical. Language and Cognitive Processes, 14:225–248.
  28. Mario Giulianelli. 2022. Towards pragmatic production strategies for natural language generation tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7978–7984, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  29. What comes next? Evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  30. Mario Giulianelli and Raquel Fernández. 2021. Analysing human strategies of information transmission as a function of discourse context. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 647–660, Online. Association for Computational Linguistics.
  31. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8271–8283, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  32. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE international conference on Acoustics, speech and signal processing-Volume 1, pages 517–520.
  33. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Computational Linguistics.
  34. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill.
  35. Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. 1984. Studies on the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, University of Amsterdam.
  36. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1059–1075, Dubrovnik, Croatia. Association for Computational Linguistics.
  37. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second meeting of the North American Chapter of the Association for Computational Linguistics.
  38. Charles L Hamblin. 1976. Questions in montague English. In Montague grammar, pages 247–259. Elsevier.
  39. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 174–180, Baltimore, Maryland. Association for Computational Linguistics.
  40. Processing effects in linguistic judgment data: (Super-)additivity and reading span scores. Language and Cognition, 6:111 – 145.
  41. The curious case of neural text degeneration. In International Conference on Learning Representations.
  42. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  43. spaCy: Industrial-strength natural language processing in Python.
  44. Laurence Robert Horn. 1972. On the semantic properties of logical operators in English. University of California, Los Angeles.
  45. Expectations over unspoken alternatives predict pragmatic inferences. Transactions of the Association for Computational Linguistics. To appear.
  46. Predicting scalar diversity with context-driven uncertainty over alternatives. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 68–74, Dublin, Ireland. Association for Computational Linguistics.
  47. T. Florian Jaeger and Roger P. Levy. 2007. Speakers optimize information density through syntactic reduction. In Advances in neural information processing systems, pages 849–856.
  48. Roni Katzir. 2007. Structurally-defined alternatives. Linguistics and philosophy, 30:669–690.
  49. Frank Keller. 2004. The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 317–324.
  50. Dave F Kleinschmidt and T Florian Jaeger. 2015. Robust speech perception: recognize the familiar, generalize to the similar, and adapt to the novel. Psychological review, 122(2):148.
  51. Daphne Koller and Nir Friedman. 2009. Probabilistic graphical models: Principles and techniques. MIT press.
  52. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations.
  53. Unsupervised prediction of acceptability judgements. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1618–1628.
  54. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241.
  55. Natural language grammatical inference with recurrent neural networks. IEEE Transactions on Knowledge & Data Engineering, 12(01):126–140.
  56. Stephen C. Levinson. 2000. Presumptive Meanings: The Theory of Generalized Conversational Implicature. The MIT Press.
  57. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177.
  58. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. MIT Press.
  59. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995.
  60. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826–833.
  61. Scott A. McDonald and Richard C. Shillcock. 2003. Low-level predictive inference in reading: the influence of transitional probabilities on eye movements. Vision Research, 43(16):1735–1751.
  62. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963–980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  63. Locally Typical Sampling. Transactions of the Association for Computational Linguistics, 11:102–121.
  64. Byung-Doh Oh and William Schuler. 2023. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics, 11:336–350.
  65. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
  66. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  67. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  68. Mats Rooth. 1992. A theory of focus interpretation. Natural language semantics, pages 75–116.
  69. Mats Rooth. 1996. Focus. The handbook of contemporary semantic theory, ed. by Shalom Lappin.
  70. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45).
  71. John R Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge University Press.
  72. John R Searle. 1975. A taxonomy of illocutionary acts. Language Mind, and Knowledge, 7.
  73. Large-scale evidence for logarithmic effects of word predictability on reading time.
  74. Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423.
  75. Structural persistence in language models: Priming as a window into abstract language representations. Transactions of the Association for Computational Linguistics, 10:1031–1050.
  76. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319.
  77. Harold Somers. 2005. Round-trip translation: What is it good for? In Proceedings of the Australasian Language Technology Workshop 2005, pages 127–133, Sydney, Australia.
  78. Antonella Sorace and Frank Keller. 2005. Gradience in linguistic data. Lingua, 115(11):1497–1524.
  79. Robert C Stalnaker. 1978. Assertion. In Pragmatics, pages 315–332. Brill.
  80. Katherine Stasaski and Marti A Hearst. 2023. Pragmatically appropriate diversity for dialogue evaluation. arXiv preprint arXiv:2304.02812.
  81. Processing advantages of lexical bundles: Evidence from self-paced reading and sentence recall tasks. Language learning, 61(2):569–613.
  82. The eyes have it: An eye movement study into the processing of formulaic sequences. In Norbert Schmitt, editor, Formulaic Sequences: Acquisition, Processing and Use, pages 153–172. John Benjamins.
  83. Scalar Diversity. Journal of Semantics, 33(1):137–175.
  84. Revisiting entropy rate constancy in text. arXiv preprint arXiv:2305.12084.
  85. Michael Wagner et al. 2005. Prosody and recursion. Ph.D. thesis, Massachusetts Institute of Technology.
  86. Investigating perception of spoken dialogue acceptability through surprisal. In Interspeech 2022: The 23rd Annual Conference of the International Speech Communication Association, pages 4506–4510. International Speech Communication Association.
  87. Do dialogue representations align with perception? an empirical study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2696–2713, Dubrovnik, Croatia. Association for Computational Linguistics.
  88. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641.
  89. A cognitive regularizer for language modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 5191–5202. Association for Computational Linguistics.
  90. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707–1713. Cognitive Science Society.
  91. Prediction during natural language comprehension. Cerebral Cortex, 26(6):2506–2516.
  92. Yang Xu and David Reitter. 2018. Information density converges in dialogue: Towards an information-theoretic model. Cognition, 170:147–163.
  93. Shaorong Yan and T Florian Jaeger. 2020. Expectation adaptation during natural reading. Language, Cognition and Neuroscience, 35(10):1394–1422.
  94. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  95. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270–278.
  96. Scalar implicature is sensitive to contextual alternatives. Cognitive science, 47(2):e13238.
Citations (9)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up