Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning (2405.01139v3)

Published 2 May 2024 in cs.CL

Abstract: Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients. Overhearers are deprived of the privilege of performing grounding acts and can only conjecture about intended meanings. Still, data generation and annotation, modelling, training and evaluation of NLP dialogue models place reliance on the overhearing paradigm. How much of the underlying grounding processes are thereby forfeited? As we show, there is evidence pointing to the impossibility of properly modelling human meta-communicative acts with data-driven learning models. In this paper, we discuss this issue and provide a preliminary analysis on the variability of human decisions for requesting clarification. Most importantly, we wish to bring this topic back to the community's table, encouraging discussion on the consequences of having models designed to only "listen in".

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Angus Addlesee and Arash Eshghi. 2024. You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarification. Frontiers in Dementia, 3:1343052.
  2. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473–4484, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  3. Madeleine Bates and Damaris Ayuso. 1991. A proposal for incremental dialogue evaluation. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991.
  4. Luciana Benotti and Patrick Blackburn. 2021a. Grounding as a collaborative process. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 515–531, Online. Association for Computational Linguistics.
  5. Luciana Benotti and Patrick Blackburn. 2021b. A recipe for annotating grounded clarifications. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4065–4077, Online. Association for Computational Linguistics.
  6. Backchannel behavior is idiosyncratic. Language and Cognition, page 1–24.
  7. Susan E. Brennan. 2000. Invited talk: Processes that shape conversation and their implications for computational linguistics. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 1–11, Hong Kong. Association for Computational Linguistics.
  8. Two minds, one dialog: Coordinating speaking and understanding. In Psychology of learning and motivation, volume 53, pages 301–344. Elsevier.
  9. Collaborating on utterances with a spoken dialogue system using an ISU-based approach to incremental dialogue management. In Proceedings of the SIGDIAL 2010 Conference, pages 233–236, Tokyo, Japan. Association for Computational Linguistics.
  10. Chat-crowd: A dialog-based platform for visual layout composition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 138–142, Minneapolis, Minnesota. Association for Computational Linguistics.
  11. Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 361–366, Stockholm, Sweden. Association for Computational Linguistics.
  12. Herbert H Clark. 1992. Arenas of language use. University of Chicago Press.
  13. Herbert H. Clark. 1996. Common ground, page 92–122. “Using” Linguistic Books. Cambridge University Press.
  14. Herbert H Clark and Susan E Brennan. 1991. Grounding in communication. In Perspectives on socially shared cognition., pages 127–149. American Psychological Association.
  15. A strategy for information presentation in spoken dialog systems. Computational Linguistics, 37(3):489–539.
  16. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10602–10621, Singapore. Association for Computational Linguistics.
  17. Quantifying the interplay of conversational devices in building mutual understanding. Journal of Experimental Psychology: General, 152(3):864.
  18. Interactive repair and the foundations of language. Trends in Cognitive Sciences.
  19. Beyond single-mindedness: A figure-ground reversal for the cognitive sciences. Cognitive science, 47(1):e13230.
  20. Arash Eshghi and Patrick G.T. Healey. 2007. Collective states of understanding. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, pages 2–9, Antwerp, Belgium. Association for Computational Linguistics.
  21. Sarah E. Finch and Jinho D. Choi. 2020. Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236–245, 1st virtual meeting. Association for Computational Linguistics.
  22. Jean E Fox Tree. 1999. Listening in on monologues and dialogues. Discourse processes, 27(1):35–53.
  23. Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions. In the 39th Annual Conference of the Cognitive Science Society (CogSci 2017), pages 2055–2060. Cognitive Science Society.
  24. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4):10049–10056.
  25. Predicting ratings of real dialogue participants from artificial data and ratings of human dialogue observers. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 726–734, Marseille, France. European Language Resources Association.
  26. User simulation for spoken dialogue systems: learning and evaluation. In Ninth International Conference on Spoken Language Processing.
  27. Jonathan Ginzburg. 2012. The interactive stance: Meaning for conversation. Oxford University Press.
  28. Phil Hayes. 1980. Expanding the horizons of natural language interfaces. In 18th Annual Meeting of the Association for Computational Linguistics, pages 71–74, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  29. Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets. Computational Linguistics, 34(4):487–511.
  30. Lixing Huang and Jonathan Gratch. 2012. Crowdsourcing backchannel feedback: understanding the individual variability from the crowds. In Feedback behaviors in dialog.
  31. CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  32. Spyridon Kousidis and David Schlangen. 2015. The power of a glance: Evaluating embodiment and turn-tracking strategies of an active robotic overhearer. In Proceedings of AAAI Spring Symposium on Turn-taking and Coordination in Human-Machine Interaction.
  33. Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769.
  34. Python code generation by asking clarification questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14287–14306, Toronto, Canada. Association for Computational Linguistics.
  35. Rethinking supervised learning and reinforcement learning in task-oriented dialogue systems. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3537–3546, Online. Association for Computational Linguistics.
  36. Dialogue act-aided backchannel prediction using multi-task learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15073–15079, Singapore. Association for Computational Linguistics.
  37. Coordinating communication in the wild: The artwalk dialogue corpus of pedestrian navigation and mobile referential communication. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3159–3166, Portorož, Slovenia. European Language Resources Association (ELRA).
  38. Brielen Madureira and David Schlangen. 2023. Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the CoDraw dataset. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2303–2319, Dubrovnik, Croatia. Association for Computational Linguistics.
  39. Brielen Madureira and David Schlangen. 2024. Taking action towards graceful interaction: The effects of performing actions on modelling policies for instruction clarification requests. In Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language, pages 1–21, Malta. Association for Computational Linguistics.
  40. Finding common ground: Annotating and predicting common ground in spoken conversations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8221–8233, Singapore. Association for Computational Linguistics.
  41. Transforming human-centered ai collaboration: Redefining embodied agents capabilities through interactive grounded language instructions. arXiv preprint arXiv:2305.10783.
  42. Conversational grounding: Annotation and analysis of grounding acts and grounding units. In Proceedings of LREC-COLING 2024.
  43. Johanna D. Moore. 2011. Language generation for spoken dialogue systems [invited talk]. In Proceedings of the 13th European Workshop on Natural Language Generation, page 132, Nancy, France. Association for Computational Linguistics.
  44. John Niekrasz and Johanna D. Moore. 2010. Annotating participant reference in English spoken conversation. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 256–264, Uppsala, Sweden. Association for Computational Linguistics.
  45. Evaluating the effectiveness of information presentation in a full end-to-end dialogue system. In Proceedings of the SIGDIAL 2009 Conference, pages 1–10, London, UK. Association for Computational Linguistics.
  46. Stanley Peters. 2010. Listening in. In Proceedings of the 24th Pacific Asia Conference on Language, Information and Computation, pages 31–31, Tohoku University, Sendai, Japan. Institute of Digital Enhancement of Cognitive Processing, Waseda University.
  47. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  48. Verena Rieser and Oliver Lemon. 2011. Reinforcement learning for adaptive dialogue systems: a data-driven methodology for dialogue management and natural language generation. Springer Science & Business Media.
  49. Kepa Joseba Rodríguez and David Schlangen. 2004. Form, intonation and function of clarification requests in german task-oriented spoken dialogues. In Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04).
  50. Antonio Roque and David Traum. 2008. Degrees of grounding based on evidence of understanding. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 54–63, Columbus, Ohio. Association for Computational Linguistics.
  51. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45–54, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial).
  52. David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 136–143, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  53. Julian J. Schlöder and Raquel Fernández. 2015. Clarifying intentions in dialogue: A corpus study. In Proceedings of the 11th International Conference on Computational Semantics, pages 46–51, London, UK. Association for Computational Linguistics.
  54. Michael F Schober and Herbert H Clark. 1989. Understanding by addressees and overhearers. Cognitive psychology, 21(2):211–232.
  55. Grounding or guesswork? large language models are presumptive grounders. arXiv preprint arXiv:2311.09144.
  56. Learning to execute actions or ask clarification questions. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2060–2070, Seattle, United States. Association for Computational Linguistics.
  57. Towards an extrinsic evaluation of referring expressions in situated dialogs. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics.
  58. Modelling human clarification strategies. In Proceedings of the SIGDIAL 2013 Conference, pages 137–141, Metz, France. Association for Computational Linguistics.
  59. Alberto Testoni and Raquel Fernández. 2024. Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 258–275, St. Julian’s, Malta. Association for Computational Linguistics.
  60. Jackson Tolins and Jean E Fox Tree. 2016. Overhearers use addressee backchannels in dialog comprehension. Cognitive science, 40(6):1412–1434.
  61. Learn what is possible, then choose what is best: Disentangling one-to-many relations in language through text-based games. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4955–4965, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  62. David Traum. 2017. Computational approaches to dialogue. The Routledge Handbook of Language and Dialogue, 1:143–161.
  63. Jean E Fox Tree and Sarah A Mayer. 2008. Overhearing single and multiple perspectives. Discourse Processes, 45(2):160–179.
  64. Generating contrastive referring expressions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 678–687, Vancouver, Canada. Association for Computational Linguistics.
  65. Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science, 28(5):811–840.
  66. Emergent conversational recommendations: A dialogue behavior approach. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, pages 63–66, Antwerp, Belgium. Association for Computational Linguistics.
  67. Steve Whittaker and Marilyn Walker. 2005. Evaluating dialogue strategies in multimodal dialogue systems. Spoken Multimodal Human-Computer Dialogue in Mobile Environments, pages 247–268.
  68. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.
  69. A practical guide to conversation research: How to study what people say to each other. Advances in Methods and Practices in Psychological Science, 6(4):25152459231183919.
  70. GrounDialog: A dataset for repair and grounding in task-oriented spoken dialogues for language learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 300–314, Toronto, Canada. Association for Computational Linguistics.
  71. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654–664, Vancouver, Canada. Association for Computational Linguistics.
  72. Reflect, not reflex: Inference-based common ground improves dialogue response quality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10450–10468, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Brielen Madureira (14 papers)
  2. David Schlangen (51 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets