Papers
Topics
Authors
Recent
Search
2000 character limit reached

Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and Perspectives

Published 17 Dec 2023 in cs.HC | (2312.10650v2)

Abstract: LLMs have created opportunities for designing chatbots that can support complex question-answering (QA) scenarios and improve news audience engagement. However, we still lack an understanding of what roles journalists and readers deem fit for such a chatbot in newsrooms. To address this gap, we first interviewed six journalists to understand how they answer questions from readers currently and how they want to use a QA chatbot for this purpose. To understand how readers want to interact with a QA chatbot, we then conducted an online experiment (N=124) where we asked each participant to read three news articles and ask questions to either the author(s) of the articles or a chatbot. By combining results from the studies, we present alignments and discrepancies between how journalists and readers want to use QA chatbots and propose a framework for designing effective QA chatbots in newsrooms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (91)
  1. BBC News Labs 2018. In-Article Bots. BBC News Labs. https://bbcnewslabs.co.uk/news/2018/in-article-bots/ Accessed on September 15, 2023.
  2. AllSides. 2023. Media Bias. https://www.allsides.com/media-bias.
  3. Understanding Journalists’ Workflows in News Curation. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 228:1–228:13. https://doi.org/10.1145/3544548.3581542
  4. BBC News Labs. 2023a. BBC News Labs Bots Project. https://bbcnewslabs.co.uk/projects/bots/. Accessed: December 17, 2023.
  5. BBC News Labs. 2023b. Scripting chatbots is hard. Here’s how we made it easier for BBC journalists. https://medium.com/bbc-news-labs/bbc-botbuilder-ba8e09b6a2e9.
  6. Blueprint for an AI Bill of Rights. 2023. The White House. https://www.whitehouse.gov/ostp/ai-bill-of-rights/.
  7. “News comes across when I’m in a moment of leisure”: Understanding the practices of incidental news consumption on social media. New Media & Society 20, 10 (2018), 3523–3539. https://doi.org/10.1177/1461444817750396
  8. Petter Bae Brandtzæg and Asbjørn Følstad. 2018. Chatbots: changing user needs and motivations. Interactions 25, 5 (2018), 38–43. https://doi.org/10.1145/3236669
  9. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
  10. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  11. Fiona Campbell. 1997. Journalistic construction of news: information gathering. New library world 98, 2 (1997), 60–64.
  12. Behind the comments section: The ethics of digital native news discussions. (2020).
  13. David Caswell. 2019. Structured journalism and the semantic units of news. Digital Journalism 7, 8 (2019), 1134–1156.
  14. David Caswell and Konstantin Dörr. 2018. Automated Journalism 2.0: Event-driven narratives: From simple descriptions to real stories. Journalism practice 12, 4 (2018), 477–496.
  15. Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. International Journal of Human–Computer Interaction 37, 8 (2021), 729–758. https://doi.org/10.1080/10447318.2020.1841438
  16. Nicole S Cohen. 2019. At work in the digital newsroom. Digital Journalism 7, 5 (2019), 571–591.
  17. Reader comments as press criticism: Implications for the journalistic field. Journalism 17, 6 (2016), 677–693. https://doi.org/10.1177/1464884915579332 arXiv:https://doi.org/10.1177/1464884915579332
  18. Anthropomorphization of AI: Opportunities and Risks. CoRR abs/2305.14784 (2023), 7. https://doi.org/10.48550/arXiv.2305.14784 arXiv:2305.14784
  19. Nicholas Diakopoulos and Mor Naaman. 2011. Towards quality discourse in online news comments. In Proceedings of the Conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 133–142. https://doi.org/10.1145/1958824.1958844
  20. Ziv Epstein, Aaron Hertzmann, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, Robert Mahari, Alex “Sandy” Pentland, Olga Russakovsky, Hope Schroeder, and Amy Smith and. 2023. Art and the science of generative AI. Science 380, 6650 (jun 2023), 1110–1111. https://doi.org/10.1126/science.adh4451
  21. Ethical Journalism Network. 2023. Five Core Principles of Ethical Journalism. https://ethicaljournalismnetwork.org/who-we-are#Mission.
  22. Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Ment Health 4, 2 (06 Jun 2017), e19. https://doi.org/10.2196/mental.7785
  23. Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge, New York, NY, USA. https://doi.org/10.4324/9780203793206
  24. Readers’ perception of computer-generated news: Credibility, expertise, and readability. Journalism 19, 5 (2018), 595–610. https://doi.org/10.1177/1464884916641269
  25. Probing a Community-Based Conversational Storytelling Agent to Document Digital Stories of Housing Insecurity. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 304, 18 pages. https://doi.org/10.1145/3544548.3581109
  26. GREGORY G HOLYK. 2011. The news industry. The Oxford handbook of American public opinion and the media (2011), 89.
  27. Hyehyun Hong and Hyun Jee Oh. 2020a. Utilizing bots for sustainable news business: Understanding users’ perspectives of news bots in the age of social media. Sustainability 12, 16 (2020), 6515.
  28. Hyehyun Hong and Hyun Jee Oh. 2020b. Utilizing Bots for Sustainable News Business: Understanding Users’ Perspectives of News Bots in the Age of Social Media. Sustainability 12, 16 (2020). https://doi.org/10.3390/su12166515
  29. Clayton J. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM). The AAAI Press, Washington, D.C., USA, 216–225. https://doi.org/10.1609/icwsm.v8i1.14550
  30. Convey: Exploring the Use of a Context View for Chatbots. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3173574.3174042
  31. Evaluating and Informing the Design of Chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 895–906. https://doi.org/10.1145/3196709.3196735
  32. Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. https://aclanthology.org/2021.acl-long.304.pdf
  33. Bot in the Bunch: Facilitating Group Chat Discussion by Improving Efficiency and Participation with a Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376785
  34. Suzanne M Kirchhoff. 2010. US newspaper industry in transition. DIANE Publishing.
  35. The future of crowd work. In Proceedings of the Conference on Computer Supported Cooperative Work. ACM, New York, NY, USA, 1301–1318. https://doi.org/10.1145/2441776.2441923
  36. Bran Knowles and John T. Richards. 2021. The Sanction of Authority: Promoting Public Trust in AI. In ACM Conference on Fairness, Accountability, and Transparency. ACM, 262–271. https://doi.org/10.1145/3442188.3445890
  37. David R Krathwohl. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice 41, 4 (2002), 212–218.
  38. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics 7 (2019), 452–466. https://doi.org/10.1162/tacl_a_00276
  39. Ro’ee Levy. 2021. Social Media, News Consumption, and Polarization: Evidence from a Field Experiment. American Economic Review 111, 3 (March 2021), 831–70. https://doi.org/10.1257/aer.20191777
  40. User comments for news recommendation in forum-based social media. Information Sciences 180, 24 (2010), 4929–4939.
  41. What Can You Do? Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS ’16). Association for Computing Machinery, New York, NY, USA, 264–275. https://doi.org/10.1145/2901790.2901842
  42. All Work and No Play?. In Proceedings of the ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173577
  43. Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, 9:1–9:21. https://doi.org/10.1145/3544548.3580652
  44. Tetyana Lokot and Nicholas Diakopoulos. 2016. News Bots. Digital Journalism 4, 6 (2016), 682–699. https://doi.org/10.1080/21670811.2015.1081822
  45. News from Generative Artificial Intelligence Is Believed Less. In ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, New York, NY, USA, 97–106. https://doi.org/10.1145/3531146.3533077
  46. Ewa Luger and Abigail Sellen. 2016. ”Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288
  47. Theodora A. Maniou and Andreas Veglis. 2020. Employing a Chatbot for News Dissemination during Crisis: Design, Implementation and Evaluation. Future Internet 12, 7 (2020). https://doi.org/10.3390/fi12070109
  48. Steven Maras. 2013. Objectivity in journalism. John Wiley & Sons.
  49. Fiona Martin and Colleen Murrell. 2021. Negotiating the conversation: How journalists learn to interact with audiences online. Journalism Practice 15, 6 (2021), 839–859.
  50. Building trust: What works for news organizations. Center for Media Engagement. https://mediaengagement. org/research/building-trust (2017).
  51. Audience engagement in a post-truth age: What it means and how to learn the activities connected with it. Digital Journalism 6, 8 (2018), 1052–1063.
  52. Melvin Mencher and Wendy P Shilton. 1997. News reporting and writing. Brown & Benchmark Publishers Madison, WI.
  53. Job demands, coping, and impacts of occupational stress among journalists: A systematic review. European Journal of Work and Organizational Psychology 25, 5 (2016), 751–772.
  54. Jacob L Nelson. 2021. The next media regime: The pursuit of ‘audience engagement’in journalism. Journalism 22, 9 (2021), 2350–2367.
  55. Killing the Comments: Why Do News Organizations Remove User Commentary Functions? Journalism and Media 2, 4 (2021), 572–583. https://doi.org/10.3390/journalmedia2040034
  56. Oda Elise Nordberg and Frode Guribye. 2023. Conversations with the News: Co-speculation into Conversational Interactions with News Content. In Proceedings of the 5th International Conference on Conversational User Interfaces, CUI. ACM, 32:1–32:11. https://doi.org/10.1145/3571884.3597123
  57. Paw Research Center. 2021. News Consumption Across Social Media in 2021. https://www.pewresearch.org/journalism/2021/09/20/news-consumption-across-social-media-in-2021/.
  58. Social participation in the media: The dialogue of digital journalists with audiences. (2021).
  59. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. CoRR abs/2302.12813 (2023). https://doi.org/10.48550/ARXIV.2302.12813 arXiv:2302.12813
  60. Guanghui Qin and Jason Eisner. 2021. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5203–5212. https://doi.org/10.18653/v1/2021.naacl-main.410
  61. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 784–789. https://doi.org/10.18653/v1/P18-2124
  62. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 2383–2392. https://doi.org/10.18653/v1/D16-1264
  63. Julian Risch and Ralf Krestel. 2020a. A Dataset of Journalists’ Interactions with Their Readership: When Should Article Authors Reply to Reader Comments?. In Proceedings of the ACM International Conference on Information and Knowledge Management. ACM, New York, NY, USA, 3117–3124. https://doi.org/10.1145/3340531.3412764
  64. Julian Risch and Ralf Krestel. 2020b. Top Comment or Flop Comment? Predicting and Explaining User Engagement in Online News Discussions. In Proceedings of the Fourteenth International AAAI Conference on Web and Social Media. AAAI Press, Washington, D.C., USA, 579–589. https://doi.org/10.1609/icwsm.v14i1.7325
  65. Guy Starkey. 2017. Balance and bias in journalism: Representation, regulation and democracy. Bloomsbury Publishing.
  66. S. Shyam Sundar. 1999. Exploring Receivers’ Criteria for Perception of Print and Online News. Journalism & Mass Communication Quarterly 76, 2 (1999), 373–386. https://doi.org/10.1177/107769909907600213
  67. The Ethnobot: Gathering Ethnographies in the Age of IoT. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174178
  68. Edson C. Tandoc and Patrick R. Ferrucci. 2017. Giving in or giving up: What makes journalists use audience feedback in their news work? Computers in Human Behavior 68 (2017), 149–156. https://doi.org/10.1016/j.chb.2016.11.027
  69. Yla R. Tausczik and James W. Pennebaker. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology 29, 1 (2010), 24–54. https://doi.org/10.1177/0261927X09351676
  70. An Investigation of Conversational Agent Interventions Supporting Historical Reasoning in Primary Education. In Proceedings of the 13th International Conference on Intelligent Tutoring Systems (Zagreb, Croatia) (ITS 2016). Springer-Verlag, Berlin, Heidelberg, 260–266. https://doi.org/10.1007/978-3-319-39583-8_27
  71. The New York Times. 2023. What’s the Future for A.I.? https://www.nytimes.com/2023/03/31/technology/ai-chatbots-benefits-dangers.html.
  72. The Verge. 2023. Can news outlets build a ‘trustworthy’ AI chatbot? https://www.theverge.com/2023/8/25/23844868/ai-chatbot-macworld-pcworld-journalism-smart-answers.
  73. Understanding Chatbot-Mediated Task Management. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3173574.3173632
  74. NewsQA: A Machine Comprehension Dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. Association for Computational Linguistics, Vancouver, Canada, 191–200. https://doi.org/10.18653/v1/W17-2623
  75. Jodie B Ullman and Peter M Bentler. 2012. Structural equation modeling. Handbook of Psychology, Second Edition 2 (2012).
  76. Andreas Veglis and Theodora A Maniou. 2019a. Chatbots on the rise: A new narrative in journalism. Studies in media and communication 7, 1 (2019), 1–6.
  77. Andreas Veglis and Theodora A Maniou. 2019b. Embedding a chatbot in a news article: Design and implementation. In Proceedings of the 23rd Pan-Hellenic Conference on Informatics. 169–172.
  78. Sourcing the Sources: An analysis of the use of Twitter and Facebook as a journalistic source over 10 years in The New York Times, The Guardian, and Süddeutsche Zeitung. Digital journalism 6, 7 (2018), 807–828.
  79. Yixue Wang and Nicholas Diakopoulos. 2021. Journalistic Source Discovery: Supporting The Identification of News Sources in User Generated Content. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker (Eds.). ACM, New York, NY, USA, 447:1–447:18. https://doi.org/10.1145/3411764.3445266
  80. Patrick Weber. 2014. Discussions in the comments section: Factors influencing participation and interactivity in online newspapers’ reader comments. New media & society 16, 6 (2014), 941–957.
  81. Ethical and social risks of harm from Language Models. CoRR abs/2112.04359 (2021). arXiv:2112.04359 https://arxiv.org/abs/2112.04359
  82. The limits of audience participation: UGC@ the BBC. In Journalists, Sources, and Credibility. Routledge, Abingdon, UK, 164–178.
  83. User Modeling with Click Preference and Reading Satisfaction for News Recommendation.. In IJCAI. 3023–3029.
  84. When Journalism and Automation Intersect: Assessing the Influence of the Technological Field on Contemporary Newsrooms. Journalism Practice 13, 10 (2019), 1238–1254. https://doi.org/10.1080/17512786.2019.1585198
  85. The Evolving Journalistic Roles on Social Media: Exploring “Engagement” as Relationship-Building between Journalists and Citizens. Journalism Practice 14, 5 (2020), 556–573. https://doi.org/10.1080/17512786.2020.1722729
  86. Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information Access. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, New York, NY, USA, 2–18. https://doi.org/10.1145/3581641.3584031
  87. If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376131
  88. Beyond Self-Diagnosis: How a Chatbot-Based Symptom Checker Should Respond. ACM Trans. Comput.-Hum. Interact. (mar 2023). https://doi.org/10.1145/3589959 Just Accepted.
  89. Informing the design of a news chatbot. In Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents. 224–231.
  90. Informing the Design of a News Chatbot. In IVA ’21: ACM International Conference on Intelligent Virtual Agents, Virtual Event, Japan, September 14-17, 2021. ACM, 224–231. https://doi.org/10.1145/3472306.3478358
  91. Lingke: a Fine-grained Multi-turn Chatbot for Customer Service. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Santa Fe, New Mexico, 108–112. https://aclanthology.org/C18-2024
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 2 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free