Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The effect of source disclosure on evaluation of AI-generated messages: A two-part study (2311.15544v2)

Published 27 Nov 2023 in cs.CL

Abstract: Advancements in AI over the last decade demonstrate that machines can exhibit communicative behavior and influence how humans think, feel, and behave. In fact, the recent development of ChatGPT has shown that LLMs can be leveraged to generate high-quality communication content at scale and across domains, suggesting that they will be increasingly used in practice. However, many questions remain about how knowing the source of the messages influences recipients' evaluation of and preference for AI-generated messages compared to human-generated messages. This paper investigated this topic in the context of vaping prevention messaging. In Study 1, which was pre-registered, we examined the influence of source disclosure on people's evaluation of AI-generated health prevention messages compared to human-generated messages. We found that source disclosure (i.e., labeling the source of a message as AI vs. human) significantly impacted the evaluation of the messages but did not significantly alter message rankings. In a follow-up study (Study 2), we examined how the influence of source disclosure may vary by the participants' negative attitudes towards AI. We found a significant moderating effect of negative attitudes towards AI on message evaluation, but not for message selection. However, for those with moderate levels of negative attitudes towards AI, source disclosure decreased the preference for AI-generated messages. Overall, the results of this series of studies showed a slight bias against AI-generated messages once the source was disclosed, adding to the emerging area of study that lies at the intersection of AI and communication.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Importance values for importance–performance analysis: A formula for spreading out values derived from preference rankings. Journal of Business Research 60, 115–121.
  2. Ordinal preference elicitation methods in health economics and health services research: using discrete choice experiments and ranking methods. British Medical Bulletin 103, 21–44.
  3. Campaigns and counter campaigns: reactions on twitter to e-cigarette education. Tobacco Control 26, 226–229.
  4. The measurement of values in surveys: A comparison of ratings and rankings. Public Opinion Quarterly 49, 535–552.
  5. Human evaluations of machine translation in an ethically charged situation. New Media & Society 25, 1087–1107.
  6. Unc perceived message effectiveness: validation of a brief scale. Annals of Behavioral Medicine 53, 732–742.
  7. Humans versus ai: whether and why we prefer human-created compared to ai-created artwork. Cognitive Research 8, 1–22.
  8. The Process of Communication. Holt, Rinehart, and Winston.
  9. Bigscience, 2022. Bigscience rail license v1.0. https://huggingface.co/spaces/bigscience/license.
  10. Assessment of patterns in e-cigarette use among adults in the us, 2017-2020. JAMA Network Open 5, e2223266–e2223266.
  11. Perceived effectiveness of objective elements of vaping prevention messages among adolescents. Tobacco Control 32, e228–e235.
  12. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 .
  13. Conservatism predicts aversion to consequential artificial intelligence. Plos One 16, e0261467.
  14. Analysis of variance; designed experiments, in: Statistical Models in S. Routledge, pp. 145–193.
  15. The heuristic-systematic model in its broader context, in: Dual-process theories in social psychology. The Guilford Press, pp. 73–96.
  16. Vaporous marketing: Uncovering pervasive electronic cigarette advertisements on twitter. PloS One 11, e0157304.
  17. Source expertise and persuasion: The effects of perceived opposition or support on message scrutiny. Personality and Social Psychology Bulletin 38, 90–100.
  18. E-cigarette marketing and communication: How e-cigarette companies market e-cigarettes and the public engages with e-cigarette information. Nicotine and Tobacco Research 21, 14–24.
  19. Textacy: Nlp, before and after spacy. https://github.com/chartbeat-labs/textacy. Accessed September 10, 2022.
  20. Transparency and the black box problem: Why we do not trust ai. Philosophy & Technology 34, 1607–1622.
  21. Reactions to messages about smoking, vaping and covid-19: Two national experiments. Tobacco Control 31, 402–410.
  22. topicmodels: An r package for fitting topic models. Journal of Statistical Software 40, 1–30.
  23. Advances in natural language processing. Science 349, 261–266. doi:10.1126/science.aaa8685.
  24. spacy: Industrial-strength natural language processing in python. https://doi.org/10.5281/zenodo.1212303.
  25. Vader: A parsimonious rule-based model for sentiment analysis of social media text, in: Proceedings of the International AAAI Conference on Web and Social Media, pp. 216–225.
  26. The effect of characteristics of source credibility on consumer behaviour: A meta-analysis. Journal of Retailing and Consumer Services 53, 101736.
  27. Ai-mediated communication: How the perception that profile text was written by ai affects trustworthiness, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13.
  28. Why are we averse towards algorithms? a comprehensive literature review on algorithm aversion. ECIS 2020 Proceedings .
  29. JustAnotherArchivist, 2021. snscrape: A social networking service scraper in python. Github. URL: https://github.com/JustAnotherArchivist/snscrape.
  30. Challenges and applications of large language models. arXiv .
  31. Working with ai to persuade: Examining a large language model’s ability to generate pro-vaccination messages. Proceedings of the ACM on Human-Computer Interaction (CSCW) 7, 1–29.
  32. Consumer preferences for food labelling attributes: Comparing direct ranking and best–worst scaling for measurement of attribute importance, preference intensity and attribute dominance. Food Quality and Preference 29, 77–88.
  33. The structure and function of communication in society, in: Bryson, L. (Ed.), The Communication of Ideas. New York: Institute for Religious and Social Studies, pp. 37–51.
  34. Artificial intelligence for health message generation: an empirical study using a large language model (llm) and prompt engineering. Frontiers in Communication 8, 1129082.
  35. Incorporating message framing into narrative persuasion to curb e‐cigarette use among college students. Risk Analysis 40, 1677–1690.
  36. Will ai console me when i lose my pet? understanding perceptions of ai-mediated email writing, in: Proceedings of the 2022 CHI conference on human factors in computing systems, pp. 1–13.
  37. Resistance to medical artificial intelligence. Journal of Consumer Research 46, 629–650.
  38. Artificial Intelligence: Structures and strategies for complex problem solving. London: Pearson Education.
  39. Vaping discussion in the covid-19 pandemic: An observational study using twitter data. PloS One 16, e0260290.
  40. User generated content and credibility evaluation of online health information: A meta analytic study. Telematics and Informatics 34, 472–486.
  41. Deep learning: A critical appraisal. arXiv .
  42. The next decade in ai: four steps towards robust artificial intelligence. arXiv URL: https://doi.org/10.48550/arXiv.2002.06177.
  43. Health chatbots acceptability moderated by perceived stigma and severity: A cross-sectional survey. Digital Health 7, 20552076211063012.
  44. Artificial Intelligence: A guide for thinking humans. Penguin UK, London.
  45. Rhetoric in the Middle Ages: A history of rhetorical theory from Saint Augustine to the Renaissance. University of California Press, Berkeley, CA.
  46. Going viral. Polity.
  47. Evaluating the actual and perceived effectiveness of e-cigarette prevention advertisements among adolescents. Addictive Behaviors 109, 106473.
  48. Persuasion: Theory and research. Sage Publications.
  49. Using best-worst scaling to rank factors affecting vaccination demand in northern nigeria. Vaccine 35, 6429–6437.
  50. Prolific. ac—a subject pool for online experiments. Journal of Behavioral and Experimental Finance 17, 22–27.
  51. The elaboration likelihood model of persuasion, in: Springer New York, pp. 1–24.
  52. The persuasiveness of source credibility: A critical review of five decades’ evidence. Journal of Applied Social Psychology 34, 243–281.
  53. The evaluation of preference and perceived quality of health communication icons associated with covid-19 prevention measures. Healthcare 9, 1115.
  54. Ai-generated vs. human artworks. a perception bias towards artificial intelligence?, in: Extended abstracts of the 2020 CHI conference on human factors in computing systems, pp. 1–10.
  55. Comprehension’s role in persuasion: The case of its moderating effect on the persuasive impact of source cues. Journal of Consumer Research 18, 52–62.
  56. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv URL: http://arxiv.org/abs/1908.10084.
  57. Comparison of message and effects perceptions for the real cost e-cigarette prevention ads. Health Communication 36, 1222–1230.
  58. Artificial intelligence: A modern approach 4th Edition. Prentice Hall, Hoboken.
  59. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 .
  60. The general attitudes towards artificial intelligence scale (gaais): Confirmatory validation and associations with personality, corporate distrust, and general trust. International Journal of Human-Computer Interaction 39, 2724–2741.
  61. Harnessing artificial intelligence for health message generation: The folic acid message engine. Journal of Medical Internet Research 24, e28858.
  62. Ai composer bias: Listeners like music less when they think it was composed by an ai. Journal of Experimental Psychology: Applied 29, 676.
  63. A mathematical theory of communication. Bell Systems Technical Journal 27, 379–423.
  64. The psychometric paradigm. Science 236, 280–285.
  65. Natural language processing with Transformers. O’Reilly Media, Inc.
  66. Identifying message content to reduce vaping: Results from online message testing trials in young adult tobacco users. Addictive Behaviors 115, 106778.
  67. Characteristics of e-cigarette use behaviors among us youth, 2020. JAMA Network Open 4, e2111336–e2111336.
  68. Moralization of e-cigarette use and regulation: A mixed-method computational analysis of opinion polarization. Health Communication 38, 1666–1676.
  69. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837.
  70. Source effects in communication and persuasion research: A meta-analysis of effect size. Journal of the Academy of Marketing Science 21, 101–112.
  71. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv URL: https://doi.org/10.48550/arXiv.2309.01219.
  72. Practical statistical power analysis using Webpower and R. ISDSA Press.
  73. An artificially intelligent, natural language processing chatbot designed to promote covid-19 vaccination: A proof-of-concept pilot study. Digital Health 9, 20552076231155679.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Sue Lim (4 papers)
  2. Ralf Schmälzle (4 papers)
Citations (12)
View on Semantic Scholar