Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Trusting the Search: Unraveling Human Trust in Health Information from Google and ChatGPT (2403.09987v1)

Published 15 Mar 2024 in cs.HC

Abstract: People increasingly rely on online sources for health information seeking due to their convenience and timeliness, traditionally using search engines like Google as the primary search agent. Recently, the emergence of generative AI has made LLM powered conversational agents such as ChatGPT a viable alternative for health information search. However, while trust is crucial for adopting the online health advice, the factors influencing people's trust judgments in health information provided by LLM-powered conversational agents remain unclear. To address this, we conducted a mixed-methods, within-subjects lab study (N=21) to explore how interactions with different agents (ChatGPT vs. Google) across three health search tasks influence participants' trust judgments of the search results as well as the search agents themselves. Our key findings showed that: (a) participants' trust levels in ChatGPT were significantly higher than Google in the context of health information seeking; (b) there is a significant correlation between trust in health-related information and trust in the search agent, however only for Google; (c) the type of search tasks did not affect participants' perceived trust; and (d) participants' prior knowledge, the style of information presentation, and the interactive manner of using search agents were key determinants of trust in the health-related information. Our study taps into differences in trust perceptions when using traditional search engines compared to LLM-powered conversational agents. We highlight the potential role LLMs play in health-related information-seeking contexts, where they excel as stepping stones for further search. We contribute key factors and considerations for ensuring effective and reliable personal health information seeking in the age of generative AI.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (90)
  1. Digital Information Credibility: Towards a Set of Guidelines for Quality Assessment of Grey Literature in Multivocal Literature Review. Applied Sciences 13, 7 (2023). https://doi.org/10.3390/app13074483
  2. Hossein Arsham and Miodrag Lovric. 2011. Bartlett’s Test. Springer Berlin Heidelberg, Berlin, Heidelberg, 87–88. https://doi.org/10.1007/978-3-642-04898-2_132
  3. Gaurav Bansal and Merrill Warkentin. 2022. Do You Still Trust? The Role of Age, Gender, and Privacy Concern on Trust after Insider Data Breaches. SIGMIS Database 52, 4 (dec 2022), 9–44. https://doi.org/10.1145/3508484.3508487
  4. The effect of source credibility on consumers’ perceptions of the quality of health information on the Internet. Medical informatics and the Internet in medicine 31, 1 (2006), 45–52.
  5. Avner Ben-Ner and Freyr Halldórsson. 2010. Trusting and trustworthiness: What are they, how to measure them, and what affects them. Journal of Economic Psychology 31 (02 2010), 64–79. https://doi.org/10.1016/j.joep.2009.10.001
  6. Som S. Biswas. 2023. Role of ChatGPT in Public Health. Annals of Biomedical Engineering 51 (2023), 868–869. https://doi.org/10.1007/s10439-023-03172-7
  7. Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.
  8. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  9. Tyler Chin. 2002. Patients put trust in Internet health information. American Medical News 45, 23 (2002), 26–27.
  10. A. Chris. 2023. Top 10 search engines in the World (2023 update). (2023). https://www.reliablesoft.net/top-10-search-engines-in-the-world/
  11. R. J. W. Cline and K. M. Haynes. 2001. Consumer health information seeking on the Internet: the state of the art. Health Education Research 16, 6 (12 2001), 671–692. https://doi.org/10.1093/her/16.6.671 arXiv:https://academic.oup.com/her/article-pdf/16/6/671/9809432/160671.pdf
  12. Online Trust and Health Information Websites. International Journal of Technology and Human Interaction 8 (01 2007). https://doi.org/10.4018/jthi.2012100106
  13. Conversational Information Seeking: Theory and Application. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Machinery, New York, NY, USA, 3455–3458. https://doi.org/10.1145/3477495.3532678
  14. Seeking and Sharing Health Information Online: Comparing Search Engines and Social Media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1365–1376. https://doi.org/10.1145/2556288.2557214
  15. Seeking and sharing health information online: comparing search engines and social media. In Proceedings of the SIGCHI conference on human factors in computing systems. 1365–1376.
  16. Understanding the Effects of Control and Transparency in Searching as Learning. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 498–509. https://doi.org/10.1145/3377325.3377524
  17. Mohan Dutta-Bergman et al. 2003. Trusted online sources of health information: differences in demographics, health beliefs, and health-information orientation. Journal of medical Internet research 5, 3 (2003), e893.
  18. Simon D’Alfonso. 2020. AI in mental health. Current Opinion in Psychology 36 (2020), 112–117.
  19. Misinformation and its Correction. Emerging Trends in the Social and Behavioral Sciences (05 2015).
  20. Expanding Explainability: Towards Social Transparency in AI Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 82, 19 pages. https://doi.org/10.1145/3411764.3445188
  21. A comparison of the information seeking patterns of researchers in the physical and social sciences. Journal of documentation 49, 4 (1993), 356–369.
  22. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 2 (May 2007), 175–191.
  23. Andrew Flanagin and Miriam Metzger. 2000. Perceptions of Internet Information Credibility. Journalism and Mass Communication Quarterly - JOURNALISM MASS COMMUN 77 (09 2000), 515–540. https://doi.org/10.1177/107769900007700304
  24. Andrew J. Flanagin and Miriam J. Metzger. 2007. The role of site features, user attributes, and information verification behaviors on the perceived credibility of web-based information. New Media & Society 9, 2 (2007), 319–342. https://doi.org/10.1177/1461444807075015 arXiv:https://doi.org/10.1177/1461444807075015
  25. What Makes Web Sites Credible? A Report on a Large Quantitative Study. Proc. CHI 3, 61–68. https://doi.org/10.1145/365024.365037
  26. Elements That Affect Web Credibility: Early Results from a Self-Report Study. In CHI ’00 Extended Abstracts on Human Factors in Computing Systems (The Hague, The Netherlands) (CHI EA ’00). Association for Computing Machinery, New York, NY, USA, 287–288. https://doi.org/10.1145/633292.633460
  27. How do users evaluate the credibility of Web sites? Proceedings of the 2003 Conference on Designing for User Experiences (06 2003), 1–15. https://doi.org/10.1145/997078.997097
  28. What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study. In Internet Science, Svetlana S. Bodrunova (Ed.). Springer International Publishing, Cham, 194–208.
  29. Trust Me, I’m Accountable: Trust and Accountability Online. In CHI ’99 Extended Abstracts on Human Factors in Computing Systems (Pittsburgh, Pennsylvania) (CHI EA ’99). Association for Computing Machinery, New York, NY, USA, 79–80. https://doi.org/10.1145/632716.632766
  30. Qi Gao and Geert-Jan Houben. 2010. A Framework for Trust Establishment and Assessment on the Web of Data. In Proceedings of the 19th International Conference on World Wide Web (Raleigh, North Carolina, USA) (WWW ’10). Association for Computing Machinery, New York, NY, USA, 1097–1098. https://doi.org/10.1145/1772690.1772822
  31. Exploring the Role of ChatGPT in patient care (diagnosis and Treatment) and medical research: A Systematic Review. medRxiv (2023). https://doi.org/10.1101/2023.06.13.23291311 arXiv:https://www.medrxiv.org/content/early/2023/06/14/2023.06.13.23291311.full.pdf
  32. Yolanda Gil and Donovan Artz. 2006. Towards Content Trust of Web Resources. In Proceedings of the 15th International Conference on World Wide Web (Edinburgh, Scotland) (WWW ’06). Association for Computing Machinery, New York, NY, USA, 565–574. https://doi.org/10.1145/1135777.1135861
  33. Yuanyuan Guo. 2022. Digital Trust and the Reconstruction of Trust in the Digital Society: An Integrated Model Based on Trust Theory and Expectation Confirmation Theory. Digit. Gov.: Res. Pract. 3, 4, Article 26 (dec 2022), 19 pages. https://doi.org/10.1145/3543860
  34. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks, Standards and Evaluations 2, 4 (2022), 100089. https://doi.org/10.1016/j.tbench.2023.100089
  35. Helena Häußler. 2023. Do Users Trust Search Engines? And If so, Why? Developing a Trust Measure and Applying It in an Experiment. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (Austin, TX, USA) (CHIIR ’23). Association for Computing Machinery, New York, NY, USA, 471–474. https://doi.org/10.1145/3576840.3578280
  36. Zhongxuan He. 2023. From ELIZA to ChatGPT: The Evolution of Chatbots in Public Health. XRDS 29, 3 (apr 2023), 59. https://doi.org/10.1145/3589658
  37. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Archives of internal medicine 165, 22 (2005), 2618–2624.
  38. ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results. J Med Internet Res 25 (25 Jul 2023), e48966. https://doi.org/10.2196/48966
  39. Myeongjun Jang and Thomas Lukasiewicz. 2023. Consistency Analysis of ChatGPT. arXiv:2303.06273 [cs.CL]
  40. The Measurement of the Propensity to Trust Automation. 476–489. https://doi.org/10.1007/978-3-030-21565-1_32
  41. Retrieve, Summarize, and Verify: How will ChatGPT impact information seeking from the medical literature? Journal of the American Society of Nephrology (2023), 10–1681.
  42. Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Research square (02 2023). https://doi.org/10.21203/rs.3.rs-2566942/v1
  43. Modelling trust formation in health information contexts. Journal of Information Science 41 (2015), 415 – 429. https://api.semanticscholar.org/CorpusID:206454953
  44. Ankit Kesharwani. 2020. Do (how) digital natives adopt a new technology differently than digital immigrants? A longitudinal study. Information & Management 57, 2 (2020), 103170. https://doi.org/10.1016/j.im.2019.103170
  45. Haruna Kibirige and Lila Ghemri. 2014. Trust Ranking of Medical Websites. In Proceedings of the 4th ACM Conference on Data and Application Security and Privacy (San Antonio, Texas, USA) (CODASPY ’14). Association for Computing Machinery, New York, NY, USA, 151–154. https://doi.org/10.1145/2557547.2557584
  46. Yong-Mi Kim. 2015. Is Seeking Health Information Online Different from Seeking General Information Online? J. Inf. Sci. 41, 2 (apr 2015), 228–241. https://doi.org/10.1177/0165551514561669
  47. Human versus Artificial Intelligence. Frontiers in Artificial Intelligence 4 (2021). https://doi.org/10.3389/frai.2021.622364
  48. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine 388, 13 (2023), 1233–1239. https://doi.org/10.1056/NEJMsr2214184 arXiv:https://doi.org/10.1056/NEJMsr2214184 PMID: 36988602.
  49. Consumer Health Information Quality, Credibility, and Trust: An Analysis of Definitions, Measures, and Conceptual Dimensions. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (Austin, TX, USA) (CHIIR ’23). Association for Computing Machinery, New York, NY, USA, 197–210. https://doi.org/10.1145/3576840.3578331
  50. Teun Lucassen and Jan Maarten Schraagen. 2010. Trust in Wikipedia: How Users Trust Information from an Unknown Source. In Proceedings of the 4th Workshop on Information Credibility (Raleigh, North Carolina, USA) (WICOW ’10). Association for Computing Machinery, New York, NY, USA, 19–26. https://doi.org/10.1145/1772938.1772944
  51. Rakesh K Maurya. 2023. Using AI Based Chatbot ChatGPT for Practicing Counseling Skills through Role-play. https://doi.org/10.31234/osf.io/s47jb
  52. MAYO CLINIC. 2023. MAYO CLINIC. https://www.mayoclinic.org/
  53. National Institutes of Health. 2023. National Institutes of Health. https://www.nih.gov
  54. new Bing. 2023. new Bing. https://www.bing.com/new?setlang=en&sid=06C46094C75F62702B3A7336C628637F
  55. In Google We Trust: Users’ Decisions on Rank, Position, and Relevance. Journal of Computer‐Mediated Communication 12 (06 2007), 801 – 823. https://doi.org/10.1111/j.1083-6101.2007.00351.x
  56. Identifying Challenges and Opportunities in Human–AI Collaboration in Healthcare. https://dl.acm.org/doi/abs/10.1145/3311957.3359433
  57. Are Men More Affected by AI Anthropomorphism? Comparative Research on the Perception of AI Human-like Characteristics Between Genders. https://doi.org/10.24818/BASIQ/2022/08/090
  58. What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry. Computers in Human Behavior 122 (2021), 106855. https://doi.org/10.1016/j.chb.2021.106855
  59. Trust and Medical AI: The challenges we face and the expertise needed to overcome them. Journal of the American Medical Informatics Association (2020). https://api.semanticscholar.org/CorpusID:221150402
  60. Puthankrissi Sankaranarayan Raju and Michael D Reilly. 1980. Product familiarity and information processing strategies: An exploratory investigation. Journal of Business Research 8, 2 (1980), 187–212.
  61. Abhishek Vaish Raju Vaishya, Anoop Misra. 2023. ChatGPT: Is this version good for healthcare and research? Diabetes and Metabolic Syndrome: Clinical Research and Reviews 17 (2023), 1871–4021. https://doi.org/10.1016/j.dsx.2023.102744
  62. Systematic Review: Trust-Building Factors and Implications for Conversational Agent Design. International Journal of Human-Computer Interaction 37 (09 2020), 1–16. https://doi.org/10.1080/10447318.2020.1807710
  63. Students’ trust judgements in online health information seeking. Health Informatics Journal 21 (2015), 316 – 327. https://api.semanticscholar.org/CorpusID:21888204
  64. Laura Sbaffi and Jennifer Rowley. 2017. Trust and Credibility in Web-Based Health Information: A Review and Agenda for Future Research. J Med Internet Res 19, 6 (19 Jun 2017), e218. https://doi.org/10.2196/jmir.7579
  65. Sebastian Schultheiß and Dirk Lewandowski. 2023. Misplaced trust? The relationship between trust, ability to identify commercially influenced results and search engine preference. J. Inf. Sci. 49, 3 (June 2023), 609–623.
  66. Trust and Distrust on the Web. Comput. Hum. Behav. 45, C (apr 2015), 39–50. https://doi.org/10.1016/j.chb.2014.11.064
  67. Role-Play with Large Language Models. arXiv:2305.16367 [cs.CL]
  68. S. S. SHAPIRO and M. B. WILK. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3-4 (dec 1965), 591–611. https://doi.org/10.1093/biomet/52.3-4.591
  69. A Revised Model of Trust in Internet-Based Health Information and Advice: Cross-Sectional Questionnaire Study. Journal of medical Internet research (2019). https://doi.org/10.2196/11125
  70. Trust and Mistrust of Online Health Sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI ’04). Association for Computing Machinery, New York, NY, USA, 663–670. https://doi.org/10.1145/985692.985776
  71. Guidelines for Developing Trust in Health Websites. In Special Interest Tracks and Posters of the 14th International Conference on World Wide Web (Chiba, Japan) (WWW ’05). Association for Computing Machinery, New York, NY, USA, 1026–1027. https://doi.org/10.1145/1062745.1062851
  72. A Framework for Understanding Trust Factors in Web-Based Health Advice. Int. J. Hum.-Comput. Stud. 64, 8 (aug 2006), 697–713. https://doi.org/10.1016/j.ijhcs.2006.02.007
  73. How do patients evaluate and make use of online health information? Social Science and Medicine 64, 9 (2007), 1853–1862. https://doi.org/10.1016/j.socscimed.2007.01.012
  74. Himani Singal and Shruti Kohli. 2016. Intellectualizing TRUST for Medical Websites. In Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies (Udaipur, India) (ICTCS ’16). Association for Computing Machinery, New York, NY, USA, Article 83, 4 pages. https://doi.org/10.1145/2905055.2905293
  75. The Nuanced Nature of Trust and Privacy Control Adoption in the Context of Google. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 591, 23 pages. https://doi.org/10.1145/3544548.3581387
  76. Liesbet Van Bulck and Philip Moons. 2023. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value, and danger of ChatGPT-generated responses to health questions. European Journal of Cardiovascular Nursing (04 2023), zvad038. https://doi.org/10.1093/eurjcn/zvad038
  77. Trust in Health Websites: A Review of an Emerging Field. In Proceedings of the 1st ACM International Health Informatics Symposium (Arlington, Virginia, USA) (IHI ’10). Association for Computing Machinery, New York, NY, USA, 700–709. https://doi.org/10.1145/1882992.1883100
  78. Marco Viviani and Gabriella Pasi. 2017. Credibility in social media: opinions, news, and health information - A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 7 (06 2017), e1209. https://doi.org/10.1002/widm.1209
  79. Ethical Considerations of Using ChatGPT in Health Care. J Med Internet Res 25 (11 Aug 2023), e48009. https://doi.org/10.2196/48009
  80. C. Wathen and Jacquelyn Burkell. 2002. Believe It or Not: Factors Influencing Credibility on the Web. JASIST 53 (01 2002), 134–144. https://doi.org/10.1002/asi.10016
  81. Wikipedia. 2023. Filter bubble - Wikipedia. https://en.wikipedia.org/wiki/Filter_bubble#:~:text=A%20filter%20bubble%20or%20ideological,can%20result%20from%20personalized%20searches. (Accessed on 09/29/2023).
  82. Wikipwdia. 2023. Hallucination (artificial intelligence) - Wikipedia. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence). (Accessed on 10/10/2023).
  83. Fast searching for information on the Internet to use in a learning context: The impact of domain knowledge. Computers and Education 52, 3 (2009), 640–648.
  84. Robert H. Wortham and Andreas Theodorou. 2017. Robot transparency, trust and utility. Connection Science 29, 3 (2017), 242–248. https://doi.org/10.1080/09540091.2017.1313816 arXiv:https://doi.org/10.1080/09540091.2017.1313816
  85. Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information Access. (2023), 2–18. https://doi.org/10.1145/3581641.3584031
  86. ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience. arXiv:2307.01135 [cs.AI]
  87. CHQ-Summ: A Dataset for Consumer Healthcare Question Summarization. arXiv:2206.06581 [cs.CL]
  88. Yahoo. 2023. L6 - Yahoo! Answers Comprehensive Questions and Answers version 1.0. https://webscope.sandbox.yahoo.com/catalog.php?datatype=l&did=11
  89. Shifting Trust: Examining How Trust and Distrust Emerge, Transform, and Collapse in COVID-19 Information Seeking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 78, 21 pages. https://doi.org/10.1145/3491102.3501889
  90. Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge? Journal of Translational Medicine 21, 1 (19 Apr 2023), 269. https://doi.org/10.1186/s12967-023-04123-5
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Xin Sun (151 papers)
  2. Rongjun Ma (1 paper)
  3. Xiaochang Zhao (2 papers)
  4. Zhuying Li (7 papers)
  5. Janne Lindqvist (9 papers)
  6. Abdallah El Ali (14 papers)
  7. Jos A. Bosch (5 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets