Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

User Privacy Harms and Risks in Conversational AI: A Proposed Framework (2402.09716v1)

Published 15 Feb 2024 in cs.HC and cs.CY

Abstract: This study presents a unique framework that applies and extends Solove (2006)'s taxonomy to address privacy concerns in interactions with text-based AI chatbots. As chatbot prevalence grows, concerns about user privacy have heightened. While existing literature highlights design elements compromising privacy, a comprehensive framework is lacking. Through semi-structured interviews with 13 participants interacting with two AI chatbots, this study identifies 9 privacy harms and 9 privacy risks in text-based interactions. Using a grounded theory approach for interview and chatlog analysis, the framework examines privacy implications at various interaction stages. The aim is to offer developers, policymakers, and researchers a tool for responsible and secure implementation of conversational AI, filling the existing gap in addressing privacy issues associated with text-based AI chatbots.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (117)
  1. AI-based chatbots in customer service and their effects on user compliance. Electronic Markets, 31, March 2020.
  2. An Overview of Chatbot Technology. In Ilias Maglogiannis, Lazaros Iliadis, and Elias Pimenidis, editors, Artificial Intelligence Applications and Innovations, IFIP Advances in Information and Communication Technology, pages 373–383, Cham, 2020. Springer.
  3. Chatbots’ effectiveness in service recovery. International Journal of Information Management, page 102679, July 2023.
  4. Computers as Bad Social Actors: Dark Patterns and Anti-Patterns in Interfaces that Act Socially. 2023. Publisher: arXiv Version Number: 3.
  5. User-centered privacy-by-design: Evaluating the appropriateness of design prototypes. International Journal of Human-Computer Studies, 154:102641, 2021.
  6. Divyaa Balaji. Assessing User Satisfaction with Information Chatbots: A Preliminary Investigation. Submitted in partial fulfillment of requirements for master of science in psychology, University of Twente, 2019.
  7. From awareness to repartee: Sharing location within social groups. In Conference on Human Factors in Computing Systems - Proceedings, page 497–506, April 2008.
  8. Using Chatbots as AI Conversational Partners in Language Learning. Applied Sciences, 12(17):8427, January 2022. Number: 17 Publisher: Multidisciplinary Digital Publishing Institute.
  9. Privacy Concerns in Chatbot Interactions: When to Trust and When to Worry. In Constantine Stephanidis, Margherita Antona, and Stavroula Ntoa, editors, HCI International 2021 - Posters, Communications in Computer and Information Science, pages 391–399, Cham, 2021. Springer.
  10. An Investigation Into the Sensitivity of Personal Information and Implications for Disclosure: A UK Perspective. Frontiers in Computer Science, 4, 2022.
  11. Understanding the impact of control levels over emotion-aware chatbots. Computers in Human Behavior, 129:107122, April 2022.
  12. Debmalya Biswas. Privacy preserving chatbot conversations. In 2020 IEEE Third International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), page 179–182, December 2020.
  13. Artificially intelligent chatbots in digital mental health interventions: a review. Expert Review of Medical Devices, 18(sup1):37–49, December 2021.
  14. Drivers of privacy concerns when interacting with a chatbot in a customer service encounter. International Journal of Bank Marketing, 40(6):1159–1181, January 2022. Publisher: Emerald Publishing Limited.
  15. EU countries vote unanimously to approve AI Act. IAPP, 2024. https://iapp.org/news/a/eu-countries-vote-unanimously-to-approve-ai-act/.
  16. Why People Use Chatbots, volume 10673 of Lecture Notes in Computer Science, page 377–392. Springer, Cham, 2017.
  17. M. Calo. The Boundaries of Privacy Harm. 86 Indiana Law Journal 1131 (2011), 86(3), July 2011.
  18. How do ai-driven chatbots impact user experience? examining gratifications, perceived privacy risk, satisfaction, loyalty, and continued use. Journal of Broadcasting & Electronic Media, 64(4):592–614, October 2020.
  19. AI-chatbots on the services frontline addressing the challenges and opportunities of agency. Journal of Retailing and Consumer Services, 63:102735, November 2021.
  20. Privacy harms. BUL Rev., 102:793, 2022.
  21. Privacy harms. Boston University Law Review, 102(3):793–864, 2022.
  22. Privacy harms. Hastings Law Journal, 69(4):1039–1098, 2017.
  23. Pedro Costa. Conversing with personal digital assistants: on gender and artificial intelligence. Journal of Science and Technology of the Arts, 10(33):59–72, September 2018.
  24. The role of attitude toward chatbots and privacy concern on the relationship between attitude toward mobile advertising and behavioral intent to use chatbots. Italian Journal of Marketing, 2021(1):83–102, June 2021.
  25. Judith Wagner DeCew. In pursuit of privacy: Law, ethics, and the rise of technology. Cornell University Press, 1997.
  26. Jayati Dev. Privacy-Preserving Conversational Interfaces. Ph.D., Indiana University, United States – Indiana, 2022. ISBN: 9798841768746.
  27. “How Can I Help You?”: User Perceptions of Privacy in Retail Chat Agents. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–6, Hamburg Germany, April 2023. ACM.
  28. Christian L. M. Dillman D. A., Smyth J. D. Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons, Inc, Hoboken, NJ, 2014.
  29. Cem Dilmaghani. Bias in AI: What it is, Types, Examples & 6 Ways to Fix it in 2024, September 2023. https://research.aimultiple.com/ai-bias/.
  30. The rise of chatbots–new personal assistants in foreign language learning. Procedia Computer Science, 169:542–546, 2020.
  31. M. V. Eeuwen. Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers. 2017.
  32. Timing is everything? the effects of timing and placement of online privacy indicators. page 319–328, April 2009.
  33. Parsing the Turing test. Philosophical and methodological issues in the quest for the thinking computer. Foreword by Daniel C. Dennett. January 2009.
  34. How AI chatbots have reshaped the frontline interface in China: examining the role of sales–service ambidexterity and the personalization–privacy paradox. International Journal of Emerging Markets, 17(4):967–986, January 2022. Publisher: Emerald Publishing Limited.
  35. Theoretical Saturation, page 1–2. John Wiley & Sons, Ltd, 2017.
  36. Towards transparency by design for artificial intelligence. Science and Engineering Ethics, 26(6):3333–3361, 2020.
  37. Anush Fernandes. NLP, NLU, NLG and how chatbots work, January 2018. https://yourstory.com/mystory/fac4d8fd9f-nlp-nlu-nlg-and-how.
  38. NIST Privacy Framework. NIST, January 2020. https://www.nist.gov/privacy-framework/privacy-framework.
  39. What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study. In Svetlana S. Bodrunova, editor, Internet Science, Lecture Notes in Computer Science, pages 194–208, Cham, 2018. Springer.
  40. Alyson Gamble. Artificial intelligence and mobile apps for mental healthcare: a social informatics perspective. Aslib Journal of Information Management, 72(4):509–523, January 2020. Publisher: Emerald Publishing Limited.
  41. “like lesbians walking the perimeter”: Experiences of u.s. lgbtq+ folks with online security, safety, and privacy advice.
  42. Trust and tam in online shopping: An integrated model. MIS Quarterly, 27(1):51–90, 2003.
  43. The more competent, the better? the effects of perceived competencies on disclosure towards conversational artificial intelligence. Social Science Computer Review, page 08944393221142787, November 2022.
  44. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior, 97:304–316, August 2019.
  45. The dark (patterns) side of ux design. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–14, 2018.
  46. Information needs and perceptions of chatbots for hypertension medication self-management: a mixed methods study. JAMIA Open, 4(2):ooab021, April 2021.
  47. Ece Gumusel. Empowering privacy-centered practices for diverse communities. September 2023. Accepted: 2023-10-02T18:56:07Z.
  48. Woodrow Hartzog. Privacy’s Blueprint. Harvard University Press, 2018.
  49. Assessing the privacy risks of data sharing in genomics. Public Health Genomics, 14(1):17–25, March 2010.
  50. Introduction to the pragmatics of computermediated communication. In Susan Herring, Dieter Stein, and Tuija Virtanen, editors, Pragmatics of Computer-Mediated Communication, pages 3–32. DE GRUYTER, January 2013.
  51. Temporal aspects of computer-mediated communication. Computers in Human Behavior, 4(2):147–165, January 1988.
  52. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in Human Behavior, 49:245–250, August 2015.
  53. Lance J Hoffman. Modern methods for computer security and privacy. Prentice-Hall Englewood Cliffs, 1977.
  54. A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, July 2021.
  55. Privacy Concerns in Chatbot Interactions. In Asbjørn Følstad, Theo Araujo, Symeon Papadopoulos, Effie Lai-Chong Law, Ole-Christoffer Granmo, Ewa Luger, and Petter Bae Brandtzaeg, editors, Chatbot Research and Design, Lecture Notes in Computer Science, pages 34–48, Cham, 2020. Springer.
  56. Guardlens: Supporting safer online browsing for people with visual impairments. In Nineteenth Symposium on Usable Privacy and Security (SOUPS 2023), pages 361–380, 2023.
  57. A Multi-Industry Analysis of the Future Use of AI Chatbots. Human Behavior and Emerging Technologies, 2022:1–14, November 2022.
  58. A Study of Today’s A.I. through Chatbots and Rediscovery of Machine Intelligence. International Journal of u- and e-Service, Science and Technology, 8:277–284, July 2015.
  59. “how technical do you get? i’m an english teacher”: Teaching and learning cybersecurity and ai ethics in high school. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2032–2032. IEEE, 2023.
  60. Chatbot Advertising As a Double-Edged Sword: The Roles of Regulatory Focus and Privacy Concerns. Journal of Advertising, 0(0):1–19, April 2022. https://doi.org/10.1080/00913367.2022.2043795.
  61. Regulating privacy dark patterns in practice-drawing inspiration from the california privacy rights act. Georgetown Law Technology Review, 5(2):250–276, 2021.
  62. Conversational ai: An overview of methodologies, applications & future scope. In 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), pages 1–7. IEEE, 2019.
  63. Trust and digital privacy: willingness to disclose personal information to banking chatbot services. Journal of Financial Services Marketing, 28(2):337–357, June 2023.
  64. Personal privacy through understanding and action: five pitfalls for designers. Personal and ubiquitous computing, 8:440–454, 2004.
  65. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine, 388(13):1233–1239, March 2023. Massachusetts Medical Society https://doi.org/10.1056/NEJMsr2214184.
  66. "I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–12, Honolulu HI USA, April 2020. ACM.
  67. The impact of gdpr on global technology development, 2019.
  68. Quantifying thematic saturation in qualitative data analysis. Field Methods, 30(3):191–207, August 2018.
  69. Frontiers: Machines vs. Humans: The Impact of Artificial Intelligence Chatbot Disclosure on Customer Purchases. Marketing Science, 38(6):937–947, November 2019. Publisher: INFORMS.
  70. AI in E-Commerce: Application of the Use and Gratification Model to The Acceptance of Chatbots. Sustainability, 14(21):14270, January 2022. Number: 21 Publisher: Multidisciplinary Digital Publishing Institute.
  71. The cost of reading privacy policies. 2008. Accepted: 2015-09-30T22:41:07Z.
  72. Fairness, accountability, transparency, and ethics (fate) in artificial intelligence (ai) and higher education: A systematic review. Computers and Education: Artificial Intelligence, 5:100152, January 2023.
  73. Joseph Mercieca. Human-Chatbot Interaction - Assessing technology acceptance, confidence and trust in chatbots within their application areas. PhD thesis, December 2019.
  74. Qualitative Data Analysis: A Methods Sourcebook. 4th edition, 2014.
  75. Strategies for reducing online privacy risks: Why consumers read (or don’t read) online privacy notices. Journal of Interactive Marketing, 18(3):15–29, August 2004.
  76. Children seen but not heard: When parents compromise children’s online privacy. In Proceedings of the 24th international conference on World Wide Web, pages 776–786, 2015.
  77. Darakhshan J. Mir. Designing for the privacy commons. Governing Privacy in Knowledge Commons, 2021.
  78. Yi Mou and Kun Xu. The media inequality: Comparing the initial human-human and human-AI social interactions. Computers in Human Behavior, 72:432–440, July 2017.
  79. Simulating the Effects of Social Presence on Trust, Privacy Concerns & Usage Intentions in Automated Bots for Finance. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 190–199, September 2020.
  80. Helen Nissenbaum. Privacy as Contextual Integrity. Washington Law Review, 79(1):119, February 2004.
  81. Privacy risk perceptions and privacy protection strategies. In Elisabeth de Leeuw, Simone Fischer-Hübner, Jimmy Tseng, and John Borking, editors, Policies and Research in Identity Management, The International Federation for Information Processing, page 121–138, Boston, MA, 2008. Springer US.
  82. Endang Wahyu Pamungkas. Emotionally-Aware Chatbots: A Survey, June 2019. arXiv:1906.09774 [cs].
  83. I, chatbot! the impact of anthropomorphism and gaze direction on willingness to disclose personal information and behavioral intentions. Psychology & Marketing, 40(7):1372–1387, 2023.
  84. Determinants and consequences of trust in AI-based customer service chatbots. The Service Industries Journal, 43(9-10):642–675, July 2023.
  85. William L. Prosser. Privacy. California Law Review, 1960.
  86. Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28, September 2020.
  87. Charles D. Raab. The distribution of privacy risks: Who needs protection? The Information Society, 14(4):263–274, November 1998.
  88. Creepiness: Its antecedents and impact on loyalty when interacting with a chatbot. Psychology & Marketing, 38(12):2339–2356, 2021.
  89. Chatbot for university related FAQs. September 2017. Pages: 1530.
  90. Chatbots in retailers’ customer communication: How to measure their acceptance? Journal of Retailing and Consumer Services, 56:102176, September 2020.
  91. Taking trust seriously in privacy law. SSRN Electronic Journal, 2015.
  92. Computer-mediated communication: Identity and social interaction in an electronic environment. Genetic, Social, and General Psychology Monographs, 124:434–464, November 1998.
  93. Rowena Rodrigues. Legal and human rights issues of ai: Gaps, challenges and vulnerabilities. Journal of Responsible Technology, 4:100005, December 2020.
  94. Privacy as Commons: Case Evaluation Through the Governing Knowledge Commons Framework. Journal of Information Policy, 8:116–166, March 2018.
  95. Governing Privacy in Knowledge Commons. Cambridge University Press, March 2021.
  96. “I just shared your responses”: Extending Communication Privacy Management Theory to Interactions with Conversational Agents. Proceedings of the ACM on Human-Computer Interaction, 4:08:1–08:18, January 2020.
  97. Chatbots: Are they Really Useful? LDV Forum, 22:29–49, July 2007.
  98. Privacy Enhancing Technologies: A Review. Privacy Enhancing Technologies.
  99. Privacy for children and teenagers on social networks from a usability perspective: a case study on facebook. In Proceedings of the 2017 ACM on Web Science Conference, pages 63–71, 2017.
  100. Daniel J. Solove. Conceptualizing privacy. California Law Review, 90(4):1087–1155, 2002.
  101. Daniel J. Solove. A Taxonomy of Privacy. University of Pennsylvania Law Review, 154(3):477–564, 2006. Publisher: The University of Pennsylvania Law Review.
  102. Will artificial intelligence replace human customer service? The impact of communication quality and privacy risks on adoption intention. Journal of Retailing and Consumer Services, 66, May 2022.
  103. Venturing into the uncanny valley of mind—The influence of mind attribution on the acceptance of human-like characters in a virtual reality setting. Cognition, 160:43–50, March 2017.
  104. The Challenge for Cities of Governing Spatial Data Privacy. February 2023.
  105. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1):15, February 2023.
  106. Personalization and privacy: a survey of privacy risks and remedies in personalization-based systems. User Modeling and User-Adapted Interaction, 22(1):203–220, April 2012.
  107. Location-sharing technologies: Privacy risks and controls. I/S: A Journal of Law and Policy for the Information Society, 6(2):119–152, 2010.
  108. Designing for data awareness: addressing privacy and security concerns about “smart” technologies. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing, pages 364–367, 2021.
  109. How to Trick AI: Users’ Strategies for Protecting Themselves from Automatic Personality Assessment. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–15, Honolulu HI USA, April 2020. ACM.
  110. Ai ethics issues in real world: Evidence from ai incident database. arXiv preprint arXiv:2206.07635, 2022.
  111. Alan Westin. Privacy And Freedom. Washington and Lee Law Review, 25(1):166, March 1968.
  112. Manuel Wörsdörfer. The eu’s artificial intelligence act: an ordoliberal assessment. AI and Ethics, pages 1–16, 2023.
  113. Computer-Mediated Communication in Personal Relationships. October 2017.
  114. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136, 2023.
  115. Jennifer Zamora. I’m Sorry, Dave, I’m Afraid I Can’t Do That: Chatbot Perception and Expectations. In Proceedings of the 5th International Conference on Human Agent Interaction, HAI ’17, pages 253–260, New York, NY, USA, October 2017. Association for Computing Machinery.
  116. “it’s a fair game”, or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents. arXiv preprint arXiv:2309.11653, 2023.
  117. “the teachers are confused as well”: A multiple-stakeholder ethics discussion on large language models in computing education. arXiv e-prints, pages arXiv–2401, 2024.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets