Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents (2309.11653v2)

Published 20 Sep 2023 in cs.HC, cs.AI, and cs.CR

Abstract: The widespread use of LLM-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Mental models of mere mortals with explanations of reinforcement learning. ACM Transactions on Interactive Intelligent Systems (TiiS) 10, 2 (2020), 1–37.
  2. The role of shared mental models in human-AI teams: a theoretical review. Theoretical Issues in Ergonomics Science 24, 2 (2023), 129–175.
  3. End user and expert perceptions of threats and potential countermeasures. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 230–239.
  4. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI conference on human computation and crowdsourcing, Vol. 7. 2–11.
  5. Industrial practitioners’ mental models of adversarial machine learning. In Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022). 97–116.
  6. What does it mean for a language model to preserve privacy?. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2280–2292.
  7. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 (2022).
  8. Extracting Training Data from Large Language Models.. In USENIX Security Symposium, Vol. 6.
  9. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) (2023).
  10. Is there a reverse privacy paradox? an exploratory analysis of gaps between privacy perspectives and privacy-seeking behaviors. Proceedings on Privacy Enhancing Technologies 1 (2023), 455–476.
  11. Russell Cropanzano and Marie S Mitchell. 2005. Social exchange theory: An interdisciplinary review. Journal of management 31, 6 (2005), 874–900.
  12. An efficient dp-sgd mechanism for large scale nlu models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 4118–4122.
  13. Trust and privacy concern within social networking sites: A comparison of Facebook and MySpace. AMCIS 2007 proceedings (2007), 339.
  14. Sheryl Estrada. 2023. A startup CFO used ChatGPT to build an FP&A tool—here’s how it went. https://fortune.com/2023/03/01/startup-cfo-chatgpt-finance-tool/ Accessed: 09/11/2023.
  15. Pedro Ferreira. 2023. Can ChatGPT Improve Technical Analysis and Trading Techniques? https://www.financemagnates.com/trending/can-chatgpt-improve-technical-analysis-and-trading-techniques/ Accessed: 09/11/2023.
  16. Andrea Fox. 2023. ChatGPT scored 72study shows. https://www.healthcareitnews.com/news/chatgpt-scored-72-clinical-decision-accuracy-mgb-study-shows Accessed: 09/11/2023.
  17. New Me: Understanding Expert and {{\{{Non-Expert}}\}} Perceptions and Usage of the Tor Anonymity Network. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017). 385–398.
  18. Koala: A dialogue model for academic research. Blog post, April 1 (2023).
  19. Thomas Germain. 2023. A Mental Health App Tested ChatGPT on Its Users. The Founder Said Backlash Was Just a Misunderstanding. https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534/ Accessed: 09/11/2023.
  20. First comes love, then comes Google: An investigation of uncertainty reduction strategies and self-disclosure in online dating. Communication Research 38, 1 (2011), 70–100.
  21. Pamela Grimm. 2010. Social desirability bias. Wiley international encyclopedia of marketing (2010).
  22. Frances Grodzinsky and Herman T Tavani. 2010. Applying the “contextual integrity” model of privacy to personal blogs in the blogoshere. (2010).
  23. The False Promise of Imitating Proprietary LLMs. arXiv preprint arXiv:2305.15717 (2023).
  24. Eszter Hargittai and Alice Marwick. 2016. “What can I really do?” Explaining the privacy paradox with online apathy. International journal of communication 10 (2016), 21.
  25. Can children understand machine learning concepts? The effect of uncovering black boxes. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–11.
  26. Privacy concerns in chatbot interactions. In Chatbot Research and Design: Third International Workshop, CONVERSATIONS 2019, Amsterdam, The Netherlands, November 19–20, 2019, Revised Selected Papers 3. Springer, 34–48.
  27. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504 (2022).
  28. Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation. arXiv preprint arXiv:2304.07854 (2023).
  29. Do people hold a humanoid robot morally accountable for the harm it causes?. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 33–40.
  30. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning. PMLR, 10697–10707.
  31. {{\{{“My}}\}} Data Just Goes {{\{{Everywhere:”}}\}} User Mental Models of the Internet and Implications for Privacy and Security. In Eleventh symposium on usable privacy and security (SOUPS 2015). 39–52.
  32. Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–14.
  33. ProPILE: Probing Privacy Leakage in Large Language Models. arXiv preprint arXiv:2307.01881 (2023).
  34. Youjeong Kim and S Shyam Sundar. 2012. Anthropomorphism of computers: Is it mindful or mindless? Computers in Human Behavior 28, 1 (2012), 241–250.
  35. Daniel Kimmel. 2023. ChatGPT Therapy Is Good, But It Misses What Makes Us Human. https://www.columbiapsychiatry.org/news/chatgpt-therapy-is-good-but-it-misses-what-makes-us-human. Accessed: 09/11/2023.
  36. Andrew Leonard. 2023. ‘Dr. Google’ meets its match: Dr. ChatGPT. https://www.latimes.com/science/story/2023-09-08/dr-google-meets-its-match-dr-chatgpt Accessed: 09/11/2023.
  37. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197 (2023).
  38. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 (2021).
  39. Privacy protection and self-disclosure across societies: A study of global Twitter users. new media & society 19, 9 (2017), 1476–1497.
  40. Q Vera Liao and Jennifer Wortman Vaughan. 2023. AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap. arXiv preprint arXiv:2306.01941 (2023).
  41. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 4188–4203.
  42. User mental models of cryptocurrency systems-a grounded theory approach. In Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020). 341–358.
  43. Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621 (2022).
  44. Understanding human impressions of artificial intelligence. Preprint]. PsyArXiv. https://doi. org/10.31234/osf. io/5ursp (2021).
  45. EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought. arXiv preprint arXiv:2305.15021 (2023).
  46. Manisha Natarajan and Matthew Gombolay. 2020. Effects of anthropomorphism and accountability on trust in human robot interaction. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. 33–42.
  47. Helen Nissenbaum. 2004. Privacy as contextual integrity. Wash. L. Rev. 79 (2004), 119.
  48. Helen Nissenbaum. 2020. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press.
  49. Donald A Norman. 2014. Some observations on mental models. In Mental models. Psychology Press, 15–22.
  50. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  51. Saurabh Pahune and Manoj Chandrasekharan. 2023. Several categories of Large Language Models (LLMs): A Short Survey. arXiv preprint arXiv:2307.10188 (2023).
  52. Privacy in the Time of Language Models. https://doi.org/10.1145/3539597.3575792
  53. Why doesn’t Jane protect her privacy?. In Privacy Enhancing Technologies: 14th International Symposium, PETS 2014, Amsterdam, The Netherlands, July 16-18, 2014. Proceedings 14. Springer, 244–262.
  54. Considerations on explainable AI and users’ mental models. In CHI 2019 Workshop: Where is the Human? Bridging the Gap Between AI and HCI. Association for Computing Machinery, Inc.
  55. Johnny Saldaña. 2015. The coding manual for qualitative researchers. Sage.
  56. Believable robot characters. AI Magazine 32, 4 (2011), 39–52.
  57. Daniel J Solove. 2021. The myth of the privacy paradox. Geo. Wash. L. Rev. 89 (2021), 1.
  58. Factors mediating disclosure in social network sites. Computers in Human Behavior 27, 1 (2011), 590–598.
  59. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013).
  60. Mikhail Taver. 2023. ChatGPT is Coming to Finance, So Let’s Talk About the Risks and Rewards. https://www.unite.ai/chatgpt-is-coming-to-finance-so-lets-talk-about-the-risks-and-rewards/. Accessed: 09/11/2023.
  61. Franziska Roesner Umar Iqbal, Tadayoshi Kohno. 2023. LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI’s ChatGPT Plugins. (2023).
  62. Measuring {{\{{Real-World}}\}} Accuracies and Biases in Modeling Password Guessability. In 24th USENIX Security Symposium (USENIX Security 15). 463–481.
  63. Ari Ezra Waldman. 2018. Privacy as trust: Information privacy for an information age. Cambridge University Press.
  64. Ari Ezra Waldman. 2021. Industry unbound: The inside story of privacy, data, and corporate power. Cambridge University Press.
  65. ” I regretted the minute I pressed share” a qualitative study of regrets on Facebook. In Proceedings of the seventh symposium on usable privacy and security. 1–16.
  66. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021).
  67. Improved trust in human-robot collaboration with ChatGPT. IEEE Access (2023).
  68. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 (2021).
  69. End user security and privacy concerns with smart homes. In thirteenth symposium on usable privacy and security (SOUPS 2017). 65–80.
  70. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 (2021).
  71. HuatuoGPT, towards Taming Language Model to Be a Doctor. arXiv preprint arXiv:2305.15075 (2023).
  72. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685 (2023).
  73. Privacy antecedents for SNS self-disclosure: The case of Facebook. Computers in Human Behavior 45 (2015), 158–167.
  74. Anthropomorphism: opportunities and challenges in human–robot interaction. International journal of social robotics 7 (2015), 347–360.
  75. Shoshana Zuboff. 2023. The age of surveillance capitalism. In Social Theory Re-Wired. Routledge, 203–213.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Zhiping Zhang (9 papers)
  2. Michelle Jia (1 paper)
  3. Bingsheng Yao (49 papers)
  4. Sauvik Das (13 papers)
  5. Ada Lerner (4 papers)
  6. Dakuo Wang (87 papers)
  7. Tianshi Li (22 papers)
  8. Hao-Ping Lee (3 papers)
Citations (35)
View on Semantic Scholar

Summary

Examining Disclosure Risks in LLM-Based Conversational Agents

The rapid adoption of LLM-based conversational agents (CAs), such as ChatGPT, in sensitive domains like healthcare and finance presents significant privacy challenges. The paper "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents" by Zhang et al. offers a comprehensive examination of how users interact with these systems, revealing the nuanced balance between utility, convenience, and privacy.

User Disclosure Patterns and Privacy Trade-offs

The paper employed both an analysis of real-world ChatGPT conversations and semi-structured interviews with users. The findings indicate that users struggle with the trade-offs between the benefits of CA usage and the associated privacy risks. Users' mental models often underestimate the privacy concerns, potentially due to erroneous assumptions about how these systems work. The research identified that users perceive varying levels of data sensitivity and adopt ad-hoc protective measures, such as falsifying data or providing only general information. However, these actions are sporadic, revealing a lack of coherent privacy strategies.

Misalignment of User Expectations and System Transparency

A significant portion of the research explores users' mental models of LLM-based systems, which influence how they perceive and manage privacy risks. The authors found that users often hold flawed mental models about how these systems utilize input data for response generation and training. This misalignment hinders users' ability to navigate privacy risks effectively and points to a crucial gap between system transparency and user expectations. Such misunderstanding can lead users to disclose more information than they otherwise would if adequately informed.

Institutional and Interdependent Privacy Concerns

The research highlights two main privacy risks: institutional (e.g., data misuse by companies) and interdependent (e.g., sharing data about others). The interdependent risk is particularly complex because users often disclose information about third parties. Interestingly, these disclosures occur in contexts where users might not fully appreciate the ramifications, underscoring the necessity for CAs to better manage and contextualize information sharing.

System Design Recommendations

Based on the insights gathered, the authors emphasize the importance of privacy-aware system design. They advocate for more granular opt-out controls and context-sensitive privacy features that can assist users in making more informed decisions about data sharing. Such enhancements should align with improved user education on how LLMs function to mitigate over- or under-sharing due to inaccurate mental models.

Conclusion and Future Directions

The paper provides essential groundwork in understanding the privacy dynamics faced by users of LLM-based CAs. Although the authors acknowledge the inherent surveillance architecture of modern LLMs, they call for a paradigmatic shift in designing these systems with privacy-by-design principles. They suggest that future research should explore user mental models further and develop user-centric privacy-preserving technologies. Engaging with these unresolved issues will be critical to crafting systems that respect user privacy without compromising the functionality these technologies offer.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube