Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Can Language Models be Instructed to Protect Personal Information? (2310.02224v1)

Published 3 Oct 2023 in cs.CL

Abstract: Large multimodal LLMs have proven transformative in numerous applications. However, these models have been shown to memorize and leak pre-training data, raising serious user privacy and information security concerns. While data leaks should be prevented, it is also crucial to examine the trade-off between the privacy protection and model utility of proposed approaches. In this paper, we introduce PrivQA -- a multimodal benchmark to assess this privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario. We also propose a technique to iteratively self-moderate responses, which significantly improves privacy. However, through a series of red-teaming experiments, we find that adversaries can also easily circumvent these protections with simple jailbreaking methods through textual and/or image inputs. We believe PrivQA has the potential to support the development of new models with improved privacy protections, as well as the adversarial robustness of these protections. We release the entire PrivQA dataset at https://LLM-access-control.github.io/.

PDF HTML Abstract

Evaluating the Efficacy of LLMs in Protecting Personal Information with PrivQA

Introduction

The pervasive use of LLMs raises significant privacy concerns due to their potential to memorize and leak personal information— a challenge that has become increasingly urgent with multimodal models like GPT-4 and Flamingo. This issue not only jeopardizes user privacy but also restricts the integration of LLMs in applications involving sensitive data. To explore the balance between privacy protection and model utility, this paper introduces PrivQA, a multimodal benchmark designed to evaluate the ability of LLMs to adhere to access control instructions intended to protect personal information. Through comprehensive experiments, including red-teaming and self-moderation techniques, the authors shed light on the limitations and potential of instructing LLMs for privacy preservation.

Privacy vs. Utility Trade-off

The core of the paper revolves around the trade-off between privacy protections and the utility of LLMs. Previous approaches to mitigate data leakage have presented an "alignment tax," affecting model performance and operational practicality. The paper critiques these methods for their significant degradation in model performance when applied to more realistic privacy control scenarios. On the other hand, reinforcement learning from human feedback (RLHF) and access control instructions emerge as promising, though not entirely effective, strategies for directing model behavior to protect privacy.

PrivQA Benchmark

PrivQA, a novel benchmark comprising textual and visual question-answering tasks, is introduced to systematically evaluate how well models can protect private information while maintaining their utility. The tasks are designed around two categories: Protected Populations and Protected Information, motivated by the General Data Protection Regulation (GDPR). The benchmark is crafted to avoid the pitfalls of using real-world private data, thus making it reproducible and safe for widespread use without sacrificing user privacy.

Empirical Evaluations and Findings

The evaluation of models on the PrivQA benchmark revealed several critical insights:

  • Access Control Instructions Inefficacy: Initial experiments demonstrated the ineffectiveness of simple access control instructions across different information types, with only marginal success in preventing privacy leaks.
  • Self-Moderation Technique: A proposed self-moderation technique significantly improved the protection scores, showcasing the potential of models to selectively respond to queries based on privacy guidelines. However, this method also revealed biases, particularly against less well-known individuals or those belonging to minority groups.
  • Adversarial Robustness: Through red-teaming experiments, the paper highlighted the vulnerability of LLMs to adversarial attacks aimed at circumventing privacy protections. Text and visual prompt injections were notably effective, raising serious concerns about the models' ability to safeguard against determined adversaries.

Theoretical and Practical Implications

The findings underscore the complexity of instructing LLMs to protect personal information in a way that does not compromise their utility. Theoretical advancements in understanding model behavior, in light of privacy instructions, are imperative for future AI safety research. Pragmatically, the paper advocates for a nuanced approach to model development, where privacy considerations are integrated into the fabric of LLMs, rather than applied as afterthoughts.

Future Directions

Looking forward, the development of LLMs with built-in privacy protection mechanisms (such as the recently released GPT4V by OpenAI) appears promising. However, this paper makes it clear that achieving robust privacy protection is a multifaceted challenge that extends beyond technical fixes. It demands a concerted effort to understand the limitations of current models, innovate on self-moderation techniques, and develop robust defenses against adversarial attacks. Future research should also prioritize mitigating the biases uncovered by this paper, ensuring equitable privacy protection across all user groups.

In summary, this paper presents a comprehensive exploration into the capabilities and limitations of LLMs in protecting personal information, offering valuable insights and laying a foundation for future advancements in the field of AI privacy and security.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Deep learning with differential privacy. In 23rd ACM Conference on Computer and Communications Security (ACM CCS), pp.  308–318, 2016.
  2. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  3. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
  4. Image hijacks: Adversarial images can control generative models at runtime, 2023.
  5. Private empirical risk minimization: Efficient algorithms and tight error bounds. 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pp.  464–473, 2014.
  6. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp.  141–159. IEEE, 2021.
  7. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp.  2280–2292, 2022.
  8. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  1877–1901. Curran Associates, Inc., 2020.
  9. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp.  77–91. PMLR, 2018.
  10. Towards making systems forget with machine unlearning. 2015 IEEE Symposium on Security and Privacy, pp.  463–480, 2015.
  11. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp.  267–284, Santa Clara, CA, August 2019. USENIX Association. ISBN 978-1-939133-06-9.
  12. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp.  2633–2650, 2021.
  13. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
  14. Explore, establish, exploit: Red teaming language models from scratch, 2023.
  15. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023.
  16. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  6491–6506, 2021. doi: 10.18653/v1/2021.emnlp-main.522.
  17. European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. URL https://data.europa.eu/eli/reg/2016/679/oj.
  18. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
  19. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.  3356–3369, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.301.
  20. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  21. Panel: Privacy challenges and opportunities in LLM-Based chatbot applications. Santa Clara, CA, September 2023. USENIX Association.
  22. Llm censorship: A machine learning challenge or a computer security problem? arXiv preprint arXiv:2307.10719, 2023.
  23. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6325–6334. IEEE Computer Society, 2017.
  24. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. IEEE/CVF International Conference on Computer Vision, 2023.
  25. Openclip, July 2021. If you use this software, please cite it as below.
  26. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, July 2023. Association for Computational Linguistics.
  27. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  28. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp.  15696–15707. PMLR, 2023.
  29. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.  252–262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1023.
  30. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
  31. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022.
  32. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
  33. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022.
  34. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1–35, 2021.
  35. Flirt: Feedback loop in-context red teaming, 2023.
  36. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022a.
  37. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.
  38. Memory-based model editing at scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  15817–15831, 2022.
  39. OpenAI. Gpt-4v(ision) system card. 2023a.
  40. OpenAI. Gpt-4 technical report. 2023b.
  41. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  42. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.  3419–3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.225.
  43. How to dp-fy ml: A practical guide to machine learning with differential privacy. arXiv preprint arXiv:2303.00654, 2023.
  44. Modifying memories in transformer models. 2020.
  45. HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.  41–58, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.4.
  46. Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2023.
  47. Naganand Yadati Sanket Shah, Anand Mishra and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In AAAI, 2019.
  48. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  31210–31227. PMLR, 23–29 Jul 2023.
  49. Stochastic gradient descent with differentially private updates. 2013 IEEE Global Conference on Signal and Information Processing, pp.  245–248, 2013.
  50. Machine unlearning: Its need and implementation strategies. In 2021 Thirteenth International Conference on Contemporary Computing (IC3-2021), pp.  241–246, 2021.
  51. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  52. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  53. Wikidata: A free collaborative knowledgebase. Communications of the ACM, 57(10):78–85, sep 2014. ISSN 0001-0782. doi: 10.1145/2629489.
  54. KGA: A general machine unlearning framework based on knowledge gap alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  13264–13276, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.740.
  55. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269, 2023b.
  56. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023.
  57. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302, 2018. doi: 10.1162/tacl_a_00021.
  58. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  2575–2584, 2020.
  59. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2950–2968, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.235.
  60. Recipes for safety in open-domain chatbots, 2021b.
  61. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.  2369–2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259.
  62. Differentially private fine-tuning of language models. In International Conference on Learning Representations (ICLR), 2022.
  63. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3640–3649, 2021.
  64. Universal and transferable adversarial attacks on aligned language models, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yang Chen (535 papers)
  2. Ethan Mendes (4 papers)
  3. Sauvik Das (13 papers)
  4. Wei Xu (536 papers)
  5. Alan Ritter (57 papers)
Citations (25)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. LLM Access Control