Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 71 tok/s
Gemini 2.5 Pro 44 tok/s Pro
GPT-5 Medium 22 tok/s Pro
GPT-5 High 25 tok/s Pro
GPT-4o 81 tok/s Pro
Kimi K2 172 tok/s Pro
GPT OSS 120B 434 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Evaluating the Usability of Differential Privacy Tools with Data Practitioners (2309.13506v3)

Published 24 Sep 2023 in cs.HC and cs.CR

Abstract: Differential privacy (DP) has become the gold standard in privacy-preserving data analytics, but implementing it in real-world datasets and systems remains challenging. Recently developed DP tools aim to make DP implementation easier, but limited research has investigated these DP tools' usability. Through a usability study with 24 US data practitioners with varying prior DP knowledge, we evaluated the usability of four Python-based open-source DP tools: DiffPrivLib, Tumult Analytics, PipelineDP, and OpenDP. Our results suggest that using DP tools in this study may help DP novices better understand DP; that Application Programming Interface (API) design and documentation are vital for successful DP implementation; and that user satisfaction correlates with how well participants completed study tasks with these DP tools. We provide evidence-based recommendations to improve DP tools' usability to broaden DP adoption.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. You are not your developer, either: A research agenda for usable security and privacy research beyond end users. 2016 IEEE Cybersecurity Development (SecDev), pages 3–8, 2016.
  2. On the usability of hadoop mapreduce, apache spark & apache flink for data science. In 2017 IEEE International Conference on Big Data (Big Data), pages 303–310. IEEE, 2017.
  3. What happened to remote usability testing? an empirical study of three methods. CHI ’07, page 1405–1414, New York, NY, USA, 2007. Association for Computing Machinery.
  4. Apple. Apple: Differential Privacy Overview, 2023. https://www.apple.com/privacy/docs/Differential_Privacy_Overview.pdf.
  5. Casual users and rational choices within differential privacy. In Proceedings of the 2024 IEEE Symposium on Security and Privacy, pages 88–88, 2024.
  6. Tumult analytics: a robust, easy-to-use, scalable, and expressive framework for differential privacy. arXiv preprint arXiv:2212.04133, 2022.
  7. Nigel Bevan. Practical issues in usability measurement. Interactions, 13(6):42–43, 2006.
  8. John Brooke. Sus: a “quick and dirty’usability. Usability evaluation in industry, 189(3), 1996.
  9. Towards understanding differential privacy: When do people trust randomized response technique? In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3833–3837, 2017.
  10. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th {normal-{\{{USENIX}normal-}\}} Security Symposium ({normal-{\{{USENIX}normal-}\}} Security 19), pages 267–284, 2019.
  11. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  12. Widespread underestimation of sensitivity in differentially private libraries and how to fix it. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 471–484, 2022.
  13. Lynne Cooke. Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53(3):202–215, 2010.
  14. " i need a better description": An investigation into user expectations for differential privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3037–3052, 2021.
  15. Damien Desfontaines. Lowering the cost of anonymization. PhD thesis, ETH Zurich, 2020.
  16. DiffPrivLib, 2023. https://github.com/IBM/differential-privacy-library.
  17. DP Creator, 2023. https://github.com/opendp/dpcreator.
  18. A practical guide to usability testing, 1999.
  19. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006.
  20. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  21. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods, 5(1):80–92, 2006.
  22. Psi ({{\{{\\\backslash\Psi}}\}}): a private data sharing interface. arXiv preprint arXiv:1609.04340, 2016.
  23. Lessons learned: Surveying the practicality of differential privacy in the industry. arXiv preprint arXiv:2211.03898, 2022.
  24. Google. Google: Differentially private heatmaps, 2023. https://blog.research.google/2023/04/differentially-private-heatmaps.html.
  25. Google’s differential privacy libraries, 2023.
  26. Douglas B Grisaffe. Questions about the ultimate question: conceptual considerations in evaluating reichheld’s net promoter score (nps). Journal of Consumer Satisfaction, Dissatisfaction and Complaining Behavior, 20:36, 2007.
  27. Precision-based attacks and interval refining: how to break, then fix, differential privacy on finite computers. arXiv preprint arXiv:2207.13793, 2022.
  28. Revisiting membership inference under realistic assumptions. arXiv preprint arXiv:2005.10881, 2020.
  29. Are we there yet? timing and floating-point attacks on differential privacy systems. In 2022 IEEE Symposium on Security and Privacy (SP), pages 473–488. IEEE, 2022.
  30. Chorus: a programming framework for building scalable differential privacy mechanisms. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 535–551. IEEE, 2020.
  31. Guidelines for implementing and auditing differentially private systems. arXiv preprint arXiv:2002.04049, 2020.
  32. Replication: The effect of differential privacy communication on german users’ comprehension and data sharing attitudes. In Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022), pages 117–134, 2022.
  33. Coconut: An ide plugin for developing privacy-friendly apps. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(4):1–35, 2018.
  34. Wired Magazine. T-Mobile’s $150 Million Security Plan Isn’t Cutting It, 2023. https://www.wired.com/story/tmobile-data-breach-again/.
  35. Frank D McSherry. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pages 19–30, 2009.
  36. Comparative evaluation of big-data systems on scientific image analytics workloads. arXiv preprint arXiv:1612.02485, 2016.
  37. Microsoft. Microsoft AI: Differential Privacy, 2023. https://www.microsoft.com/en-us/ai/ai-lab-differential-privacy.
  38. Ilya Mironov. On significance of the least significant bits for differential privacy. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 650–661, 2012.
  39. Usable differential privacy: A case study with psi. arXiv preprint arXiv:1809.04103, 2018.
  40. Why do developers get password storage wrong? a qualitative usability study. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 311–328, 2017.
  41. Visualizing privacy-utility trade-offs in differentially private data releases. arXiv preprint arXiv:2201.05964, 2022.
  42. Jakob Nielsen. Usability engineering. Morgan Kaufmann, 1994.
  43. Jakob Nielsen. Usability metrics: Tracking interface improvements. IEEE software, 13(6):1–2, 1996.
  44. OpenDP, 2023. https://github.com/opendp/opendp.
  45. Nicolas Papernot. Machine learning at scale with differential privacy in {{\{{TensorFlow}}\}}. In 2019 {normal-{\{{USENIX}normal-}\}} Conference on Privacy Engineering Practice and Respect ({normal-{\{{PEPR}normal-}\}} 19), 2019.
  46. PipelineDP, 2023.
  47. Associated Press. Wawa agrees to payment, security changes for ’19 data breach, 2022. https://apnews.com/article/technology-pennsylvania-malware-attorney-generals-office-0ebedd8dce8bf0e21833f52944a48b56.
  48. Privacy on Beam, 2023.
  49. Chorus Repository, 2023.
  50. Don’t look at the data! how differential privacy reconfigures the practices of data science. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–19, 2023.
  51. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18. IEEE, 2017.
  52. Benchmarking differential privacy python tools. https://github.com/dsaidgovsg/benchmarking-differential-privacy-tools, 2023.
  53. U.S. Census Bureau. Why the Census Bureau Chose Differential Privacy, 2023. https://www.census.gov/library/publications/2023/decennial/c2020br-03.html.
  54. Differentially private sql with bounded user contribution. Proceedings on Privacy Enhancing Technologies, 2:230–250, 2020.
  55. Towards effective differential privacy communication for users’ data sharing decision and comprehension. In 2020 IEEE Symposium on Security and Privacy (SP), pages 392–410. IEEE, 2020.
  56. Using illustrations to communicate differential privacy trust models: An investigation of users’ comprehension, perception, and data sharing decision. arXiv preprint arXiv:2202.10014, 2022.
  57. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298, 2021.
  58. ZetaSQL differential privacy extension, 2023.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 post and received 0 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube