Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimal Unbiased Randomizers for Regression with Label Differential Privacy (2312.05659v1)

Published 9 Dec 2023 in cs.LG and cs.CR

Abstract: We propose a new family of label randomizers for training regression models under the constraint of label differential privacy (DP). In particular, we leverage the trade-offs between bias and variance to construct better label randomizers depending on a privately estimated prior distribution over the labels. We demonstrate that these randomizers achieve state-of-the-art privacy-utility trade-offs on several datasets, highlighting the importance of reducing bias when training neural networks with label DP. We also provide theoretical results shedding light on the structural properties of the optimal unbiased randomizers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Deep learning with differential privacy. In CCS, pages 308–318, 2016.
  2. Differentially private simple linear regression. PETS, 2:184–204, 2022.
  3. An equivalence between private classification and online prediction. In FOCS, 2020.
  4. Differentially private learning with margin guarantees. In NeurIPS, 2022.
  5. Private learning and sanitization: Pure vs. approximate differential privacy. ToC, 12(1):1–61, 2016.
  6. Heavy hitters and the structure of local privacy. In PODS, pages 435–447, 2018.
  7. Sample complexity bounds for differentially private learning. In COLT, pages 155–186, 2011.
  8. Data poisoning attacks to local differential privacy protocols. In USENIX Security, pages 947–964, 2021.
  9. Differentially private empirical risk minimization. JMLR, 12(3), 2011.
  10. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022.
  11. Differentially private learning of structured discrete distributions. In NIPS, pages 2566–2574, 2015.
  12. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265–284, 2006.
  13. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211–407, 2014.
  14. Limiting privacy breaches in privacy preserving data mining. In PODS, pages 211–222, 2003.
  15. Label differential privacy via clustering. In AISTATS, pages 7055–7075, 2022.
  16. Deep learning with label differential privacy. NeurIPS, pages 27131–27145, 2021.
  17. Regression with label differential privacy. In ICLR, 2023.
  18. Differentially private aggregation in the shuffle model: Almost central accuracy in almost a single message. In ICML, pages 3692–3701, 2021.
  19. Learning a logistic model from aggregated data. AdKDD Workshop, 2021.
  20. The optimal mechanism in differential privacy. In ISIT, pages 2371–2375, 2014.
  21. Charlie Harrison. Consider a randomized-response-like privacy mechanism, April 2023. github.com/patcg-individual-drafts/ipa/issues/60.
  22. Elad Hazan. Introduction to Online Convex Optimization. MIT Press, 2022.
  23. Exploring the limits of differentially private deep learning with group-wise clipping. In ICLR, 2023.
  24. Differentially private online learning. In COLT, pages 24:1–24:34, 2012.
  25. Toward training at imagenet scale with differential privacy. CoRR, abs/2201.12328, 2022.
  26. What can we learn privately? SIAM J. Comput., 40(3):793–826, 2011.
  27. Extremal mechanisms for local differential privacy. JMLR, 17:17:1–17:51, 2016.
  28. Private convex empirical risk minimization and high-dimensional regression. In COLT, pages 25.1–25.40, 2012.
  29. Label leakage and protection in two-party split learning. In ICLR, 2021.
  30. Antipodes of label differential privacy: PATE and ALIBI. NeurIPS, 34:6934–6945, 2021.
  31. Semi-supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.
  32. Hyperparameter tuning with Rényi differential privacy. In ICLR, 2022.
  33. Scalable private learning with PATE. In ICLR, 2018.
  34. Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data, March 2019. blog.tensorflow.org.
  35. Differentially private regression with Gaussian processes. In AISTATS, 2018.
  36. Stochastic gradient descent with differentially private updates. In GlobalSIP, pages 245–248, 2013.
  37. TAN without a burn: Scaling laws of DP-SGD. In ICML, pages 29937–29949, 2023.
  38. PyTorch Differential Privacy Series Part 1: DP-SGD Algorithm Explained, August 2020. medium.com.
  39. Machine learning with differentially private labels: Mechanisms and frameworks. PETS, 4:332–350, 2022.
  40. Reacting to variations in product demand: An application for conversion rate (CR) prediction in sponsored search. In IEEE BigData, pages 1856–1864, 2018.
  41. Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. In UAI, pages 93–103, 2018.
  42. Stanley L Warner. Randomized response: A survey technique for eliminating evasive answer bias. JASA, 60(309):63–69, 1965.
  43. Answering multi-dimensional analytical queries under local differential privacy. In SIGMOD, pages 159–176, 2019.
  44. Di Wang and Jinhui Xu. On sparse linear regression in the local differential privacy model. In ICML, pages 6628–6637, 2019.
  45. Differentially private fine-tuning of language models. In ICLR, 2022.
  46. Label private deep learning training based on secure multiparty computation and differential privacy. In NeurIPS Workshop Privacy in Machine Learning, 2021.
  47. Functional mechanism: regression analysis under differential privacy. VLDB, 5(11):1364–1375, 2012.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now