Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation (2402.13531v1)

Published 21 Feb 2024 in cs.LG and cs.CR

Abstract: We provide an improved analysis of standard differentially private gradient descent for linear regression under the squared error loss. Under modest assumptions on the input, we characterize the distribution of the iterate at each time step. Our analysis leads to new results on the algorithm's accuracy: for a proper fixed choice of hyperparameters, the sample complexity depends only linearly on the dimension of the data. This matches the dimension-dependence of the (non-private) ordinary least squares estimator as well as that of recent private algorithms that rely on sophisticated adaptive gradient-clipping schemes (Varshney et al., 2022; Liu et al., 2023). Our analysis of the iterates' distribution also allows us to construct confidence intervals for the empirical optimizer which adapt automatically to the variance of the algorithm on a particular data set. We validate our theorems through experiments on synthetic data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Deep learning with differential privacy. In Proc. of the 2016 ACM SIGSAC Conf. on Computer and Communications Security (CCS’16), pp.  308–318, 2016.
  2. Differentially private simple linear regression. Proceedings on Privacy Enhancing Technologies, 2022.
  3. Easy differentially private linear regression. arXiv preprint arXiv:2208.07353, 2022.
  4. Differentially private inference via noisy optimization. arXiv preprint arXiv:2103.11003, 2021.
  5. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proc. of the 2014 IEEE 55th Annual Symp. on Foundations of Computer Science (FOCS), pp.  464–473, 2014.
  6. Bootstrap inference and differential privacy: Standard errors for free. 2018.
  7. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pp.  635–658. Springer, 2016.
  8. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics, 49(5):2825–2850, 2021.
  9. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(Mar):1069–1109, 2011.
  10. Statistical inference for model parameters in stochastic gradient descent. Annals of Statistics, 48(1):251–273, 2020.
  11. Unbiased statistical estimation and valid confidence intervals under differential privacy. arXiv preprint arXiv:2110.14465, 2021.
  12. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742–864, 2019.
  13. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp.  371–380, 2009.
  14. Concentrated differential privacy. arXiv preprint arXiv:1603.01887, 2016.
  15. Calibrating noise to sensitivity in private data analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC), pp.  265–284, 2006. URL http://dx.doi.org/10.1007/11681878_14.
  16. Analyze gauss: optimal bounds for privacy-preserving principal component analysis. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pp.  11–20, 2014.
  17. Parametric bootstrap for differentially private confidence intervals. In International Conference on Artificial Intelligence and Statistics, pp.  1598–1618. PMLR, 2022.
  18. On the theory and practice of privacy-preserving bayesian data analysis. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pp.  192–201, 2016.
  19. An analysis of random design linear regression. arXiv preprint arXiv:1106.2363, 6, 2011.
  20. Privately learning high-dimensional distributions. In Conference on Learning Theory, pp.  1853–1902. PMLR, 2019.
  21. New lower bounds for private estimation and a generalized fingerprinting lemma. Advances in Neural Information Processing Systems, 35:24405–24418, 2022.
  22. What can we learn privately? In 49th Annual IEEE Symp. on Foundations of Computer Science (FOCS), pp.  531–540, 2008.
  23. Differentially private linear regression via medians. 2022.
  24. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201.12328, 2022.
  25. Differentially private deep learning can be effective with self-supervised models. https://differentialprivacy.org/dp-fine-tuning/, 2022. URL https://differentialprivacy.org/dp-fine-tuning/.
  26. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory, pp.  1167–1246. PMLR, 2022.
  27. Near optimal private and robust linear regression. arXiv preprint arXiv:2301.13273, 2023.
  28. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), pp.  94–103. IEEE, 2007.
  29. Differentially private regression with unbounded covariates. In International Conference on Artificial Intelligence and Statistics, pp.  3242–3273. PMLR, 2022.
  30. Narayanan, S. Better and simpler fingerprinting lower bounds for differentially private estimation. 2023.
  31. Training private models that know what they don’t know. arXiv preprint arXiv:2305.18393, 2023.
  32. Analyzing the differentially private theil-sen estimator for simple linear regression. arXiv preprint arXiv:2207.13289, 2022.
  33. Sheffet, O. Differentially private ordinary least squares. In International Conference on Machine Learning, pp. 3105–3114. PMLR, 2017.
  34. Sheffet, O. Old techniques in differentially private linear regression. In Algorithmic Learning Theory, pp.  789–827. PMLR, 2019.
  35. Recycling scraps: Improving private learning by leveraging intermediate checkpoints. arXiv preprint arXiv:2210.01864, 2022.
  36. Characterizing private clipped gradient descent on convex generalized linear problems. arXiv preprint arXiv:2006.06783, 2020.
  37. Improved differentially private regression via gradient boosting. arXiv preprint arXiv:2303.03451, 2023.
  38. (nearly) optimal private linear regression via adaptive clipping. arXiv preprint arXiv:2207.04686, 2022.
  39. Differential privacy for clinical trial data: Preliminary evaluations. In 2009 IEEE International Conference on Data Mining Workshops, pp.  138–143. IEEE, 2009.
  40. Differentially private confidence intervals for empirical risk minimization. Journal of Privacy and Confidentiality, 9(1), 2019.
  41. Wang, Y.-X. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. arXiv preprint arXiv:1803.02596, 2018.
  42. Differentially private bootstrap: New privacy analysis and inference strategies. arXiv preprint arXiv:2210.06140, 2022.
  43. Federated learning of gboard language models with differential privacy. arXiv preprint arXiv:2305.18465, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now