Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Near-Optimal differentially private low-rank trace regression with guaranteed private initialization (2403.15999v1)

Published 24 Mar 2024 in stat.ML, cs.LG, and cs.CR

Abstract: We study differentially private (DP) estimation of a rank-$r$ matrix $M \in \mathbb{R}{d_1\times d_2}$ under the trace regression model with Gaussian measurement matrices. Theoretically, the sensitivity of non-private spectral initialization is precisely characterized, and the differential-privacy-constrained minimax lower bound for estimating $M$ under the Schatten-$q$ norm is established. Methodologically, the paper introduces a computationally efficient algorithm for DP-initialization with a sample size of $n \geq \widetilde O (r2 (d_1\vee d_2))$. Under certain regularity conditions, the DP-initialization falls within a local ball surrounding $M$. We also propose a differentially private algorithm for estimating $M$ based on Riemannian optimization (DP-RGrad), which achieves a near-optimal convergence rate with the DP-initialization and sample size of $n \geq \widetilde O(r (d_1 + d_2))$. Finally, the paper discusses the non-trivial gap between the minimax lower bound and the upper bound of low-rank matrix estimation under the trace regression model. It is shown that the estimator given by DP-RGrad attains the optimal convergence rate in a weaker notion of differential privacy. Our powerful technique for analyzing the sensitivity of initialization requires no eigengap condition between $r$ non-zero singular values.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. The modernization of statistical disclosure limitation at the us census bureau. US Census Bureau.
  2. Optimization Algorithms on Matrix Manifolds. Princeton, NJ: Princeton University Press.
  3. Differentially private assouad, fano, and le cam. In Algorithmic Learning Theory, pp.  48–78. PMLR.
  4. Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA Journal of Numerical Analysis 22(3), 359–390.
  5. Apple Differential Privacy Team (2017). Learning with privacy at scale.
  6. Practical privacy: The sulq framework. Proceedings of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 128–138.
  7. Fast, sample-efficient, affine-invariant private mean and covariance estimation for subgaussian distributions. In G. Neu and L. Rosasco (Eds.), Proceedings of Thirty Sixth Conference on Learning Theory, Volume 195 of Proceedings of Machine Learning Research, pp.  5578–5579. PMLR.
  8. Burer, S. and R. D. Monteiro (2003). A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming 95(2), 329–357.
  9. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics 49, 2825–2850.
  10. Score attack: A lower bound technique for optimal differentially private learning. arXiv preprint arXiv:2303.07152.
  11. Optimal differentially private pca and estimation for spiked covariance matrices. arXiv preprint arXiv:2401.03820.
  12. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory 57(4), 2342–2359.
  13. Decoding by linear programming. IEEE transactions on information theory 51(12), 4203–4215.
  14. Near-optimal differentially private principal components. Advances in Neural Information Processing Systems 25.
  15. Non-convex projected gradient descent for generalized low-rank tensor regression. The Journal of Machine Learning Research 20(1), 172–208.
  16. Chen, Y. and M. J. Wainwright (2015). Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees.
  17. Private alternating least squares: Practical private matrix completion with tighter rates. pp.  1877–1887.
  18. Calibrating noise to sensitivity in private data analysis. Theory of Cryptography Conference, 265–284.
  19. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9(3–4), 211–407.
  20. Analyze gauss: optimal bounds for privacy-preserving principal component analysis. Proceedings of the forty-sixth annual ACM symposium on Theory of computing, 11–20.
  21. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications 20(2), 303–353.
  22. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, CCS ’14, New York, NY, USA, pp.  1054–1067. Association for Computing Machinery.
  23. On low-rank trace regression under general sampling distribution. Journal of Machine Learning Research 23(321), 1–49.
  24. Differentially private matrix completion revisited. pp.  2215–2224.
  25. Privately learning high-dimensional distributions. In Conference on Learning Theory, pp.  1853–1902. PMLR.
  26. Private mean estimation of heavy-tailed distributions. In J. Abernethy and S. Agarwal (Eds.), Proceedings of Thirty Third Conference on Learning Theory, Volume 125 of Proceedings of Machine Learning Research, pp.  2204–2235. PMLR.
  27. Koltchinskii, V. (2011). Von neumann entropy penalization and low-rank matrix estimation.
  28. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion.
  29. Optimal estimation of low rank density matrices. J. Mach. Learn. Res. 16(53), 1757–1792.
  30. A pretty fast algorithm for adaptive private mean estimation. In G. Neu and L. Rosasco (Eds.), Proceedings of Thirty Sixth Conference on Learning Theory, Volume 195 of Proceedings of Machine Learning Research, pp.  2511–2551. PMLR.
  31. Dp-pca: Statistically optimal and differentially private pca. Advances in Neural Information Processing Systems 35, 29929–29943.
  32. Fast differentially private matrix factorization. In Proceedings of the 9th ACM Conference on Recommender Systems, pp.  171–178.
  33. Luo, Y. and A. R. Zhang (2022). Tensor-on-tensor regression: Riemannian optimization, over-parameterization, statistical-computational gap, and their interplay. arXiv preprint arXiv:2206.08756.
  34. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  627–636.
  35. Negahban, S. and M. J. Wainwright (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling.
  36. Pajor, A. (1998). Metric entropy of the grassmann manifold. Convex Geometric Analysis 34(181-188), 0942–46013.
  37. Rohde, A. and A. B. Tsybakov (2011). Estimation of high-dimensional low-rank matrices.
  38. Computationally efficient and statistically optimal robust high-dimensional linear regression. arXiv preprint arXiv:2305.06199.
  39. Tropp, J. A. et al. (2015). An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning 8(1-2), 1–230.
  40. Vadhan, S. (2017). The complexity of differential privacy. Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich, 347–450.
  41. Vandereycken, B. (2013). Low-rank matrix completion by riemannian optimization. SIAM Journal on Optimization 23(2), 1214–1236.
  42. Vershynin, R. (2015). Estimation in high dimensions: a geometric perspective. In Sampling Theory, a Renaissance: Compressive Sensing and Other Developments, pp.  3–66. Springer.
  43. Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science, Volume 47. Cambridge university press.
  44. Differentially private matrix completion through low-rank matrix factorization.  206, 5731–5748.
  45. Wang, Y.-X. (2018). Revisiting differentially private linear regression: optimal and adaptive prediction and estimation in unbounded domain. In UAI 2018.
  46. Guarantees of riemannian optimization for low rank matrix recovery. SIAM Journal on Matrix Analysis and Applications 37(3), 1198–1222.
  47. Xia, D. (2021). Normal approximation and confidence region of singular subspaces. Electronic Journal of Statistics 15(2), 3798–3851.
  48. A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets