Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy (2405.13453v2)

Published 22 May 2024 in cs.LG and cs.CR

Abstract: Privacy protection of users' entire contribution of samples is important in distributed systems. The most effective approach is the two-stage scheme, which finds a small interval first and then gets a refined estimate by clipping samples into the interval. However, the clipping operation induces bias, which is serious if the sample distribution is heavy-tailed. Besides, users with large local sample sizes can make the sensitivity much larger, thus the method is not suitable for imbalanced users. Motivated by these challenges, we propose a Huber loss minimization approach to mean estimation under user-level differential privacy. The connecting points of Huber loss can be adaptively adjusted to deal with imbalanced users. Moreover, it avoids the clipping operation, thus significantly reducing the bias compared with the two-stage approach. We provide a theoretical analysis of our approach, which gives the noise strength needed for privacy protection, as well as the bound of mean squared error. The result shows that the new method is much less sensitive to the imbalance of user-wise sample sizes and the tail of sample distributions. Finally, we perform numerical experiments to validate our theoretical analysis.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265–284. Springer, 2006.
  2. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 51–60. IEEE, 2010.
  3. What can we learn privately? SIAM Journal on Computing, 40(3):793–826, 2011.
  4. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  5. Instance-optimal mean estimation under differential privacy. Advances in Neural Information Processing Systems, 34:25993–26004, 2021.
  6. Optimal algorithms for mean estimation under local differential privacy. In International Conference on Machine Learning, pages 1046–1056. PMLR, 2022.
  7. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 1406–1417. 2022.
  8. Collecting and analyzing data from smart device users with local differential privacy. arXiv preprint arXiv:1606.05053, 2016.
  9. Personalized privacy-preserving frequent itemset mining using randomized response. The Scientific World Journal, 2014, 2014.
  10. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
  11. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464–473. IEEE, 2014.
  12. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32, 2019.
  13. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 439–449. 2020.
  14. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning, pages 393–403. PMLR, 2021.
  15. Shuffle private stochastic convex optimization. In International Conference on Learning Representations. 2022.
  16. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310–1321. 2015.
  17. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318. 2016.
  18. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963, 2017.
  19. Federated linear contextual bandits with user-level differential privacy. In International Conference on Machine Learning, pages 14060–14095. PMLR, 2023.
  20. Federated recommendation system via differential privacy. In 2020 IEEE international symposium on information theory (ISIT), pages 2592–2597. IEEE, 2020.
  21. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 627–636. 2009.
  22. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557, 2017.
  23. A general approach to adding differential privacy to iterative training procedures. arXiv preprint arXiv:1812.06210, 2018.
  24. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security, 15:3454–3469, 2020.
  25. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  26. Differentially private federated learning: A systematic review. arXiv preprint arXiv:2405.08299, 2024.
  27. Learning discrete distributions: user vs item-level privacy. Advances in Neural Information Processing Systems, 33:20965–20976, 2020.
  28. Learning with user-level privacy. Advances in Neural Information Processing Systems, 34:12466–12479, 2021.
  29. User-level differentially private learning via correlated sampling. Advances in Neural Information Processing Systems, 34:20172–20184, 2021.
  30. User-level differential privacy with few examples per user. Advances in Neural Information Processing Systems, 36, 2023.
  31. Hadamard matrix analysis and synthesis: with applications to communications and signal/image processing, vol. 383. Springer Science & Business Media, 2012.
  32. Fair resource allocation in federated learning. In International Conference on Learning Representations. 2019.
  33. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59–71, 2020.
  34. Adaptive client clustering for efficient federated learning over non-iid and imbalanced data. IEEE Transactions on Big Data, 2022.
  35. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pages 3964–3975. PMLR, 2021.
  36. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827–5837. PMLR, 2019.
  37. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383–15393, 2020.
  38. Improved convergence in high probability of clipped gradient methods with heavy tailed noise. Advances in Neural Information Processing Systems, 36:24191–24222, 2023.
  39. Huber, P. J. Robust statistics, vol. 523. John Wiley & Sons, 2004.
  40. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 75–84. 2007.
  41. Poisoning-assisted property inference attack against federated learning. IEEE Transactions on Dependable and Secure Computing, 2022.
  42. Robust and differentially private mean estimation. Advances in neural information processing systems, 34:3887–3901, 2021.
  43. Label robust and differentially private linear regression: Computational and statistical efficiency. Advances in Neural Information Processing Systems, 36, 2023.
  44. Towards the robustness of differentially private federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pages 19911–19919. 2024.
  45. On robustness and local differential privacy. The Annals of Statistics, 51(2):717–737, 2023.
  46. Smith, A. Privacy-preserving statistical estimation with optimal convergence rates. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 813–822. 2011.
  47. Private mean estimation of heavy-tailed distributions. In Conference on Learning Theory, pages 2204–2235. PMLR, 2020.
  48. User-level private stochastic convex optimization with optimal rates. In International Conference on Machine Learning, pages 1838–1851. PMLR, 2023.
  49. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In International Conference on Artificial Intelligence and Statistics, pages 4240–4248. PMLR, 2024.
  50. Mean estimation with user-level privacy under data heterogeneity. Advances in Neural Information Processing Systems, 35:29139–29151, 2022.
  51. Stability is stable: Connections between replicability, privacy, and adaptive generalization. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 520–527. 2023.
  52. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pages 371–380. 2009.
  53. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory, pages 1167–1246. PMLR, 2022.
  54. Average-case averages: Private algorithms for smooth sensitivity and mean estimation. Advances in Neural Information Processing Systems, 32, 2019.
  55. From robustness to privacy and back. In International Conference on Machine Learning, pages 1121–1146. PMLR, 2023.
  56. Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms. Advances in neural information processing systems, 33:14106–14117, 2020.
  57. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742–864, 2019.
  58. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning, pages 999–1008. PMLR, 2017.
  59. Algorithmic high-dimensional robust statistics. Cambridge university press, 2023.
  60. Adaptive m-estimation in nonparametric regression. The annals of statistics, pages 1712–1728, 1990.
  61. Robust nonparametric regression under poisoning attack. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17007–17015. 2024.
  62. Robust wavelet denoising. IEEE transactions on signal processing, 49(6):1146–1152, 2001.
  63. A huber loss minimization approach to byzantine robust federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 21806–21814. 2024.
  64. User-level privacy-preserving federated learning: Analysis and performance optimization. IEEE Transactions on Mobile Computing, 21(9):3388–3401, 2021.
  65. On the point for which the sum of the distances to n given points is minimum. Annals of Operations Research, 167:7–41, 2009.
  66. Weiszfeld’s method: Old and new results. Journal of Optimization Theory and Applications, 164:1–40, 2015.
  67. IPUMS USA: Version 15.0 [dataset], 2024.
  68. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning, 8(1-2):1–230, 2015.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now