Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

$L_q$ Lower Bounds on Distributed Estimation via Fisher Information (2402.01895v2)

Published 2 Feb 2024 in cs.IT, math.IT, math.ST, and stat.TH

Abstract: Van Trees inequality, also known as the Bayesian Cram\'er-Rao lower bound, is a powerful tool for establishing lower bounds for minimax estimation through Fisher information. It easily adapts to different statistical models and often yields tight bounds. Recently, its application has been extended to distributed estimation with privacy and communication constraints where it yields order-wise optimal minimax lower bounds for various parametric tasks under squared $L_2$ loss. However, a widely perceived drawback of the van Trees inequality is that it is limited to squared $L_2$ loss. The goal of this paper is to dispel that perception by introducing a strengthened version of the van Trees inequality that applies to general $L_q$ loss functions by building on the Efroimovich's inequality -- a lesser-known entropic inequality dating back to the 1970s. We then apply the generalized van Trees inequality to lower bound $L_q$ loss in distributed minimax estimation under communication and local differential privacy constraints. This leads to lower bounds for $L_q$ loss that apply to sequentially interactive and blackboard communication protocols. Additionally, we show how the generalized van Trees inequality can be used to obtain \emph{local} and \emph{non-asymptotic} minimax results that capture the hardness of estimating each instance at finite sample sizes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Unified lower bounds for interactive high-dimensional estimation under information constraints. Advances in Neural Information Processing Systems, 36.
  2. Inference under information constraints i: Lower bounds from chi-square contraction. IEEE Transactions on Information Theory, 66(12):7835–7855.
  3. Inference under information constraints ii: Communication constraints and shared randomness. IEEE Transactions on Information Theory, 66(12):7856–7877.
  4. Hadamard response: Estimating distributions privately, efficiently, and with little communication. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1120–1129.
  5. cpsgd: Communication-efficient and differentially-private distributed sgd. In Advances in Neural Information Processing Systems, pages 7564–7575.
  6. A family of bayesian cramér-rao bounds, and consequences for log-concave priors. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 2699–2703. IEEE.
  7. Fast optimal locally private mean estimation via random projections. In Neural Information Processing Systems, volume 37.
  8. Optimal algorithms for mean estimation under local differential privacy. In International Conference on Machine Learning, pages 1046–1056. PMLR.
  9. Fisher information under local differential privacy. IEEE Journal on Selected Areas in Information Theory, 1(3):645–659.
  10. Fisher information for distributed estimation under a blackboard communication protocol. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 2704–2708. IEEE.
  11. Lower bounds for learning distributions under communication constraints via fisher information.
  12. Communication lower bounds for statistical estimation problems via a distributed data processing inequality. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 1011–1020.
  13. Breaking the communication-privacy-accuracy trilemma. Advances in Neural Information Processing Systems, 33.
  14. Breaking the dimension dependence in sparse distribution estimation under communication constraints. In Conference on Learning Theory, pages 1028–1059. PMLR.
  15. Pointwise bounds for distribution estimation under communication constraints. Advances in Neural Information Processing Systems, 34.
  16. Cramér, H. (1999). Mathematical methods of statistics, volume 43. Princeton university press.
  17. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 429–438. IEEE.
  18. Lower bounds for locally private estimation via communication complexity. In Conference on Learning Theory, pages 1161–1191. PMLR.
  19. The right complexity measure in locally private estimation: It is not the fisher information. arXiv preprint arXiv:1806.05756.
  20. Efroimovich, S. Y. (1980). Information contained in a sequence of observations. Problems in Information Transmission, 15:24–39.
  21. Private frequency estimation via projective geometry. In International Conference on Machine Learning, pages 6418–6433. PMLR.
  22. Lossless compression of efficient private local randomizers. In International Conference on Machine Learning, pages 3208–3219. PMLR.
  23. vqsgd: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, pages 2197–2205. PMLR.
  24. On communication cost of distributed statistical estimation and dimensionality. In Advances in Neural Information Processing Systems, pages 2726–2734.
  25. Applications of the van trees inequality: a bayesian cramér-rao bound. Bernoulli, 1(1-2):59–79.
  26. Hájek, J. (1961). Some extensions of the wald-wolfowitz-noether theorem. The Annals of Mathematical Statistics, pages 506–523.
  27. Distributed statistical estimation of high-dimensional and nonparametric distributions. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 506–510. IEEE.
  28. Geometric lower bounds for distributed parameter estimation under communication constraints. In Conference On Learning Theory, pages 3163–3188. PMLR.
  29. Exact optimality of communication-privacy-utility tradeoffs in distributed mean estimation. Advances in Neural Information Processing Systems, 37.
  30. Discrete distribution estimation under local privacy. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 2436–2444, New York, New York, USA.
  31. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977.
  32. Kushilevitz, E. (1997). Communication complexity. In Advances in Computers, volume 44, pages 331–360. Elsevier.
  33. Lee, K.-Y. (2022). New Information Inequalities with Applications to Statistics. PhD thesis, UC Berkeley.
  34. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR.
  35. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963.
  36. Rao, C. R. (1992). Information and the accuracy attainable in the estimation of statistical parameters. In Breakthroughs in Statistics: Foundations and basic theory, pages 235–247. Springer.
  37. Optimal compression of locally differentially private mechanisms. In International Conference on Artificial Intelligence and Statistics, pages 7680–7723. PMLR.
  38. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3329–3337. JMLR.org.
  39. Tsybakov, A. B. (2004). Introduction to nonparametric estimation, 2009. URL https://doi. org/10.1007/b13794. Revised and extended from the, 9(10).
  40. Van der Vaart, A. W. (2000). Asymptotic statistics, volume 3. Cambridge university press.
  41. Van Trees, H. L. (2004). Detection, estimation, and modulation theory, part I: detection, estimation, and linear modulation theory. John Wiley & Sons.
  42. Locally differentially private protocols for frequency estimation. In 26th {normal-{\{{USENIX}normal-}\}} Security Symposium ({normal-{\{{USENIX}normal-}\}} Security 17), pages 729–745.
  43. Optimal schemes for discrete distribution estimation under local differential privacy. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 759–763.
  44. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. In Advances in Neural Information Processing Systems, pages 2328–2336.
  45. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems, 25.
  46. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 21st ACM Conference on Computer and Communications Security, Scottsdale, Arizona.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now