Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Differentially Private Distributed Estimation and Learning (2306.15865v5)

Published 28 Jun 2023 in cs.LG, cs.SI, cs.SY, eess.SY, math.ST, stat.AP, stat.ML, and stat.TH

Abstract: We study distributed estimation and learning problems in a networked environment where agents exchange information to estimate unknown statistical properties of random variables from their privately observed samples. The agents can collectively estimate the unknown quantities by exchanging information about their private observations, but they also face privacy risks. Our novel algorithms extend the existing distributed estimation literature and enable the participating agents to estimate a complete sufficient statistic from private signals acquired offline or online over time and to preserve the privacy of their signals and network neighborhoods. This is achieved through linear aggregation schemes with adjusted randomization schemes that add noise to the exchanged estimates subject to differential privacy (DP) constraints, both in an offline and online manner. We provide convergence rate analysis and tight finite-time convergence bounds. We show that the noise that minimizes the convergence time to the best estimates is the Laplace noise, with parameters corresponding to each agent's sensitivity to their signal and network characteristics. Our algorithms are amenable to dynamic topologies and balancing privacy and accuracy trade-offs. Finally, to supplement and validate our theoretical results, we run experiments on real-world data from the US Power Grid Network and electric consumption data from German Households to estimate the average power consumption of power stations and households under all privacy regimes and show that our method outperforms existing first-order, privacy-aware, distributed optimization methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. Personalized differential privacy for ridge regression. arXiv preprint arXiv:2401.17127.
  2. Private Weighted Sum Aggregation. IEEE Transactions on Control of Network Systems, 9(1):219–230.
  3. Towards Private Data-driven Control. In IEEE Conference on Decision and Control (CDC 2020), pages 5449–5456. IEEE.
  4. Apple Differential Privacy Team (2017). Learning with privacy at scale. https://machinelearning.apple.com/research/learning-with-privacy-at-scale. Accessed: 2023-05-18.
  5. Joint Estimation and Localization in Sensor Networks. IEEE Conference on Decision and Control (CDC 2014), pages 6875–6882.
  6. Distributed Algorithms for Stochastic Source Seeking with Mobile Robot Networks. Journal of Dynamic Systems, Measurement, and Control.
  7. Aumann, R. J. (1976). Agreeing to Disagree. The Annals of Statistics, pages 1236–1239.
  8. A face is exposed for AOL searcher no. 4417749. New York Times, 9(2008):8.
  9. Mathematical Statistics: Basic Ideas and Selected Topics, volume I. CRC Press.
  10. Federated Learning and Privacy: Building Privacy-preserving Systems for Machine Learning and Data Science on Decentralized Data. Queue, 19(5):87–114.
  11. Asymptotic Agreement in Distributed Estimation. IEEE Transactions on Automatic Control, 27(3):650–655.
  12. Fastest Mixing Markov Chain on a Graph. SIAM Review, 46(4):667–689.
  13. Gossip Algorithms: Design, Analysis and Applications. In IEEE Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM 2005), volume 3, pages 1653–1664. IEEE.
  14. Distributed control of robotic networks: a mathematical approach to motion coordination algorithms, volume 27. Princeton University Press.
  15. Differentially Private Histograms under Continual Observation: Streaming Selection into the Unknown. In International Conference on Artificial Intelligence and Statistics (AISTATS 2022), pages 2397–2419. PMLR.
  16. Statistical Inference, volume 2. Duxbury Pacific Grove, CA.
  17. Decentralized Detection in Sensor Networks. IEEE Transactions on Signal Processing, 51(2):407–416.
  18. Chatterjee, S. (2023). Spectral Gap of Nonreversible Markov Chains. arXiv preprint arXiv:2310.10876.
  19. Chazelle, B. (2011). The Total s𝑠sitalic_s-energy of a Multiagent System. SIAM Journal on Control and Optimization, 49(4):1680–1706.
  20. Geographic Gossip: Efficient Averaging for Sensor Networks. IEEE Transactions on Signal Processing, 56(3):1205–1216.
  21. Dwork, C. (2011). A Firm Foundation for Private Data Analysis. Communications of the ACM, 54(1):86–95.
  22. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407.
  23. Erlingsson, U. (2014). Learning statistics with privacy, aided by the flip of a coin. https://ai.googleblog.com/2014/10/learning-statistics-with-privacy-aided.html. Accessed: 2023-05-18.
  24. We can’t Disagree Forever. Journal of Economic Theory, 28(1):192–200.
  25. Privacy Enhanced Data Communication Protocol for Wireless Body Area Network. In International Conference on Advanced Computing and Communication Systems (ICACCS 2017), pages 1–5. IEEE.
  26. Guevara, M. (2019). Enabling developers and organizations to use differential privacy. https://developers.googleblog.com/2019/09/enabling-developers-and-organizations.html. Accessed: 2023-05-18.
  27. Differential Privacy Techniques for Cyber Physical Systems: a Survey. IEEE Communications Surveys & Tutorials, 22(1):746–789.
  28. Jackson, M. O. (2008). Social and Economic Networks. Princeton University Press, Princeton, NJ.
  29. Coordination of Groups of Mobile Autonomous Agents using Nearest Neighbor Rules. IEEE Transactions on Automatic Control, 48(6):988–1001.
  30. Conservative or Liberal? personalized Differential Privacy. In IEEE International Conference on Data Engineering (ICDE 2015), pages 1023–1034. IEEE.
  31. Secure, Privacy-preserving and Federated Machine Learning in Medical Imaging. Nature Machine Intelligence, 2(6):305–311.
  32. Distributed parameter estimation in sensor networks: Nonlinear observation models and imperfect communication. IEEE Transactions on Information Theory, 58, no. 6, pp. 3575–3605.
  33. Random Geometric Graphs as Model of Wireless Sensor Networks. In International Conference on Computer and Automation Engineering (ICCAE 2010), volume 4, pages 103–107. IEEE.
  34. The internet of federated things (IoFT). IEEE Access, 9:156071–156113.
  35. Optimality of the laplace mechanism in differential privacy. arXiv preprint arXiv:1504.00065.
  36. Diffusing Private Data over Networks. IEEE Transactions on Control of Network Systems, 5(3):1027–1037.
  37. Social learning and bayesian games in multiagent signal processing: How do local and global decision makers interact? IEEE Signal Processing Magazine,, 30(3):43–57.
  38. On Anonymizing Query Logs via Token-based Hashing. In International conference on World Wide Web (WWW 2017), pages 629–638.
  39. Social Learning and Distributed Hypothesis Testing. IEEE International Symposium on Information Theory (ISIT 2014), pages 551–555.
  40. Markov Chains and Mixing Times. American Mathematical Society.
  41. Data Security and Privacy in Wireless Body Area Networks. IEEE Wireless communications, 17(1):51–58.
  42. Federated learning with formal differential privacy guarantees. https://blog.research.google/2022/02/federated-learning-with-formal.html. Accessed: 2024-03-26.
  43. Graph Theoretic Methods in Multiagent Networks. Princeton University Press.
  44. Milojkovic, F. (2018). GEM House Opendata: German Electricity Consumption in Many Households over Three Years 2018–2020 (Fresh Energy).
  45. Robust De-anonymization of Large Sparse Datasets. In IEEE Symposium on Security and Privacy (SP 2008), pages 111–125.
  46. Net metering practices should be revised to better reflect the value of integrating distributed electricity generation into the nation’s power grid. https://www.nationalacademies.org/news/2023/05/net-metering-practices-should-be-revised-to-better-reflect-the-value-of-integrating-distributed-electricity-generation-into-the-nations-power-grid. Accessed: 2023-05-20.
  47. The Role of Net Metering in the Evolving Electricity System. https://www.nationalacademies.org/our-work/the-role-of-net-metering-in-the-evolving-electricity-system. Accessed: 2023-05-20.
  48. Nonasymptotic Convergence Rates for Cooperative Learning over Time-varying Directed Graphs. In American Control Conference (ACC 2015), pages 5884–5889. IEEE.
  49. Fast Convergence Rates for Distributed non-Bayesian Learning. IEEE Transactions on Automatic Control, 62(11):5538–5553.
  50. Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges. IEEE Communications Magazine, 58(6):46–51.
  51. Smooth Sensitivity and Sampling in Private Data Analysis. In ACM Symposium on Theory of Computing (STOC 2007), pages 75–84.
  52. Consensus Filters for Sensor Networks and Distributed Sensor Fusion. IEEE Conference on Decision and Control (CDC 2005), pages 6698 – 6703.
  53. Olshevsky, A. (2014). Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-agent Control. arXiv preprint arXiv:1411.4186.
  54. Group Decision-Making among Privacy-Aware Agents. AAAI Workshop on Privacy-preserving Artificial Intelligence (PPAI).
  55. Bayesian learning without recall. IEEE Transactions on Signal and Information Processing over Networks, 3(3):592–606.
  56. Distributed Estimation and Learning over Heterogeneous Networks. In Communication, Control, and Computing (Allerton 2016), pages 1314–1321. IEEE.
  57. Group decision making and social learning. In Decision and Control (CDC), 2016 IEEE 55th Conference on, pages 6783–6794. IEEE.
  58. Differentially private network data collection for influence maximization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 2795–2797.
  59. Seeding with differentially private network information. arXiv preprint arXiv:2305.16590.
  60. Enforcing Privacy in Distributed Learning with Performance Guarantees. IEEE Transactions on Signal Processing.
  61. LinkedIn’s Audience Engagements API: A Privacy Preserving Data Analytics System at Scale. arXiv preprint arXiv:2002.05839.
  62. Sayed, A. H. et al. (2014). Adaptation, Learning, and Optimization over Networks. Foundations and Trends® in Machine Learning, 7(4-5):311–801.
  63. Seneta, E. (2006). Non-negative Matrices and Markov Chains. Springer.
  64. Distributed detection: Finite-time analysis and impact of network topology. IEEE Transactions on Automatic Control, 61(11):3256–3268.
  65. Ensemble Models in Federated Learning for Improved Generalization and Uncertainty Quantification. IEEE Transactions on Automation Science and Engineering.
  66. Sweeney, L. (1997). Weaving Technology and Policy Together to Maintain Confidentiality. The Journal of Law, Medicine & Ethics, 25(2-3):98–110.
  67. Sweeney, L. (2015). Only you, your Doctor, and many others may know. Technology Science, 2015092903(9):29.
  68. Distributed consensus over Network with Noisy Links. In International Conference on Information Fusion (FUSION 2009), pages 146–154. IEEE.
  69. A Hybrid Approach to Privacy-preserving Federated Learning. In ACM Workshop on Artificial Intelligence and Security, pages 1–11.
  70. Privacy preservation in Federated Learning: An insightful Survey from the GDPR Perspective. Computers & Security, 110:102402.
  71. Tsitsiklis, J. N. (1993). Decentralized Detection. Advances in Statistical Signal Processing, 2(2):297–344.
  72. Convergence and asymptotic agreement in distributed decision problems. Automatic Control, IEEE Transactions on, 29(1):42–50.
  73. US Census (2020). 2020 decennial census: Processing the count: Disclosure avoidance modernization. https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance.html. Accessed: 2023-05-18.
  74. Social Learning with Bayesian Agents and Random Decision Making. IEEE Transactions on Signal Processing, 63(12):3241–3250.
  75. Warner, S. L. (1965). Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias. Journal of the American Statistical Association, 60(309):63–69.
  76. Collective Dynamics of “Small-world” Networks. Nature, 393(6684):440–442.
  77. Differentially Private SQL with Bounded User Contribution. Proceedings on privacy enhancing technologies, 2020(2):230–250.
  78. Estimation of the lipschitz constant of a function. Journal of Global Optimization, 8:91–103.
  79. A Scheme for Robust Distributed Sensor Fusion based on Average Consensus. In International Symposium on Information Processing in Sensor Networks (IPSN 2005), pages 63–70.
  80. A Space-time Diffusion Scheme for Peer-to-peer Least-squares Estimation. In International Conference on Information Processing in Sensor Networks (IPSN 2006), pages 168–176.
  81. Security-aware Waveforms for Enhancing Wireless Communications Privacy in Cyber-physical systems via Multipath Receptions. IEEE Internet of Things Journal, 4(6):1924–1933.
  82. Federated Data Analytics: A Study on Linear Models. IISE Transactions, pages 1–25. in-press.
  83. Privacy and Performance Trade-off in Cyber-physical Systems. IEEE Network, 30(2):62–66.
  84. Understanding Clipping for Federated Learning: Convergence and Client-level Differential Privacy. In International Conference on Machine Learning (ICML 2022).
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets