Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (2405.14457v2)

Abstract: Machine learning models can be trained with formal privacy guarantees via differentially private optimizers such as DP-SGD. In this work, we focus on a threat model where the adversary has access only to the final model, with no visibility into intermediate updates. In the literature, this hidden state threat model exhibits a significant gap between the lower bound from empirical privacy auditing and the theoretical upper bound provided by privacy accounting. To challenge this gap, we propose to audit this threat model with adversaries that \emph{craft a gradient sequence} designed to maximize the privacy loss of the final model without relying on intermediate updates. Our experiments show that this approach consistently outperforms previous attempts at auditing the hidden state model. Furthermore, our results advance the understanding of achievable privacy guarantees within this threat model. Specifically, when the crafted gradient is inserted at every optimization step, we show that concealing the intermediate model updates in DP-SGD does not amplify privacy. The situation is more complex when the crafted gradient is not inserted at every step: our auditing lower bound matches the privacy upper bound only for an adversarially-chosen loss landscape and a sufficiently large batch size. This suggests that existing privacy upper bounds can be improved in certain regimes.

• The paper introduces adversarial gradient-crafting to maximize privacy loss in DP-SGD without using intermediate model updates.
• Empirical results demonstrate that crafted gradients reveal no privacy amplification, thereby tightening auditing bounds.
• The study underscores implications for model deployment and privacy accounting in non-convex settings, guiding future research.

Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

Authors: Tudor Cebere, Aurélien Bellet, Nicolas Papernot

The paper explores the privacy guarantees of machine learning models trained using Differentially Private Stochastic Gradient Descent (DP-SGD) within the confines of the hidden state threat model. This threat model, distinct from the standard model, restricts adversaries to only the final model without access to intermediate updates, challenging prevalent assumptions and seeking to tighten privacy auditing in this constrained environment.

Core Contributions

The researchers introduce a new set of gradient-crafting adversaries that escalate privacy loss for the final model in the hidden state threat model. Their proposed adversaries ingeniously craft a gradient sequence without intermediate model knowledge to yield the highest possible privacy loss for the final model. This approach fundamentally contrasts existing methods that generally release intermediate models, helping bridge the notable gap between empirical privacy auditing lower bounds and privacy accounting theoretical upper bounds.

Key contributions include:

• Adversarial Gradient-Crafting: The research posits adversaries that craft gradients before the training begins, incorporating mechanisms to intensify privacy loss without intermediate model updates.
• Empirical Validation: Through experimental results, the authors assert their adversaries consistently outperform previous attempts in the hidden state model. They demonstrate that inserting crafted gradients at every optimization step reveals no privacy amplification, challenging prevailing beliefs.
• Performance in Non-Convex Settings: By not inserting crafted gradients at every step, the researchers show evidence of privacy amplification in general non-convex settings. The paper meticulously outlines scenarios where existing privacy upper bounds can be refined, suggesting privacy amplification phenomena despite the weaker outcomes in non-convex regimes versus convex scenarios.

Strong Numerical Results and Claims

The paper presents strong numerical results underpinning their claims:

• Non-Amplification with Frequent Gradient Insertion: Experimental evidence shows that when crafted gradients are inserted at each training step, privacy does not amplify, offering critical insights into the robustness of DP-SGD under certain adversarial conditions.
• Sigificant Outperformance: The newly proposed gradient-crafting adversaries significantly outperform canary-crafting adversaries from previous research, delivering tighter lower bounds on the privacy parameters.
• Privacy Amplification Trends: In scenarios where gradients are not inserted at every step, the emerging privacy amplification phenomenon—although weaker in non-convex regimes—indicates room for improvement in existing privacy upper bounds.

Implications and Future Developments

Practical Implications:

1. Model Deployment: For practitioners open-sourcing trained models, these findings emphasize that withholding intermediate models alone may not suffice to ensure privacy, particularly if adversarially crafted gradients are inserted at each step.
2. Privacy Accounting Enhancement: The insights from this research could be employed to refine privacy accounting mechanisms, making privacy guarantees more robust in non-convex regimes.

Theoretical Implications:

1. New Tight Auditing Techniques: By providing tighter auditing methods for the hidden state model, the paper sets the stage for future audits that enhance the understanding and effectiveness of DP mechanisms.
2. Understanding Privacy Amplification: The evidence supporting weaker yet existing privacy amplification in non-convex settings opens avenues for further research into privacy dynamics beyond convex regimes.

Future Research Directions:

• Adversarial Crafting Feasibility: Investigating practical scenarios where adversaries can craft optimal gradient sequences and exploring the feasibility and limitations of such crafting in real-world deployments.
• Extended Threat Models: Broadening the paper to more complex threat models, including federated learning environments with partial participation, where intermediate models remain hidden to a subset of clients.
• Improved Privacy Accountants: Developing new privacy accountants that integrate findings on privacy amplification within non-convex regimes, thereby enhancing both theoretical models and practical implementations.

Conclusion

By challenging and refining the existing understanding of privacy guarantees in DP-SGD under the hidden state threat model, this paper lays a substantial groundwork for more precise and robust privacy auditing methods. The implications of their findings resonate through both practical deployments and theoretical advances, marking critical steps toward strengthening the privacy foundations of differentially private learning mechanisms.