Privacy Side Channels in Machine Learning Systems (2309.05610v2)

Abstract: Most current approaches for protecting privacy in ML assume that models exist in a vacuum. Yet, in reality, these models are part of larger systems that include components for training data filtering, output monitoring, and more. In this work, we introduce privacy side channels: attacks that exploit these system-level components to extract private information at far higher rates than is otherwise possible for standalone models. We propose four categories of side channels that span the entire ML lifecycle (training data filtering, input preprocessing, output post-processing, and query filtering) and allow for enhanced membership inference, data extraction, and even novel threats such as extraction of users' test queries. For example, we show that deduplicating training data before applying differentially-private training creates a side-channel that completely invalidates any provable privacy guarantees. We further show that systems which block LLMs from regenerating training data can be exploited to exfiltrate private keys contained in the training set--even if the model did not memorize these keys. Taken together, our results demonstrate the need for a holistic, end-to-end privacy analysis of machine learning systems.

• The paper demonstrates novel privacy side channel attacks that exploit systemic ML components to amplify privacy breaches.
• It analyzes four key areas—training data filtering, input preprocessing, output filtering, and query filtering—to expose unique attack vectors.
• Findings highlight that traditional isolated privacy measures are inadequate, urging a holistic, end-to-end approach for robust ML privacy protection.

Privacy Side Channels in Machine Learning Systems

The paper "Privacy Side Channels in Machine Learning Systems" addresses a critical yet often overlooked vulnerability in the deployment of ML models: privacy side channels. It argues that while current privacy-preserving approaches in machine learning assume that models operate in isolation, true privacy risks emerge when considering the broader system in which these models exist. This system includes various components such as training data filtering, input preprocessing, output monitoring, and query filtering. The authors propose and rigorously analyze a new type of attack—privacy side channel attacks—that exploit these systemic components, leading to notably increased rates of privacy breaches.

Overview of Privacy Side Channels

The manuscript introduces the concept of privacy side channels, which arise when adversaries exploit systemic components of ML systems, enabling the extraction of private information at enhanced rates compared to when models are analyzed in isolation. The authors categorize these side channels into four main areas spanning the ML lifecycle:

1. Training Data Filtering: This refers to the deduplication and filtering of training data to remove duplicates or abnormal examples. The paper reveals that such processes inadvertently create dependencies among different user data, which adversaries can leverage to amplify privacy attacks, such as through membership inference or data poisoning.
2. Input Preprocessing: In certain applications, models require input data to go through preprocessing steps such as tokenization. When these steps are informed by the training data, they can inadvertently create side channels that leak information about rare or unique training inputs.
3. Model Output Filtering: To protect against the leakage of training data, many systems employ output filters to suppress the reproduction of verbatim data. However, the paper demonstrates that this approach can backfire, leading to near-perfect membership inference attacks whereby the participation of specific training data points can be ascertained with high confidence.
4. Query Filtering: The paper expands on attack vectors that exploit input query filters, which are designed to reject adversarial inputs. By crafting strategic queries, adversaries can infer the existence of others' queries, thus breaching test-time privacy.

Key Findings and Implications

Through empirical evaluations on various datasets and models, the authors uncover significant vulnerabilities when conventional privacy-preserving measures are applied without consideration of systemic interactions. One striking result is the potential for data deduplication to nullify the formal privacy guarantees offered by differential privacy mechanisms, leading to a profound underestimation of actual privacy risks.

The findings underscore the imperative for a holistic, end-to-end approach to privacy analysis in ML systems. Isolated evaluations of individual components or phase-specific privacy measures are insufficient. Instead, privacy guarantees must encompass the full system, including interactions among filters, preprocessors, and other systemic mechanisms.

Speculation on Future Developments in AI

This research signals a necessary paradigm shift in how privacy is conceptualized within AI systems. As ML models become increasingly embedded in broader societal applications, the demand for robust system-level privacy protections will grow. Future work must address the complex interplay between security and privacy, where enhancements in one may lead to compromises in the other. Advances in this area will likely involve the development of sophisticated privacy analysis tools that consider the entirety of the ML pipeline.

Conclusion

The paper presents a comprehensive and technical exploration of privacy side channels in ML systems, unveiling critical insights into the systemic sources of privacy leakage. By highlighting how standard practices can inadvertently increase privacy risks, the work sets a foundation for developing more resilient privacy-preserving mechanisms in machine learning. This paper challenges the research community to rethink privacy in the context of the entire ML system, fostering advancements that will secure ML deployments against sophisticated adversarial threats.