Investigating Privacy Bias in Training Data of Language Models (2409.03735v2)

Abstract: As LLMs are integrated into sociotechnical systems, it is crucial to examine the privacy biases they exhibit. A privacy bias refers to the skew in the appropriateness of information flows within a given context that LLMs acquire from large amounts of non-publicly available training data. This skew may either align with existing expectations or signal a symptom of systemic issues reflected in the training datasets. We formulate a novel research question: how can we examine privacy biases in the training data of LLMs? We present a novel approach to assess the privacy biases using a contextual integrity-based methodology to evaluate the responses from different LLMs. Our approach accounts for the sensitivity of responses across prompt variations, which hinders the evaluation of privacy biases. We investigate how privacy biases are affected by model capacities and optimizations.

• The paper introduces the LLM-CI framework to assess privacy norms using a multi-prompt method grounded in Contextual Integrity theory.
• It employs modular tools to evaluate how model capacity, alignment, and quantization impact the encoding of societal privacy norms.
• Experimental results reveal significant response variability among models, highlighting the need for nuanced privacy norm assessments.

Assessing Contextual Integrity Norms in LLMs: A Review of LLM-CI Framework

Recent advancements in LLMs have positioned them as critical components in sociotechnical systems such as education and healthcare. Despite remarkable performance improvements, LLMs often inadvertently encode societal preferences and norms during training, leading to potential privacy violations if these norms are misaligned with societal expectations. The paper "LLM-CI: Assessing Contextual Integrity Norms in LLMs" presents a novel framework aimed at ensuring that LLMs conform to established privacy norms grounded in the theory of contextual integrity (CI).

Framework Overview

LLM-CI introduces a comprehensive, modular, and open-source framework for assessing the privacy norms encoded in LLMs. Utilizing a Contextual Integrity-based factorial vignette methodology, LLM-CI evaluates norms across various contexts and models. One of the key challenges addressed by the framework is prompt sensitivity—small variations in prompts that can yield different responses from LLMs. To tackle this, the authors propose a multi-prompt assessment methodology, which focuses on the prompts that yield consistent responses across multiple variants. This approach enhances the reliability of the norm assessment.

The framework includes several key modules:

1. Vignette Module: Generates vignettes based on the essential CI parameters and converts them into prompts compatible with various LLMs.
2. Inference Module: Loads pre-trained models and leverages inference engines like vLLM to execute models efficiently.
3. Clean-up Module: Filters and processes responses to extract relevant Likert scale values.
4. Analysis and Plotting Module: Summarizes statistics, generates heatmaps, and performs statistical significance tests to identify and evaluate encoded norms.

Experimental Setup and Methodology

The framework's effectiveness was tested using two datasets: IoT and COPPA, covering over 8,000 vignettes to evaluate information flow appropriateness in various contexts. The authors used ten different prompt variants to empirically demonstrate prompt sensitivity and implemented a methodology that evaluates norms based on the majority of consistent responses across prompt variants.

Evaluation and Results

The evaluation included ten state-of-the-art LLMs such as the tulu-2 models, llama-3.1-8B-Instruct, and gpt-4o-mini, examining factors like model capacity, alignment, and quantization. Key findings are summarized below:

1. Model Type and Capacity: Different LLMs exhibited significant variability in responses, influenced by their training datasets and inherent prompt sensitivity. Higher-capacity models (e.g., 13B) demonstrated different encoded norms compared to lower-capacity models (e.g., 7B).
2. Model Alignment: Alignment techniques such as Direct Preference Optimization (DPO) significantly influenced the encoded norms. Aligned models often presented more acceptable information flow scenarios compared to their non-aligned counterparts.
3. Quantization: Quantization techniques like Activation-Aware Quantization (AWQ) impacted the norms significantly, sometimes reducing the quality of responses by making the models more conservative or, conversely, less stringent about certain norms.
4. Combined Effects (A{content}Q): Models that employed both alignment and quantization exhibited complex behavior, showing inconsistency in certain information flow scenarios and highlighting the need for nuanced evaluations.

Pairwise Wilcoxon Signed-Rank tests confirmed the statistical significance of the observed differences, indicating that factors such as alignment and quantization significantly affect the norms encoded in LLMs.

Implications and Future Directions

The implications of this research are profound for both practical applications and theoretical advancements. Practically, the LLM-CI framework can assist model developers, regulators, and researchers in evaluating and enhancing the privacy norms encoded in LLMs to ensure they align with societal expectations. Theoretically, the findings open new avenues for exploring the relationship between model training processes and the societal norms they encode.

Future developments in AI can leverage the insights from LLM-CI to design training strategies that inherently align encoded norms with socially accepted standards. Further research could extend LLM-CI to include different types of majority thresholds, explore other alignment techniques, and incorporate additional datasets to cover a wider array of contexts.

In conclusion, LLM-CI provides a robust and reliable framework for assessing the normative aspects of LLMs, paving the way for more socially responsible AI systems. The multi-prompt assessment methodology enhances the reliability of norm extraction, making LLM-CI a valuable tool in the ongoing effort to align AI technologies with societal values.