Cape: Context-Aware Prompt Perturbation Mechanism with Differential Privacy (2505.05922v2)

Abstract: LLMs have gained significant popularity due to their remarkable capabilities in text understanding and generation. However, despite their widespread deployment in inference services such as ChatGPT, concerns about the potential leakage of sensitive user data have arisen. Existing solutions primarily rely on privacy-enhancing technologies to mitigate such risks, facing the trade-off among efficiency, privacy, and utility. To narrow this gap, we propose Cape, a context-aware prompt perturbation mechanism based on differential privacy, to enable efficient inference with an improved privacy-utility trade-off. Concretely, we introduce a hybrid utility function that better captures the token similarity. Additionally, we propose a bucketized sampling mechanism to handle large sampling space, which might lead to long-tail phenomenons. Extensive experiments across multiple datasets, along with ablation studies, demonstrate that Cape achieves a better privacy-utility trade-off compared to prior state-of-the-art works.

CAPE: Context-Aware Prompt Perturbation Mechanism with Differential Privacy

The recent proliferation of LLMs in services such as ChatGPT has ushered in significant capabilities for text understanding and generation. However, this growth has been paralleled by escalating concerns regarding the potential leakage of sensitive user data through plaintext prompts sent to servers. Existing privacy solutions, such as secure multi-party computation (MPC) and homomorphic encryption (HE), offer provable guarantees but suffer from considerable computation and communication overheads, hindering their practical deployment.

The paper introduces CAPE, a novel Context-Aware Prompt Perturbation Mechanism designed to mitigate privacy risks while maintaining a beneficial privacy-utility balance. CAPE leverages differential privacy (DP) to perturb user prompts, thus obscuring sensitive information from inference service providers without necessitating extensive architectural modifications to existing systems.

Key Contributions and Findings

1. Hybrid Utility Function: CAPE introduces a sophisticated utility function combining contextual information from logits with token embedding distances (e.g., Euclidean distance). This allows CAPE to retain contextual coherence while ensuring semantic similarity in perturbations. The paper observes that merely relying on token distances, as prior approaches have done, fails to capture the nuances of semantic contexts, often impairing text utility.
2. Bucketized Sampling Mechanism: The research presents a bucketized exponential mechanism to efficiently handle large sampling spaces inherent in NLP tasks. By segmenting candidates into utility-based buckets and sampling based on average bucket utility, CAPE effectively reduces the probability of selecting low-utility replacements. This approach addresses long-tail distributions encountered in large vocabularies, consequently ameliorating the privacy-utility trade-off.
3. Empirical Evaluations: Extensive experiments conducted across text classification and open-ended text generation tasks demonstrate the advantages of CAPE over existing methods such as SANTEXT, CUSTEXT, and InferDPT. CAPE consistently outperforms peers in terms of privacy protection against KNN and Masked Token Inference attacks, while maintaining or improving upon the utility in static inference models across datasets like SST-2, QNLI, and Wikitext-103-v1. For instance, under stringent privacy budgets (e.g., ε = 1), CAPE provides significantly better task utility than SANTEXT and InferDPT.

Implications and Future Directions

The implications of CAPE are twofold. Practically, it offers a scalable solution to the privacy challenges faced by LLM-based services, ensuring that user data remains protected without sacrificing system performance or requiring complex infrastructure changes. Theoretically, it advances the discourse on the integration of contextual information within differential privacy frameworks, presenting new avenues to combat adversarial attacks more robustly.

Looking forward, the framework established by CAPE sets the stage for further exploration into token-specific privacy mechanisms that adjust sensitivity based on semantic categories, potentially enhancing both privacy protections and utility outcomes. Moreover, the methodology invites investigation into other domains where large-scale model inference intersects with privacy concerns, broadening the applicability of context-aware perturbation techniques.

Ultimately, by safeguarding sensitive information with a rigorously defined privacy mechanism, CAPE not only enhances trust in AI systems but further promotes their responsible and ethical adoption.