Differential Privacy in RAG

• Differential Privacy in RAG is defined as using (ε, δ)-DP guarantees during generation to prevent sensitive corpus leakage in retrieval-augmented systems.
• Key methodologies include Privacy-Aware Decoding with Gaussian noise and Adaptive Private Mixing with Laplace-based screening, offering adaptive privacy accounting via RDP.
• Empirical results demonstrate up to 70% leakage reduction with tailored DP schemes, while highlighting challenges in achieving complete end-to-end retrieval privacy.

Differential privacy (DP) mechanisms for Retrieval-Augmented Generation (RAG) address the problem of protecting sensitive content exposed during retrieval and generation activities in LLMs. RAG systems, widely deployed to enhance factuality and context, pose novel privacy risks, especially when retrieved corpora contain confidential information. This survey covers the principles, formalizations, and state-of-the-art DP mechanisms specifically tailored for RAG, as well as the security, utility, and implementation trade-offs.

1. Privacy Threats in RAG and the Necessity for Differential Privacy

RAG architectures fuse LLM inference with retrieval from external corpora, introducing a new vector for privacy leakage: an attacker may extract verbatim or semantically similar fragments of sensitive documents via extraction attacks on the generation process. Standard LLM DP mechanisms, focused on training-time privacy, do not account for inference-time corpus leakage.

Recent work has demonstrated that RAG systems, if left unprotected, may leak sensitive data present in retrieved contexts through output sequences. Empirical studies show substantial private-context extraction rates, motivating rigorous, formal DP mechanisms for the generation (decoding) phase of RAG (Wang et al., 5 Aug 2025). Furthermore, speculative decoding acceleration methods can introduce new timing and size side-channels if not handled with care (Wei et al., 2024).

2. Formal Frameworks for Differential Privacy in RAG

The formal goal for DP in RAG is the standard $(\epsilon, \delta)$-differential privacy, interpreted per-response: the distribution over generated sequences should be approximately invariant to a single change—such as adding/removing a document—within the retrieved private corpus (Majmudar et al., 2022, Wang et al., 5 Aug 2025).

Mechanisms are generally classified by two main axes:

• Perturbation targets: Logit vector, token probability distribution, support set truncation.
• Privacy accounting: Pure $(\epsilon,0)$-DP, Rènyi differential privacy (RDP), or relaxed/approximate metrics (e.g., $\gamma$-fraction token-wise DP).

For RAG, RDP is especially suited due to its tight composition and tractability when performing multiple noise injections over the response sequence (Wang et al., 5 Aug 2025, Flemings et al., 2024).

3. Core Mechanisms: Gaussian and Mixing-based Perturbations

3.1 Privacy-Aware Decoding (PAD) via Gaussian Mechanisms

The leading approach for RAG is Privacy-Aware Decoding (PAD) (Wang et al., 5 Aug 2025). PAD injects calibrated Gaussian noise into token logits at each generation step:

• Initialization: For each step $t$, the raw model logits $s_t$ are computed.
• Token-level screening: Confidence- and margin-based screening identifies tokens with potential privacy risk.
• Noise scale setting: The local sensitivity $\Delta_t$ is estimated; the context-aware calibration function adjusts the noise according to entropy, token position, and prediction confidence.
• Perturbation: Additive noise $n_t \sim \mathcal{N}(0, \sigma_t^2 I)$ is added to $s_t$; $\sigma_t$ is chosen to satisfy the RDP constraint given estimated sensitivity. The token distribution is sampled post-perturbation.
• RDP accounting: Each noise injection incurs a step-wise $\epsilon_t^{\mathrm{RDP}}$; cumulative privacy loss is tracked and then converted to $(\epsilon,0)$0-DP for those tokens.

The mechanism achieves selective (fraction-$(\epsilon,0)$1) DP protection and reports exact $(\epsilon,0)$2 bounds per response, enabling explicit utility–privacy tuning.

3.2 Adaptive Private Mixing (AdaPMixED)

An alternative to Gaussian perturbation is Adaptive Private Mixing (AdaPMixED) (Flemings et al., 2024):

• Uses a mixture of private and public model distributions: for each token, the sampling is from $(\epsilon,0)$3.
• Noisy screening: Each query’s KL divergence between private and public distributions is computed and compared—plus Laplace noise—to a threshold. If passed, mixing-based sampling is allowed; if failed, only the public distribution is used, incurring zero private cost.
• Adaptive accountability: Privacy cost for each query is data-dependent—queries similar to public content incur less cost; total privacy loss is computed as the sum of screening and mixing costs.

Table: Key characteristics of prominent DP mechanisms for RAG

Mechanism	Perturbation Target	Privacy Accounting
PAD (Gaussian)	Logits	RDP, per-token, $(\epsilon,0)$4
AdaPMixED	Probability distribution	RDP, data-dependent
Uniform Mixing	Probability distribution	Pure $(\epsilon,0)$5-DP

4. Application to RAG: Algorithms, Hyperparameters, and Practical Issues

4.1 PAD Algorithmic Details

The PAD procedure for RAG integrates:

• Confidence-based screening: Only high-risk tokens (low entropy, large logit gap) get maximum noise, reducing utility loss on benign tokens.
• Context-aware calibration: Combines token entropy, generation step, and confidence to modulate noise.
• Parameterization: Screening thresholds (e.g., $(\epsilon,0)$6, $(\epsilon,0)$7), base $(\epsilon,0)$8 (e.g., 0.2), and amplification factor $(\epsilon,0)$9.

Parameter tuning seeks the knee point where leakage is sharply reduced but output perplexity remains acceptable (e.g., $\gamma$0 yields strong privacy, minor degradation in fluency).

4.2 AdaPMixED Implementation

AdaPMixED introduces:

• Noisy screening with Laplace noise: Accept queries only if $\gamma$1 (with $\gamma$2).
• Per-query mixing: For accepted queries, choose mixing weight $\gamma$3 to satisfy an RDP constraint $\gamma$4.
• Data-dependent cost: Privacy loss is accumulated adaptively, typically yielding 2–16$\gamma$5 savings in $\gamma$6 for real datasets.

In both schemes, no retraining of the underlying LLM or retriever is required; these are strictly inference-time defenses.

5. Privacy–Utility Trade-offs and Empirical Outcomes

Empirical evaluation on medical QA, corporate email, and general text domains demonstrates:

• PAD with $\gamma$7 reduces verbatim leakage by 50–70% relative to extraction baselines, at perplexity equal or superior to static or summarization-based baselines(Wang et al., 5 Aug 2025).
• Data-dependent AdaPMixED achieves %%%%28$(\epsilon,0)$629%%%% smaller total $t$0 and lower perplexity than fixed-budget PMixED or DP-SGD in practice(Flemings et al., 2024).
• The PAD approach yields a relaxed guarantee: only a fraction $t$1 of tokens are formally $t$2-DP-protected, typically targeting high-risk spans.
• Overaggressive noise or low $t$3 can degrade fluency to unusable levels; judicious parameter choice is critical.

6. Security and Theoretical Guarantees

Mechanisms rely on formal accounting under Rènyi DP with tight composition, leveraging per-step logit or probability mixing in a black-box inference setting(Wang et al., 5 Aug 2025, Flemings et al., 2024). Selective DP mechanisms (e.g., $t$4-relaxed PAD) report precise cumulative privacy loss for the protected segment, but offer no guarantee for unprotected tokens.

Attack-based evaluations further emphasize that side channels (e.g., speculative decoding timing, packet size) can bypass semantic DP defenses if not addressed at the network/protocol level(Wei et al., 2024).

7. Limitations, Challenges, and Future Directions

Current DP mechanisms for RAG face several challenges:

• Partial (fractional) protection: Only a subset of tokens may be protected; adversaries may exploit unprotected spans.
• Sensitivity estimation: Heuristics based on logit margins or entropy may not fully capture actual privacy risk; improved, possibly smoother, sensitivity estimators are needed(Wang et al., 5 Aug 2025).
• Hyperparameter adaptation: Screening thresholds and calibration weights are static; adaptive thresholding and tuning per-query or per-session remain open.
• Retrieval leakage closure: True end-to-end privacy requires joint DP over retrieval and decoding steps(Wang et al., 5 Aug 2025).
• Extension to multimodal and complex RAG architectures: Existing approaches focus on text; generalizing to images, code, or multi-hop retrieval settings is an active research area.

Practical deployment of DP for RAG thus involves balancing selective, adaptive privacy guarantees against generation utility, throughput, and the risk of auxiliary side-channel leaks. Explicit reporting of per-run privacy loss and coverage is advised in sensitive applications.

---

Principal references: (Majmudar et al., 2022, Wang et al., 5 Aug 2025, Flemings et al., 2024, Wei et al., 2024)