PrivacyBench: AI Privacy Benchmarking

• PrivacyBench is a series of independent, large-scale evaluation frameworks and datasets that rigorously benchmark privacy risks and detection in AI models across diverse modalities.
• It employs detailed privacy taxonomies, hybrid human-ML pipelines, and multiple metrics (e.g., PRR, RA, AUC) to assess model alignment and leakage, illustrating key utility-privacy trade-offs.
• PrivacyBench drives best practices by promoting privacy-by-design, emphasizing retrieval-level safeguards and targeted prompt engineering to mitigate structural vulnerabilities.

PrivacyBench is a term that encompasses a suite of independent, large-scale evaluation frameworks and datasets designed to rigorously benchmark privacy risks, detection, and preservation capabilities of intelligent systems across natural language, vision, audio, and multi-agent contexts. These diverse PrivacyBench resources focus on measuring model alignment, risk detection, empirical leakage, and practical safeguards, with public benchmarks covering MLLMs, VLMs, LLMs, RAG systems, smartphone and home agents, and federated settings. Below, the most salient frameworks and their technical foundations are summarized, drawing on major works establishing or employing the “PrivacyBench” concept in state-of-the-art privacy evaluation for AI.

1. Multi-dimensional Benchmarks for Privacy Awareness in Multimodal Agents

A dominant instantiation of PrivacyBench is SAPA-Bench, a large-scale, privacy-context-driven benchmark for evaluating MLLM-powered smartphone agents (Lin et al., 27 Aug 2025). SAPA-Bench covers 7,138 real-world scenarios, each annotated for privacy type, sensitivity, and context modality. Key aspects include:

• Privacy Taxonomy: Eight categories, including Account Credentials, Personal Information, Financial/Payment, Communication Content, Location/Environment, Device Permissions/Operations, Media/Files, and Behavior/Browsing History. Sensitivity is discretized into Low, Medium, and High, corresponding to distinct real-world risk gradients.
• Scenario Sources: Cases are mined from 50 popular apps and filtered screenshots (GUI-Odyssey/OS-Atlas, ~80k raw screens).
• Annotation Pipeline: Hybrid human–MLLM, consisting of data cleaning, synthetic instruction–response generation, human review, automatic structured annotation, and final consensus-based verification.

Agents are evaluated by five principal privacy metrics:

Metric	Definition
PRR (Privacy Recog. Rate)	Fraction of all scenarios flagged as privacy-related.
PLR (Privacy Loc. Rate)	Proportion of flagged cases with correct localization of privacy exposure (image vs. instruction).
PLAR (Level Awareness Rate)	Fraction with correct sensitivity-level assignment among flagged scenarios.
PCAR (Cat. Awareness Rate)	Fraction with correct privacy category assignment.
RA (Risk Awareness)	Among truly private cases, rate of producing an acceptable privacy warning by means of semantically matched response compared to the reference (obtained by LLM comparison).

Closed-source agents (e.g., Gemini 2.0-flash, GPT-4o) consistently outperform open-source VL agents (InternVL2.5, LLaVA-NeXT), with the top RA under explicit prompting at 67% (Gemini 2.0-flash). Detection strongly depends on scenario sensitivity, with high-sensitivity scenarios achieving up to 91% PRR (Gemini), but low/medium tiers remain inadequately recognized (PRR <20–78%). Prompt engineering substantially boosts RA (+10–30 percentage points), but the utility–privacy trade-off is pronounced and often unbalanced, as high utility (task completion rate, SR) is often achieved at the expense of privacy safeguards (Lin et al., 27 Aug 2025).

2. Conversational PrivacyBench: Leakage in Retrieval-Augmented Generation

PrivacyBench has also been formalized as the first end-to-end conversational framework measuring how RAG (retrieval-augmented generation) systems handle explicit “secrets” embedded in large, socially grounded datasets (Mukhopadhyay et al., 31 Dec 2025). This PrivacyBench combines:

• Community Simulation with Embedded Secrets: Synthetic digital footprints, where each secret is formally defined $(c_i, A_i, t_i)$ (content, authorized audience, timestamp), embedded into dense noise for realistic coverage of complex contexts.
• Multi-turn Probing Protocol: Automated adversarial LLMs engage in up to 10 turns with the target assistant, probing both directly (“When is the surprise party?”) and indirectly (“How is Alex doing?”) to elicit leakage.
• System Architecture: Standard RAG baseline employs ChromaDB for embedding search. At each turn, the top-k retrieved documents plus context are sent to the generator LLM. A single “privacy-aware” prompt instructs the model never to reveal secrets to unauthorized users.
• Key Metrics:
  • Leakage Rate (LR): Fraction of conversations with full secret disclosure to an unauthorized party; any nonzero LR indicates a catastrophic privacy failure.
  • Inappropriate Retrieval Rate (IRR): Fraction of conversational turns in which the retriever surfaces a secret-bearing document to an unauthorized entity.

Empirically, baseline RAG assistants leak in 15.8% of conversations; privacy-aware prompting reduces this to 5.1%—yet the IRR remains at 62% under all conditions. This exposes a structural flaw: privacy enforcement only at the generation stage cannot provide strong guarantees—as soon as a secret document is retrieved, it is in the LLM’s context window, and potential “jailbreaks” or hallucinations may cause leakage. Accordingly, authors recommend privacy-by-design at the retrieval layer, including access-control tagging, policy-based filtering, and development of privacy-preserving embeddings (Mukhopadhyay et al., 31 Dec 2025).

3. Visual and Multimodal PrivacyBench Instantiations

Visual privacy assessment is addressed at both the perception and reasoning levels:

a. Image-Privacy Benchmarking and Minimal Fine-tuning

PrivBench, introduced in (Samson et al., 2024), evaluates VLMs' capacity to rate and reason about privacy-sensitive content (debit card, face, license plate, nudity, etc.), with a compact, taxonomy-driven, GDPR-aligned design. The companion PrivBench-H employs visually plausible but non-private “hard negatives” to stress-test detection. Models report privacy scores and are evaluated by AUC-ROC, precision, recall, and F1.

TinyLLaVa, fine-tuned with as few as 150 PrivTune images, surpasses 0.90 AUC on PrivBench (from 0.53 to 0.96), demonstrating large privacy-sensitivity gains can be achieved with minimal data. Trade-offs with non-privacy tasks are small (<4 points on VQA, GQA, ScienceQA).

b. Individual-level Reasoning and Linkage (MultiPriv/PrivacyBench)

MultiPriv, described in (Sun et al., 21 Nov 2025), systematically evaluates VLMs across nine tasks—spanning direct/indirect identifier recognition, extraction, spatial localization, re-identification, and multi-hop chaining—on a bilingual, synthetic profile dataset. Metrics such as F1, accuracy, mIoU, and completion rates are used. Top open-source models score up to 0.87 on reasoning-based risk; closed-source models reach 0.81–0.83, but perception–reasoning correlations are weak (r≈0.2)—suggesting superficial privacy cues are not predictive of advanced leakage.

Open-source VLM frameworks, notably Qwen3-VL and InternVL3.5, present higher individual-level leakage than commercial APIs. Even high refusal models (e.g., GPT-5) do not erase all risk, especially on chained and cross-modal association tasks.

4. PrivacyBench in NLP and Textual Systems

PrivacyBench in the NLP context provides an attack–defense evaluation platform encompassing membership inference, model inversion, attribute inference, and extraction, with composable “attack chaining” (e.g., model extraction $\rightarrow$ attribute inference) (Huang et al., 2024). The platform supports diverse models and auxiliary knowledge sources (in-domain, partial, cross-domain). Attack success is reported as accuracy, AUC, F1-score, and perplexity-based metrics.

Intriguingly, membership and extraction attacks remain effective even with cross-domain data; targeted knowledge-distillation (KD) can raise MIA success from ~50% to ~70% in these settings. Standard defenses include DP-SGD, self-distillation ensembles, and representation hiding, with trade-offs reported as drops in attack metrics versus drops in target task accuracy.

Empirical privacy leakage is essential: differential privacy alone may eliminate membership leaks and extraction, but masked embeddings remain vulnerable to inversion attacks. This distinction between training-data privacy and inference-data privacy is central to realistic benchmarking (Li et al., 2023).

5. Limitations, Best Practices, and Future Directions

PrivacyBench frameworks have established a rigorous multi-dimensional structure for benchmarking privacy in AI systems.

• Systematic annotation workflows (hybrid LLM–human pipelines) and multi-attribute taxonomies are required for high-fidelity labels and effective error analysis.
• Evaluation metrics should always cover not just true-positive detection but also correct localization, scenario sensitivity, risk-awareness, and practical risk of downstream leakage.
• Closed-source systems currently outperform open-source models in privacy alignment due to extensive RLHF and safety-specific instruction tuning, but even the best models fail to reach deployment-grade performance across all sensitivity tiers.
• Prompt engineering—especially explicit or context-structured privacy hints—can substantially improve privacy detection, but does not solve structural privacy risks arising from retrieval or cross-modal reasoning.

Recommendations include the incorporation of explicit privacy schemas in prompting, multi-objective reward optimization (simultaneously optimizing for utility and privacy risk), and the development of retrieval-level and embedding-level controls for privacy-by-design assurance (Lin et al., 27 Aug 2025, Mukhopadhyay et al., 31 Dec 2025). Benchmarks should iteratively expand to cover dynamically evolving privacy contexts, cross-app flows, and scenarios where users define personalized privacy policies.

6. Representative PrivacyBench Frameworks and Their Focus

Benchmark	Domain/Modality	Core Metrics	Signature Contribution	Reference
SAPA-Bench	MLLM-powered agents	PRR, PLR, PLAR, PCAR, RA, SR	Largest-scale privacy scenario suite for smartphone MLLMs	(Lin et al., 27 Aug 2025)
PrivacyBench	RAG/system-level	Leakage Rate, IRR, Secret Coverage	Conversational, end-to-end RAG privacy probing with socially grounded secrets	(Mukhopadhyay et al., 31 Dec 2025)
PrivBench/PrivTune	VLM image content	AUC-ROC, Precision/Recall/F1	Minimal data fine-tuning for vision privacy, GDPR-aligned evaluation	(Samson et al., 2024)
MultiPriv/PrivacyBench	VLM cross-modal reasoning	F1, mIoU, accuracy, completion	Systematic benchmarking of individual-level linkage and reasoning privacy risks	(Sun et al., 21 Nov 2025)
PrivacyBench (NLP)	Text models	Attack success (AUC, F1), leakage	Unified, extensible evaluation for membership, inversion, extraction, chaining	(Huang et al., 2024)

7. Summary and Canonical Implications

PrivacyBench, as a meta-concept spanning multiple technical artifacts, defines the current gold standard for privacy-centric benchmarking in AI. By instantiating exhaustive privacy taxonomies, annotation protocols, and rigorous, empirically grounded risk metrics, these benchmarks collectively diagnose deep, under-measured vulnerabilities in state-of-the-art AI agents. The prevailing findings across modalities converge: even SOTA models frequently fail to meet robust privacy standards, especially in nuanced or low-sensitivity cases, and privacy risks are often structurally decoupled from utility and surface-level detection. Future benchmarks and system designs will need to operationalize privacy safeguards not merely as a model output post-processing step but throughout the architectural stack, with privacy-by-design as a primary constraint. The ongoing development and refinement of the PrivacyBench family will be central to both empirical model evaluation and deployment-time privacy risk management in intelligent systems (Lin et al., 27 Aug 2025, Mukhopadhyay et al., 31 Dec 2025, Samson et al., 2024, Sun et al., 21 Nov 2025, Huang et al., 2024).