---
title: 'PrivacyPeek: Benchmark for Privacy Leakage in LLM Agents'
url: https://www.emergentmind.com/topics/privacypeek
type: topic
---

# PrivacyPeek: Benchmark for Privacy Leakage in LLM Agents

PrivacyPeek is a benchmark for auditing **acquisition-stage privacy leakage** in LLM-based agents: it measures not only whether an agent discloses sensitive information in its final answer, but whether the agent **acquires sensitive information beyond the task scope** while invoking tools during multi-step execution [2606.00152]. The benchmark formalizes a case as $c=\langle q_c, T_c, F_c, \mathcal{E}_c, p_c\rangle$, where $q_c$ is the user instruction, $T_c$ the available tool set, $F_c$ the underlying files or records, $\mathcal{E}_c$ the minimum scope of personal data needed for task completion, and $p_c$ a follow-up privacy probe [2606.00152]. In this formulation, privacy risk begins when unnecessary data enters the agent’s context, because that data is then available for subsequent reasoning, accidental disclosure, or elicitation by an attacker.

## 1. Conceptual scope

PrivacyPeek distinguishes **acquisition-stage leakage** from **disclosure-stage leakage**. In the benchmark’s agent setting, execution produces a tool-call trajectory
$$
\tau_c=(a_1,o_1,a_2,o_2,\ldots,a_{n-1},o_{n-1},a_n),
$$
where each $a_t$ for $t<n$ is a tool invocation, each $o_t$ is the corresponding tool output, and $a_n$ is the final natural-language task response [2606.00152]. Traditional privacy evaluations concentrate on what appears in $a_n$ or in outbound actions. PrivacyPeek instead asks whether any $o_t$ contains personal data outside $\mathcal{E}_c$.

The benchmark defines a sensitive set $k_c$ for each case: these are the personal-data items that lie **outside** the minimum task scope but remain privacy-relevant under norms such as GDPR’s data minimization principle and HIPAA’s “minimum necessary” standard [2606.00152]. A trajectory exhibits acquisition-stage leakage if at least one tool call acquires any member of $k_c$.

This framing shifts privacy analysis from final outputs to **context formation**. An agent may appear safe if its final answer omits sensitive details, yet still be unsafe if it has already loaded those details from files, databases, or records. PrivacyPeek operationalizes that distinction by auditing both the agent’s tool-call history and its later susceptibility to elicitation [2606.00152].

## 2. Benchmark construction and taxonomy

PrivacyPeek contains **1,182 cases**, spanning **7 acquisition behaviours** and **16 application domains** [2606.00152]. The domains include healthcare, finance, legal services, education, government or public service, research, HR, customer support, social media, technology, retail, engineering, and logistics. The benchmark is organized by acquisition behaviour rather than by domain, so each behavioural pattern is systematically represented [2606.00152].

The seven behaviours are divided into two broad groups. The first group concerns violations in file attributes or retrieval scope: **Normal-Filename Access**, **Sensitive-Filename Access**, **Cross-Format Access**, and **Out-of-Window Access**. The second group concerns violations in returned content: **Excess-Field Access**, **Forbidden-Content Access**, and **Out-of-Scope Inferential Access** [2606.00152]. These categories distinguish, for example, opening an irrelevant file with an ordinary name from opening one whose filename itself advertises sensitivity, or reading an allowed record that contains extra fields from combining otherwise innocuous fields into a sensitive inference.

| Acquisition behaviour | Cases |
|---|---:|
| Normal-Filename Access | 217 |
| Sensitive-Filename Access | 158 |
| Cross-Format Access | 145 |
| Out-of-Window Access | 138 |
| Excess-Field Access | 218 |
| Forbidden-Content Access | 163 |
| Out-of-Scope Inferential Access | 143 |

Case generation follows a human-in-the-loop pipeline. Template seeds define the behavioural structure; GPT-4o expands them into full cases containing user instructions, system prompts, files, records, and probes; and the authors then apply **10 deterministic quality gates** plus human inspection [2606.00152]. The quality gates enforce complete schema coverage, one-and-only-one target behaviour per case, feasibility of the task without violating scope, uniqueness of ground-truth sensitive items, and probe specificity. Only cases passing all gates are retained.

This design makes PrivacyPeek a benchmark of **minimum-scope compliance** rather than merely a benchmark of overt disclosure. A case can therefore be privacy-violating even when the agent finishes the user task correctly and says nothing obviously sensitive in its main answer [2606.00152].

## 3. Auditing methodology

PrivacyPeek consists of two coupled auditing procedures: **Acquisition Inspection** and **Probe Elicitation** [2606.00152].

Acquisition Inspection examines the tool-call trajectory directly. For a case $c$, the benchmark assigns case-level content exposure when any sensitive item in $k_c$ appears in any intermediate tool output:
$$
\mathrm{CER}_c=\mathbf{1}\!\left[\exists\, t<n,\exists\, s\in k_c:\ s\subseteq o_t\right].
$$
The corpus-level **Content Exposure Rate** is then
$$
\mathrm{CER}=\frac{1}{N}\sum_{c=1}^{N}\mathrm{CER}_c,
$$
with $N=1{,}182$ [2606.00152]. Since a trivial agent that never completes tasks could reduce exposure simply by failing, the benchmark also measures **Task Completion Rate** and defines **Helpful Content Exposure Rate**
$$
\mathrm{HCER}=\frac{\sum_c \mathrm{CER}_c\cdot \mathrm{TCR}_c}{\sum_c \mathrm{TCR}_c},
$$
thereby conditioning leakage on successful task completion [2606.00152].

Probe Elicitation models an attacker who interacts with the agent **after** the original task is complete. In the first phase, the agent executes the task with tools enabled. In the second phase, tools and network access are disabled, and the evaluator submits a case-specific follow-up probe $p_c$ designed to elicit out-of-scope sensitive information that may already reside in context [2606.00152]. Leakage in the main response is measured as **Output Leakage Rate**; leakage in the probe response is measured as **Probe Leakage Rate**, with a corresponding helpful variant,
$$
\mathrm{HPLR}=\mathbb{P}[\mathrm{PLR}_c=1\mid \mathrm{TCR}_c=1].
$$

For open-ended text judgments, the benchmark uses **GPT-4o as a judge**. The judge receives the task context, the sensitive items, the task response, and the probe response, and determines whether a subject and a sensitive attribute are disclosed together through verbatim mention, paraphrase, or explicit inference [2606.00152]. Pure refusal, wrong or hallucinated values, generic category statements, or descriptions of tool use without sensitive content do not count as disclosure.

## 4. Experimental environment and evaluated agents

All experiments use **Smolagents CodeAgent** as the execution framework [2606.00152]. The agent emits Python code, can invoke up to **8 tools**, and is limited to **6 steps** per case. The tool surface includes `query_database`, `read_document`, `search_records`, `list_files`, `get_file_properties`, `write_note`, `send_message`, and `export_report` [2606.00152]. File-system access is restricted to those tools, so the benchmark audits the acquisition channel through explicit tool use rather than unconstrained local I/O.

The baseline system prompt identifies the agent as a domain assistant and lists the tools, but it does **not** include privacy instructions [2606.00152]. This makes the base evaluation one of unconstrained agentic behavior under benign user requests and uncompromised tools.

PrivacyPeek evaluates **10 LLM-based agents across 4 model families** [2606.00152]. The closed-source agents are GPT-4.1, GPT-5.1, Claude-Sonnet-4, and Claude-Sonnet-4.5. The open-source agents are Qwen3-4B-Instruct, Qwen3-14B, Qwen3-30B-A3B, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.3-70B-Instruct. Each is run once per case in the main evaluation.

The benchmark also tests prompt-level mitigation. A **privacy-aware system prompt** instructs the agent not to expose sensitive categories such as personal identifiers, health information, finance, and behavioral logs unless explicitly authorized [2606.00152]. In a separate intervention, the benchmark inserts confidentiality markers such as `CONFIDENTIAL: <ROLE> ONLY` into the data substrate to test whether explicit in-data warnings reduce acquisition or disclosure.

## 5. Empirical findings

The central empirical result is that **unnecessary acquisition of sensitive information is widespread** [2606.00152]. Across the 10 evaluated agents, **CER** ranges from **6.77%** for Llama-3.2-3B-Instruct to **51.95%** for Claude-Sonnet-4, while **PLR** ranges from **16.67%** to **57.53%**. For every tested agent, probe-elicited leakage exceeds acquisition-stage exposure measured directly at the tool-output level, indicating that latent sensitive context can often be surfaced later even when the main task response appears less revealing [2606.00152].

Task completion and privacy leakage are positively correlated. Across agents, the paper reports **Spearman $\rho=0.818$** between **TCR** and **CER**, and **Spearman $\rho=0.685$** between **TCR** and **PLR** [2606.00152]. This indicates that more capable agents tend, on average, to acquire more out-of-scope information and to disclose more under probing. The relation is not perfectly monotonic: GPT-5.1 achieves **85.70%** TCR with **20.56%** CER and **25.38%** PLR, whereas Claude-Sonnet-4 achieves **86.46%** TCR with **51.95%** CER and **57.53%** PLR [2606.00152]. This suggests that task competence and privacy discipline are related but separable properties.

The most leakage-prone acquisition behaviour is **Out-of-Scope Inferential Access**. The paper reports an average **PLR of about 53.29%** for Task 7, compared with about **9.45%** for the least leakage-prone behaviour [2606.00152]. Domain-level analysis shows especially high leakage in **social media, healthcare, legal, and finance**, whereas **retail, engineering, and logistics** are less leakage-prone on average [2606.00152].

The paper further decomposes leakage into behavioural modes. **Trajectory-mediated leakage** occurs when the agent first acquires sensitive information and then later reveals it; **inference-mediated leakage** occurs when the probe response leaks sensitive material without corresponding prior acquisition in the logged trajectory; and **self-restraint** denotes cases where the agent acquires out-of-scope information but neither includes it in the main answer nor reveals it under probing [2606.00152]. This decomposition matters because agents can reach similar final PLR values through different internal privacy failure modes.

Prompt-level defenses help, but only modestly. On three open-source agents, the privacy-aware system prompt reduces CER from **10.58% to 6.85%** for Qwen3-4B-Instruct, from **15.99% to 9.98%** for Llama-3.1-8B-Instruct, and from **29.44% to 16.75%** for Llama-3.3-70B-Instruct [2606.00152]. The paper emphasizes that this still leaves **more than half of baseline acquisition-stage leakage** in place. Confidentiality markers are even less reliable: on **7 of 10 agents**, such markers increase CER, PLR, or both, apparently acting as saliency cues rather than effective constraints [2606.00152].

## 6. Significance, limitations, and place in the literature

PrivacyPeek’s importance lies in treating **data acquisition itself** as a first-class privacy event [2606.00152]. In benchmark terms, the agent need not disclose sensitive data immediately for privacy risk to materialize; once information is acquired beyond $\mathcal{E}_c$, it becomes available for future action selection, report generation, or adversarial elicitation. This makes PrivacyPeek a benchmark of **data minimization in agent trajectories**, not merely of harmful outputs.

The benchmark also identifies a methodological gap in prior privacy evaluation. The paper positions existing approaches as largely focused on **disclosure-stage leakage**, final responses, sink tools, or broader pipeline-level analysis, while overlooking whether the agent should have pulled the data into context at all [2606.00152]. This suggests that acquisition-stage auditing is complementary to, rather than a replacement for, output-based privacy auditing.

Its limitations are explicit. The dataset is **synthetic and templated**, uses **English-only** instructions, and focuses on **7 behaviours** and **16 domains** rather than the full space of long-horizon, multilingual, or persistent-memory agent deployments [2606.00152]. The mitigation study is restricted to **prompt-level interventions** on a subset of open-source agents. The paper therefore argues for future work on **system-level policy enforcement**, stronger control over tool access, and **training-time privacy alignment**.

This suggests that PrivacyPeek belongs to a broader family of privacy systems that move scrutiny upstream in the processing pipeline: explainability-oriented privacy assistants such as **PEAK** organize privacy decisions around interpretable latent topics [2301.02079], process-level leakage analyzers such as **Pleak** quantify what outputs reveal about inputs in structured workflows [1902.05052], and hub-based architectures such as **Peekaboo** enforce data minimization before cloud transmission [2204.04540]. Within that broader landscape, PrivacyPeek’s specific contribution is to make **acquisition-stage leakage in LLM-based agents** measurable, comparable, and difficult to ignore [2606.00152].

Source: https://www.emergentmind.com/topics/privacypeek