PrivacyPeek: Benchmark for Privacy Leakage in LLM Agents

Updated 6 July 2026

PrivacyPeek is a benchmark for auditing acquisition-stage privacy leakage in LLM-based agents by tracking tool-call trajectories and out-of-scope data acquisition.
It categorizes 7 distinct acquisition behaviors across 16 domains and employs both acquisition inspection and probe elicitation to measure privacy risk.
Empirical findings reveal that more capable agents tend to acquire and later expose sensitive data, underscoring the need for stronger privacy mitigation strategies.

PrivacyPeek is a benchmark for auditing acquisition-stage privacy leakage in LLM-based agents: it measures not only whether an agent discloses sensitive information in its final answer, but whether the agent acquires sensitive information beyond the task scope while invoking tools during multi-step execution (Zhang et al., 29 May 2026). The benchmark formalizes a case as $c=\langle q_c, T_c, F_c, \mathcal{E}_c, p_c\rangle$ , where $q_c$ is the user instruction, $T_c$ the available tool set, $F_c$ the underlying files or records, $\mathcal{E}_c$ the minimum scope of personal data needed for task completion, and $p_c$ a follow-up privacy probe (Zhang et al., 29 May 2026). In this formulation, privacy risk begins when unnecessary data enters the agent’s context, because that data is then available for subsequent reasoning, accidental disclosure, or elicitation by an attacker.

1. Conceptual scope

PrivacyPeek distinguishes acquisition-stage leakage from disclosure-stage leakage. In the benchmark’s agent setting, execution produces a tool-call trajectory

$\tau_c=(a_1,o_1,a_2,o_2,\ldots,a_{n-1},o_{n-1},a_n),$

where each $a_t$ for $t<n$ is a tool invocation, each $o_t$ is the corresponding tool output, and $q_c$ 0 is the final natural-language task response (Zhang et al., 29 May 2026). Traditional privacy evaluations concentrate on what appears in $q_c$ 1 or in outbound actions. PrivacyPeek instead asks whether any $q_c$ 2 contains personal data outside $q_c$ 3.

The benchmark defines a sensitive set $q_c$ 4 for each case: these are the personal-data items that lie outside the minimum task scope but remain privacy-relevant under norms such as GDPR’s data minimization principle and HIPAA’s “minimum necessary” standard (Zhang et al., 29 May 2026). A trajectory exhibits acquisition-stage leakage if at least one tool call acquires any member of $q_c$ 5.

This framing shifts privacy analysis from final outputs to context formation. An agent may appear safe if its final answer omits sensitive details, yet still be unsafe if it has already loaded those details from files, databases, or records. PrivacyPeek operationalizes that distinction by auditing both the agent’s tool-call history and its later susceptibility to elicitation (Zhang et al., 29 May 2026).

2. Benchmark construction and taxonomy

PrivacyPeek contains 1,182 cases, spanning 7 acquisition behaviours and 16 application domains (Zhang et al., 29 May 2026). The domains include healthcare, finance, legal services, education, government or public service, research, HR, customer support, social media, technology, retail, engineering, and logistics. The benchmark is organized by acquisition behaviour rather than by domain, so each behavioural pattern is systematically represented (Zhang et al., 29 May 2026).

The seven behaviours are divided into two broad groups. The first group concerns violations in file attributes or retrieval scope: Normal-Filename Access, Sensitive-Filename Access, Cross-Format Access, and Out-of-Window Access. The second group concerns violations in returned content: Excess-Field Access, Forbidden-Content Access, and Out-of-Scope Inferential Access (Zhang et al., 29 May 2026). These categories distinguish, for example, opening an irrelevant file with an ordinary name from opening one whose filename itself advertises sensitivity, or reading an allowed record that contains extra fields from combining otherwise innocuous fields into a sensitive inference.

Acquisition behaviour	Cases
Normal-Filename Access	217
Sensitive-Filename Access	158
Cross-Format Access	145
Out-of-Window Access	138
Excess-Field Access	218
Forbidden-Content Access	163
Out-of-Scope Inferential Access	143

Case generation follows a human-in-the-loop pipeline. Template seeds define the behavioural structure; GPT-4o expands them into full cases containing user instructions, system prompts, files, records, and probes; and the authors then apply 10 deterministic quality gates plus human inspection (Zhang et al., 29 May 2026). The quality gates enforce complete schema coverage, one-and-only-one target behaviour per case, feasibility of the task without violating scope, uniqueness of ground-truth sensitive items, and probe specificity. Only cases passing all gates are retained.

This design makes PrivacyPeek a benchmark of minimum-scope compliance rather than merely a benchmark of overt disclosure. A case can therefore be privacy-violating even when the agent finishes the user task correctly and says nothing obviously sensitive in its main answer (Zhang et al., 29 May 2026).

3. Auditing methodology

PrivacyPeek consists of two coupled auditing procedures: Acquisition Inspection and Probe Elicitation (Zhang et al., 29 May 2026).

Acquisition Inspection examines the tool-call trajectory directly. For a case $q_c$ 6, the benchmark assigns case-level content exposure when any sensitive item in $q_c$ 7 appears in any intermediate tool output:

$q_c$ 8

The corpus-level Content Exposure Rate is then

$q_c$ 9

with $T_c$ 0 (Zhang et al., 29 May 2026). Since a trivial agent that never completes tasks could reduce exposure simply by failing, the benchmark also measures Task Completion Rate and defines Helpful Content Exposure Rate

$T_c$ 1

thereby conditioning leakage on successful task completion (Zhang et al., 29 May 2026).

Probe Elicitation models an attacker who interacts with the agent after the original task is complete. In the first phase, the agent executes the task with tools enabled. In the second phase, tools and network access are disabled, and the evaluator submits a case-specific follow-up probe $T_c$ 2 designed to elicit out-of-scope sensitive information that may already reside in context (Zhang et al., 29 May 2026). Leakage in the main response is measured as Output Leakage Rate; leakage in the probe response is measured as Probe Leakage Rate, with a corresponding helpful variant,

$T_c$ 3

For open-ended text judgments, the benchmark uses GPT-4o as a judge. The judge receives the task context, the sensitive items, the task response, and the probe response, and determines whether a subject and a sensitive attribute are disclosed together through verbatim mention, paraphrase, or explicit inference (Zhang et al., 29 May 2026). Pure refusal, wrong or hallucinated values, generic category statements, or descriptions of tool use without sensitive content do not count as disclosure.

4. Experimental environment and evaluated agents

All experiments use Smolagents CodeAgent as the execution framework (Zhang et al., 29 May 2026). The agent emits Python code, can invoke up to 8 tools, and is limited to 6 steps per case. The tool surface includes query_database, read_document, search_records, list_files, get_file_properties, write_note, send_message, and export_report (Zhang et al., 29 May 2026). File-system access is restricted to those tools, so the benchmark audits the acquisition channel through explicit tool use rather than unconstrained local I/O.

The baseline system prompt identifies the agent as a domain assistant and lists the tools, but it does not include privacy instructions (Zhang et al., 29 May 2026). This makes the base evaluation one of unconstrained agentic behavior under benign user requests and uncompromised tools.

PrivacyPeek evaluates 10 LLM-based agents across 4 model families (Zhang et al., 29 May 2026). The closed-source agents are GPT-4.1, GPT-5.1, Claude-Sonnet-4, and Claude-Sonnet-4.5. The open-source agents are Qwen3-4B-Instruct, Qwen3-14B, Qwen3-30B-A3B, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.3-70B-Instruct. Each is run once per case in the main evaluation.

The benchmark also tests prompt-level mitigation. A privacy-aware system prompt instructs the agent not to expose sensitive categories such as personal identifiers, health information, finance, and behavioral logs unless explicitly authorized (Zhang et al., 29 May 2026). In a separate intervention, the benchmark inserts confidentiality markers such as CONFIDENTIAL: <ROLE> ONLY into the data substrate to test whether explicit in-data warnings reduce acquisition or disclosure.

5. Empirical findings

The central empirical result is that unnecessary acquisition of sensitive information is widespread (Zhang et al., 29 May 2026). Across the 10 evaluated agents, CER ranges from 6.77% for Llama-3.2-3B-Instruct to 51.95% for Claude-Sonnet-4, while PLR ranges from 16.67% to 57.53%. For every tested agent, probe-elicited leakage exceeds acquisition-stage exposure measured directly at the tool-output level, indicating that latent sensitive context can often be surfaced later even when the main task response appears less revealing (Zhang et al., 29 May 2026).

Task completion and privacy leakage are positively correlated. Across agents, the paper reports Spearman $T_c$ 4 between TCR and CER, and Spearman $T_c$ 5 between TCR and PLR (Zhang et al., 29 May 2026). This indicates that more capable agents tend, on average, to acquire more out-of-scope information and to disclose more under probing. The relation is not perfectly monotonic: GPT-5.1 achieves 85.70% TCR with 20.56% CER and 25.38% PLR, whereas Claude-Sonnet-4 achieves 86.46% TCR with 51.95% CER and 57.53% PLR (Zhang et al., 29 May 2026). This suggests that task competence and privacy discipline are related but separable properties.

The most leakage-prone acquisition behaviour is Out-of-Scope Inferential Access. The paper reports an average PLR of about 53.29% for Task 7, compared with about 9.45% for the least leakage-prone behaviour (Zhang et al., 29 May 2026). Domain-level analysis shows especially high leakage in social media, healthcare, legal, and finance, whereas retail, engineering, and logistics are less leakage-prone on average (Zhang et al., 29 May 2026).

The paper further decomposes leakage into behavioural modes. Trajectory-mediated leakage occurs when the agent first acquires sensitive information and then later reveals it; inference-mediated leakage occurs when the probe response leaks sensitive material without corresponding prior acquisition in the logged trajectory; and self-restraint denotes cases where the agent acquires out-of-scope information but neither includes it in the main answer nor reveals it under probing (Zhang et al., 29 May 2026). This decomposition matters because agents can reach similar final PLR values through different internal privacy failure modes.

Prompt-level defenses help, but only modestly. On three open-source agents, the privacy-aware system prompt reduces CER from 10.58% to 6.85% for Qwen3-4B-Instruct, from 15.99% to 9.98% for Llama-3.1-8B-Instruct, and from 29.44% to 16.75% for Llama-3.3-70B-Instruct (Zhang et al., 29 May 2026). The paper emphasizes that this still leaves more than half of baseline acquisition-stage leakage in place. Confidentiality markers are even less reliable: on 7 of 10 agents, such markers increase CER, PLR, or both, apparently acting as saliency cues rather than effective constraints (Zhang et al., 29 May 2026).

6. Significance, limitations, and place in the literature

PrivacyPeek’s importance lies in treating data acquisition itself as a first-class privacy event (Zhang et al., 29 May 2026). In benchmark terms, the agent need not disclose sensitive data immediately for privacy risk to materialize; once information is acquired beyond $T_c$ 6, it becomes available for future action selection, report generation, or adversarial elicitation. This makes PrivacyPeek a benchmark of data minimization in agent trajectories, not merely of harmful outputs.

The benchmark also identifies a methodological gap in prior privacy evaluation. The paper positions existing approaches as largely focused on disclosure-stage leakage, final responses, sink tools, or broader pipeline-level analysis, while overlooking whether the agent should have pulled the data into context at all (Zhang et al., 29 May 2026). This suggests that acquisition-stage auditing is complementary to, rather than a replacement for, output-based privacy auditing.

Its limitations are explicit. The dataset is synthetic and templated, uses English-only instructions, and focuses on 7 behaviours and 16 domains rather than the full space of long-horizon, multilingual, or persistent-memory agent deployments (Zhang et al., 29 May 2026). The mitigation study is restricted to prompt-level interventions on a subset of open-source agents. The paper therefore argues for future work on system-level policy enforcement, stronger control over tool access, and training-time privacy alignment.

This suggests that PrivacyPeek belongs to a broader family of privacy systems that move scrutiny upstream in the processing pipeline: explainability-oriented privacy assistants such as PEAK organize privacy decisions around interpretable latent topics (Ayci et al., 2023), process-level leakage analyzers such as Pleak quantify what outputs reveal about inputs in structured workflows (Toots et al., 2019), and hub-based architectures such as Peekaboo enforce data minimization before cloud transmission (Jin et al., 2022). Within that broader landscape, PrivacyPeek’s specific contribution is to make acquisition-stage leakage in LLM-based agents measurable, comparable, and difficult to ignore (Zhang et al., 29 May 2026).