PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

Abstract: LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce \emph{PrivacyPeek}, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with $1{,}182$ cases across $7$ acquisition behaviours and $16$ application domains. Specifically, \emph{Acquisition Inspection} examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. \emph{Probe Elicitation} then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on 10 LLM-based agents across 4 model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.

• The paper presents a novel benchmark, PrivacyPeek, to audit sensitive data acquisition in LLM-based agents using metrics like CER and PLR.
• The study employs a dual-phase evaluation methodology—acquisition inspection and probe elicitation—to systematically assess out-of-scope data collection.
• Findings reveal that higher task performance often correlates with increased privacy leakage and that certain mitigations, like in-data markers, can backfire.

PrivacyPeek: Auditing Data Acquisition in LLM-Based Agents

Motivation and Problem Formulation

Current evaluation paradigms for LLM-based agents emphasize only the privacy risks evident in responses or outgoing actions. This overlooks the acquisition phase, during which agents collect data via tool calls, potentially amassing sensitive information not strictly required by the user instruction. Such unnecessary acquisition violates data-minimization principles and leaves sensitive content within the agent’s context, susceptible to accidental, induced, or adversarial leaks, as shown in (Figure 1).

Figure 1: An LLM-based agent over-acquiring sensitive information beyond the task scope.

The paper introduces the PrivacyPeek benchmark to systematically assess acquisition-stage privacy leakage by LLM-based agents, focusing on what these agents acquire during task execution rather than exclusively on what is ultimately disclosed.

Benchmark Construction and Coverage

PrivacyPeek formalizes "acquisition-stage leakage" as any instance where an agent, via its tool-call trajectory, invokes services or retrieves data extending beyond the declared minimal scope necessary to complete the task. This is instantiated in 1,182 human-verified cases spanning 7 distinct acquisition behaviors and 16 application domains. The behaviors cover attribute-level and content-level violations, including but not limited to: accessing files by misleading names, out-of-window data acquisition, exfiltration of excess fields, and inferential leakage via aggregation.

Figure 2: Overview of PrivacyPeek, detailing acquisition behaviours, dataset generation, and dual-phase evaluation.

PrivacyPeek’s domain coverage is cross-sectoral and balanced by acquisition behavior rather than by domain, with case distributions documented in detail (Figure 3).

Figure 3: Topic distribution of PrivacyPeek cases across the sixteen application domains.

Evaluation Methodology

PrivacyPeek employs two complementary evaluators:

• Acquisition Inspection: Direct auditing of every tool-call and its returned data in the agent’s trajectory to detect whether any out-of-scope sensitive element has been acquired (binary $\mathrm{CER}$ indicator).
• Probe Elicitation: After agent execution, tools and network are disabled and an adversarial probe is issued to determine if previously acquired but undisclosed sensitive information can be elicited (producing $\mathrm{PLR}$ via an LLM-based judge with explicit evaluation criteria).

To ensure robust evaluation, outcome rates such as CER (Content Exposure Rate), TCR (Task Completion Rate), and HCER (Helpful Content Exposure Rate) are tracked, conditioning leakage only on successful task completion.

Experimental Results

Ten agents from four major families (GPT, Claude, Llama, Qwen) are evaluated. The results show that acquisition-stage privacy violations are severe and widespread:

• Maximum CER: $51.95\%$ (Claude-Sonnet-4)
• Maximum PLR: $57.53\%$ (Claude-Sonnet-4)
• Minimum CER/PLR: $6.77\%/16.67\%$ (Llama-3.2-3B-Instruct)

Notably, task capability and privacy leakage are strongly correlated ($\rho=0.818$ for CER and $\rho=0.685$ for PLR), but the relationship is not strict (see GPT-5.1 in Figure 4), indicating that higher-utility agents tend to over-acquire more, but architectural or alignment choices can alter this relationship non-monotonically.

Figure 4: Capability--privacy paradox: higher task-completion rates correlate with more probe-elicitable leakage, but outliers exist.

Domain- and behavior-specific analysis finds that inferential access (Task 7) yields the highest mean PLR ($53.29\%$). Domains such as social media, healthcare, legal, and finance have higher observed risk. The study highlights that risks are not only behavior-specific but also domain-specific, with limited mitigation from in-data confidentiality markers or prompt-level refusals.

Behavioral Analysis

Three canonical agent behaviors are identified:

• Trajectory-mediated leakage: Agent acquires sensitive data and subsequently discloses it.
• Inference-mediated leakage: Agent does not acquire clear sensitive data, but probe-induced disclosure arises from generalization or reasoning.
• Self-restraint: Agent acquires sensitive data but refrains from any form of output leakage.

Closed-source agents demonstrate higher rates of self-restraint relative to open-source models, implying that observed privacy alignment depends less on scale and more on explicit post-training or alignment interventions.

Key findings include:

• Upgrading Claude-Sonnet-4 to 4.5 yields only modest improvement in privacy leakage, most of which is due to increased refusals rather than better internal privacy reasoning (Figure 5).
• Confidentiality markers inside data increase probe-leakage rates for most agents, indicating that such markers attract rather than avert attention (Figure 6).

  Figure 5: Decomposition of the $\mathrm{PLR}$ reduction from Claude-Sonnet-4 to Claude-Sonnet-4.5; most reduction is due to lower task completion.

  

  Figure 6: Per-agent change in $\mathrm{PLR}$ on the marker subset shows that markers often backfire, increasing leakage.

Mitigation Strategies

Prompt-level directives with privacy instructions reduce CER by up to $\mathrm{PLR}$0 but leave most acquisition-stage leakage unresolved. Reductions are consistent but bounded and are less effective than architectural or training-level controls. Importantly, in-data confidentiality markers have counterproductive effects on most systems, reinforcing that robust privacy safeguards require external policy enforcement, fine-tuned alignment, or explicit self-restraint reasoning modules.

Practical and Theoretical Implications

The results demonstrate that a focus solely on outbound actions underestimates privacy risk. The acquisition context is pivotal for minimizing downstream privacy violations—especially under adversarial probing—since one over-acquisition event makes later leakage almost inevitable via accident, inference, or attack. Current evaluations overestimate privacy if acquisition is not separately constrained and audited.

Practically, the findings suggest that future agent deployments, especially in high-stakes domains, must embed acquisition-aware auditing, implement training- or architecture-level privacy alignment, and not rely on simple prompt-level heuristics or in-data markers.

Theoretically, PrivacyPeek delineates a taxonomy for acquisition behaviors, reframes privacy audit terminology for LLM-based agents, and exposes new privacy-capability tradeoffs for future research. Agents must be designed with context discipline—a capacity to minimize loaded sensitive information, not just to withhold it after-the-fact.

Conclusion

PrivacyPeek offers an acquisition-focused benchmark for LLM-based agents, demonstrating that out-of-scope sensitive information acquisition is pervasive. The study unmasks strong correlations between utility and privacy risk, shows prompt-level and in-data mitigations as fundamentally inadequate, and establishes acquisition-inspection as essential for trustworthy agent deployment. Future work should prioritize training-time privacy alignment mechanisms, policy-enforced decision layers, and explorations of reasoning-based self-restraint to mitigate privacy leakage at the source.