MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agents

Abstract: Deep research agents increasingly combine private local documents with external tools like web retrieval, creating a privacy risk: an agent's external queries may leak sensitive information from its local context. This risk is amplified by the mosaic effect, where individual queries may appear harmless but become revealing in aggregate. We introduce MosaicLeaks, a benchmark of 1,001 multi-hop deep research tasks that chain private enterprise documents and a public web corpus, forcing agents to make external queries that depend on local information. We evaluate leakage with an adversary LLM that observes only the agent's external queries and attempts to infer private information at three levels: the agent's research intent, answers to specific private questions and verifiable claims about the enterprise documents. We find that models across families and sizes frequently leak at all three levels, that zero-shot privacy prompting reduces but does not eliminate leakage and that reinforcement learning for task performance alone worsens leakage. To address this, we propose Privacy-Aware Deep Research (PA-DR), an RL framework that combines situational rewards for task success with a learned privacy classifier to provide dense credit assignment over both per-query and mosaic-level leakage. Training Qwen3-4B-Instruct with PA-DR improves accuracy from 48.7% to 58.7% and reduces answer and full-information leakage from 34.0% to 9.9%.

• The paper introduces MosaicLeaks, a framework that operationalizes the mosaic effect to quantify and mitigate privacy leakage in deep research agents.
• It employs a multi-hop task dataset and an adversarial LLM setup to measure leakage across intent, answer, and full-information axes, highlighting the limits of prompt-based mitigation.
• A reinforcement learning algorithm, PA-DR, is proposed to generate less revealing web queries, significantly reducing privacy leakage while enhancing sample efficiency.

Privacy Risks in Querying-in-the-Open for Deep Research Agents: An Analysis of MosaicLeaks

Introduction

MosaicLeaks introduces a comprehensive evaluation framework and dataset for quantifying and mitigating privacy leakage in deep research agents operating with both private enterprise documents and open web retrieval. The Mosaic Effect—the process by which seemingly innocuous external queries, when viewed in aggregate, disclose sensitive enterprise information—is explicitly operationalized, setting the stage for new methods in privacy-preserving agent training. This work goes beyond previous efforts by systematically measuring multiple privacy leakage axes, exposing the limits of prompting-based mitigation, and offering a reinforcement learning algorithm—Privacy-Aware Deep Research (PA-DR)—that provides strong empirical improvements in both downstream accuracy and privacy protection (2605.30727).

Defining the Mosaic Effect and Privacy Leakage Axes

MosaicLeaks conceptualizes privacy leakage through three axes of increasing severity: Intent Leakage (adversary infers the agent's goals), Answer Leakage (adversary can answer specific private questions using only observed web queries), and Full-Information Leakage (adversary asserts concrete, verifiable private facts without prompts).

Figure 1: Illustration of the Mosaic Effect; a series of dependent queries incrementally reveals private enterprise facts, such as Lee's Market's 15% online traffic growth in 2020.

Crucially, the dataset's multi-hop task construction and adversary setup approximate enterprise deployment conditions where an external observer can only view the agent's sequence of web queries, not its local reasoning or retrieval steps.

MosaicLeaks Benchmark Construction

The MosaicLeaks benchmark consists of 1,001 multi-hop question chains, synthesized to maximize interleaving between local (enterprise) and web sources, thus promoting information dependencies that heighten privacy risk. Each task is a chain of sub-questions, alternating between local and web pivots, with subsequent hops conditioned on answers found in prior hops. This design explicitly induces situations where web queries are grounded in private knowledge—a precondition for mosaic leakage.

An adversarial LLM is employed to quantify leakage by attempting to infer the agent's intent, retrieve private answers given only the queries, and generate independent, factual claims about enterprise documents. The adversary's performance on these axes becomes the empirical measure of privacy risk for each agent.

Baseline Agents, Privacy Prompting, and Core Empirical Findings

Initial results across a range of open- and closed-source LLMs (Qwen3-4B-Instruct, Qwen3-8B, StepFun-3.5-Flash, GPT-OSS-20B, Chroma Context-1, Gemma 4 31B IT) indicate that privacy leakage is significant for all families and model sizes, even before task performance fine-tuning.

The prevalent mitigation strategy—prompting agents with a privacy warning at web-query generation time—was tested directly. While this reduces leakage rates modestly, it fails to provide robust privacy protection and mainly achieves its effect by reducing the number of web queries rather than by encouraging less informative query formulations.

Figure 2: Comparison of strict chain success and privacy leakage for agents with and without a privacy-aware prompt; minor reduction in leakage does not eliminate vulnerability.

Furthermore, outcome-driven RL training designed solely to improve research performance was found to increase privacy leakage substantially: RL-tuned agents generated more queries and reads, systematically exposing more private information to the adversary, while achieving higher accuracy on downstream tasks.

Agentic Behavior Analysis

MosaicLeaks also investigates agentic behavior at the trajectory level. Chains in MosaicLeaks require elaborate, tool-mediated, multi-stage decision-making spanning both local and web retrieval at each hop.

Figure 3: MosaicLeaks agent trajectories exemplify multiple dependent hops, each comprising planning, retrieval, selection, reading, and resolution, iterating over both local and web sources.

Behavioral analyses indicate that, while naive privacy prompting leads to a reduction in total web queries, it does not incentivize agents to phrase their queries in less revealing ways. Instead, it induces superficial query suppression, with local document utilization remaining flat.

Figure 4: Privacy-aware prompting only modestly decreases web queries; local accesses do not compensate, which limits the effectiveness of this mitigation.

Privacy-Aware Deep Research (PA-DR): Joint Performance and Privacy Optimization

To move beyond prompt engineering, MosaicLeaks formulates privacy-aware agent training as a reinforcement learning problem with a joint objective. The proposed PA-DR algorithm employs a dense credit assignment mechanism: task rewards are assigned situationally (i.e., at the Plan and Choose stages, exclusively when the correct document source or gold document is accessible), while a learned privacy classifier penalizes specific Plan stages—those where web queries reveal private facts, either directly or via mosaic composition with prior queries.

The privacy classifier is trained as a binary model using both whole and partial trajectories, and provides rewards that specifically target the incremental and compositional nature of mosaic leakage—aligned with the observable adversary described in the benchmark.

Empirically, situational RL rewards in PA-DR yield strongly improved sample efficiency and effectiveness compared to traditional outcome-level RL, with up to 5–6× fewer samples needed for comparable downstream accuracy.

Figure 5: Situational RL rewards accelerate convergence and reduce token costs; outcome-reward-only RL exacerbates privacy leakage.

Empirical Results: Advancing the Pareto Frontier

PA-DR-trained agents demonstrate significant improvements:

• Accuracy: Strict chain accuracy reaches $58.7\%$ (compared to $48.7\%$ for the base and $59.3\%$ for task-only RL).
• Privacy Leakage: Combined Answer and Full-Information Leakage drops from $34.0\%$ in the base to $9.9\%$ with PA-DR, and further to $7.6\%$ when combined with privacy prompting.
• PA-DR achieves these gains without reducing query frequency; rather, the trained agent internalizes how to craft web queries that are less semantically revealing, while maintaining informativeness necessary to answer the multi-hop task.

This approach outperforms both prompting and naive RL, achieving a new privacy/utility Pareto frontier in deep research agent training.

Figure 6: Over the RL training trajectory, PA-DR simultaneously increases chain success and reduces privacy leakage; task-only RL increases leakage before late, modest improvements.

Post-training query analysis supports the claim that privacy-preservation is achieved through less informative queries (as measured by adversarial reconstructions) rather than crude output suppression.

Figure 7: After PA-DR training, the number of web and local queries increases; privacy gains are achieved not by reducing query volume, but via less information-dense web queries.

Detailed Effects Across Privacy Leakage Axes

Different forms of leakage (intent, answer, full-information) show similar trends: privacy prompting yields only minor improvement across leakage types, while PA-DR drastically reduces the most severe forms (answer, full-information leakage), aligning practical risk mitigation with enterprise requirements.

Figure 8: Privacy prompting provides modest, systematic reduction across all leakage axes, but does not close the gap; PA-DR provides step-function improvement, especially in the critical Answer and Full-Information categories.

Implications and Future Directions

MosaicLeaks exposes the limitations of prompt-based privacy interventions—particularly for compositional, mosaic-style attacks—and provides rigorous evidence that RL-based, credit-assigned, privacy-preserving policies are a feasible and necessary evolution for enterprise research agents. The explicit operationalization of privacy as answer and fact leakage, measured against specific adversarial capabilities, enables more faithful estimation of enterprise risks.

Practically, these results necessitate training regimes that explicitly incorporate privacy leakage rewards. The PA-DR methodology, with task-aligned and privacy-aligned reward shaping at the action level, points to new agent architectures that can be both high-performing and privacy-respecting without requiring user-side prompt engineering.

While the current dataset is limited in enterprise context diversity and task types (focused on QA rather than long-form report synthesis), MosaicLeaks provides a critical benchmark infrastructure and modeling template. Future work should expand context domains, adversary modeling, and compositional task diversity, as well as address generalization and alignment issues for agents trained in even more heterogeneous environments.

Conclusion

MosaicLeaks delivers a robust framework and dataset for evaluating and mitigating privacy leakage in deep research agents. Its identification of the mosaic effect as a central privacy threat, empirical debunking of prompt-based mitigations, and introduction of PA-DR as an RL-based privacy shield establish it as a foundational methodology for the next generation of enterprise-capable LLM agents. The findings decisively argue for the necessity of RL-based privacy-aware objectives as standard protocol in overlapping local-web agentic systems.