---
title: 'MosaicLeaks: Privacy Leakage Benchmark'
url: https://www.emergentmind.com/topics/mosaicleaks
type: topic
---

# MosaicLeaks: Privacy Leakage Benchmark

MosaicLeaks is a benchmark and evaluation framework for privacy leakage in deep research agents that combine private local documents with external tools such as web retrieval. The central claim is that querying-in-the-open creates a distinct privacy risk: external queries may expose sensitive information drawn from local enterprise context, and this risk is amplified by the mosaic effect, in which individually innocuous queries become revealing in aggregate. The benchmark introduces 1,001 multi-hop deep research tasks that chain private enterprise documents and a public web corpus, and it evaluates whether an adversary that observes only the agent’s external queries can infer private information at the level of research intent, answers to specific private questions, or verifiable claims about enterprise documents [2605.30727].

## 1. Conceptual basis and threat model

MosaicLeaks is organized around a specific failure mode of tool-using agents: the dependence of external querying behavior on private local state. In the setting studied, a deep research agent must solve enterprise research tasks by interleaving access to private local documents with access to public web resources. Because the web queries are generated conditionally on what the agent learns from local documents, those queries can leak internal facts even when the agent never directly outputs the private documents themselves [2605.30727].

The benchmark frames this risk through the mosaic effect. In privacy theory, the mosaic effect refers to the phenomenon whereby multiple individually harmless pieces of information can be aggregated to reveal sensitive facts not apparent in isolation. Applied to agent systems, the relevant object is not a single query but the sequence of external queries over a trajectory. An illustrative example uses the query sequence `Lee's Market 2020 traffic growth`, `Lee's Market online traffic growth 2020`, and, after local retrieval, `Which stores reported 15% online traffic growth in 2020?`. None of the first two queries alone discloses that Lee’s Market grew 15% in 2020, but the third query allows an observer to infer that the internally recovered answer was 15% [2605.30727].

This threat model is narrower than generic model memorization and broader than single-query privacy auditing. It concerns information flow from private enterprise context into public tool calls during task execution. A plausible implication is that conventional output-side safety checks are insufficient if they do not reason over multi-step querying behavior.

## 2. Benchmark construction and task structure

MosaicLeaks is a benchmark of 1,001 chains or tasks and 3,403 hops or questions. The tasks are constructed as multi-hop chains of length 2–7 hops, with average length approximately 3.4, and they interleave local private enterprise documents with public web documents. The construction uses a graph-based approach inspired by InfoSeeker and WebShaper: entities and answers from one hop are reused as bridge entities in the next hop, increasing dependency across steps and thereby increasing opportunities for privacy leakage [2605.30727].

The local corpus comes from DRBench, described as enterprise synthetic documents, while the public corpus comes from BrowseComp-Plus. Each task requires sequential reasoning across both corpora, including retrieval, reading, and synthesis across private and public sources. The intended scenario is an enterprise research workflow in which sensitive internal information is needed to shape external information-gathering [2605.30727].

The benchmark statistics reported in the paper are:

| Quantity | Value |
|---|---:|
| Chains / tasks | 1,001 |
| Hops / questions | 3,403 |
| Local documents | ~1,238 |
| Private question-answer pairs | 3,686 |

These design choices make the benchmark structurally different from isolated QA or retrieval tasks. The chaining mechanism forces agents to condition later external queries on earlier local findings, which is precisely where mosaic-level leakage becomes measurable.

## 3. Leakage taxonomy and evaluation protocol

Leakage is measured using an adversary LLM, StepFun-3.5-Flash, that observes only the sequence of the agent’s external web queries. The adversary does not see the local documents, the agent’s hidden state, or the final internal reasoning trace. A judge model, again StepFun-3.5-Flash, evaluates the adversary’s outputs against ground truth [2605.30727].

The benchmark defines three levels of privacy leakage.

**Intent leakage** is the weakest level. The question is whether the adversary can predict what the agent was researching from the query sequence alone.

**Answer leakage** is an intermediate level. The adversary is prompted with specific private questions about enterprise documents and must answer them using only the observed query sequence. The paper notes that this setting may overestimate adversary power, but still treats it as realistic.

**Full-information leakage** is the strongest level. The adversary must produce verifiable factual claims about the private documents without being prompted with the exact private questions [2605.30727].

This three-tier taxonomy is analytically useful because it separates coarse topical disclosure from direct factual disclosure. It also guards against the misconception that privacy leakage is exhausted by explicit answer recovery. In MosaicLeaks, an agent may leak at the level of intent even when exact answer extraction is harder, and it may leak full factual claims without any prompt specifying the target question.

## 4. Empirical findings

The reported empirical result is that models across families and sizes frequently leak at all three levels. The experiments cover six models: Qwen3-4B-Instruct, Qwen3-8B, StepFun-3.5-Flash, GPT-OSS-20B, Chroma Context-1, and Gemma4-31B-IT. The paper’s central qualitative findings are that zero-shot privacy prompting reduces but does not eliminate leakage, and that reinforcement learning optimized only for task performance worsens leakage [2605.30727].

For Qwen3-4B-Instruct, the paper reports the following results:

| Model / setup | Strict chain success | Privacy leakage |
|---|---:|---:|
| Qwen3-4B-Instruct | 48.7% | 34.0% |
| + privacy prompt | 44.5% | 25.5% |
| Task RL (perf.-only) | 59.3% | 51.7% |
| Task + PA-DR (joint obj.) | 58.7% | 9.9% |
| PA-DR + privacy prompt | 59.3% | 7.6% |

The finer-grained leakage breakdown for the baseline Qwen3-4B-Instruct model is 51.1% intent leakage, 18.7% answer leakage, and 11.7% full-information leakage. Under privacy prompting, these become 39.2%, 14.4%, and 7.9%, respectively. Under task-only RL, they increase to 63.5%, 28.5%, and 17.1%. Under Task + PA-DR, they fall to 29.8%, 3.3%, and 1.7%; under PA-DR + privacy prompt, they become 23.4%, 4.0%, and 1.1% [2605.30727].

These results support two nontrivial conclusions. First, simple privacy warnings reduce leakage largely by reducing web querying, but this reduction does not robustly solve the information-flow problem. Second, optimizing only for task success encourages agents to externalize more internal information in their queries, which increases what an observer can reconstruct from the overall query sequence.

## 5. Privacy-Aware Deep Research (PA-DR)

To mitigate leakage, the paper proposes Privacy-Aware Deep Research (PA-DR), a reinforcement learning framework that jointly optimizes task performance and privacy preservation. The framework combines situational rewards for task success with a learned privacy classifier that provides dense credit assignment over both per-query leakage and mosaic-level leakage [2605.30727].

The privacy classifier is a Qwen3-4B-Instruct classifier trained on approximately 27k examples. Its role is to predict, at each Plan step where web queries are made, whether a given web-query batch leaks private information. On held-out test data, the classifier achieves ROC AUC \(= 0.878\), precision \(62.5\%\), and recall \(76.3\%\) [2605.30727].

For each batch of web queries \(w_i\), the paper defines direct leakage as

$$
c_{\text{direct}} = \max(0, P(w_i) - \tau),
$$

where \(P(w_i)\) is the classifier’s predicted leakage probability and \(\tau = 0.5\).

Mosaic leakage is defined as

$$
c_{\text{mosaic}} = \max\!\bigl(0, P(\mathcal{W}_i) - \max(P(\mathcal{W}_i \setminus w_i), \tau)\bigr),
$$

where \(\mathcal{W}_i\) is the context window including previous and current web-query batches.

The final privacy reward is

$$
r_{\text{privacy}} = -\max(c_{\text{direct}}, c_{\text{mosaic}}).
$$

These privacy rewards are combined with stage-specific task rewards. For the plan stage, the paper gives the following example:

$$
r_{\mathrm{plan}} =
\begin{cases}
1.25 & \text{if gold doc. not yet retrieved, retrieved this step} \\
1.00 & \text{if already retrieved, and stopped searching} \\
0.25 & \text{if correct type but not retrieved} \\
-1 & \text{unparseable output} \\
0 & \text{otherwise}
\end{cases}
$$

The methodological significance of PA-DR is its use of dense, situational feedback for long-horizon agent trajectories. Outcome-only privacy optimization would be sparse and noisy; PA-DR instead attributes leakage risk to specific query batches and to their incremental contribution to the mosaic effect.

## 6. Interpretation, limitations, and relation to similarly named work

The paper’s broader implication is that privacy should be treated as a first-class optimization objective in deep research agents. The benchmark indicates that multi-hop, interleaved local-external tasks significantly increase leakage risk in practical enterprise settings, and that performance maximization alone is dangerous because it tends to externalize more private facts through external search behavior [2605.30727].

A common misconception is that privacy prompting is an adequate fix. The reported results do not support that view: prompting lowers leakage but does not eliminate it, and it may reduce agent effectiveness. Another misconception is that leakage must appear in a single obviously revealing query. MosaicLeaks is built precisely to show the opposite: the revealing signal may emerge only when the query sequence is interpreted in aggregate.

The name should also be distinguished from other arXiv works that use related “Mosaic” or “MOSAIC” terminology for unrelated problems, including multimodal jailbreaks against closed-source VLMs [2604.09253], compositional vulnerability induction in coding agents [2605.03952], and CLI command-composition attacks in coding agents [2607.02857]. Those works also study multi-step compositional failure modes, but they address different attack surfaces and should not be conflated with privacy leakage in querying-in-the-open.

Within agent security research, MosaicLeaks suggests a shift from single-prompt or single-action auditing toward trajectory-level analysis of information disclosure. A plausible implication is that privacy-preserving tool use will require both benchmarked evaluation and training-time incentives that explicitly penalize query-level and sequence-level leakage, rather than assuming that generic alignment or task-only optimization will produce private behavior by default.

Source: https://www.emergentmind.com/topics/mosaicleaks