LeakDojo: RAG Leakage Evaluation Benchmark

• LeakDojo is a configurable framework for evaluating leakage in retrieval-augmented generation systems by simulating black-box attacks aimed at reproducing retrieved database chunks.
• It decomposes leakage into retrieval coverage and leak-triggering behavior, enabling systematic comparisons across multiple attacks, LLMs, datasets, and RAG architectures.
• The framework employs metrics like ARC, SLT, CCL, and CRR to quantify leakage, revealing a trade-off between system faithfulness and privacy risk.

LeakDojo is a configurable framework for controlled evaluation of leakage in retrieval-augmented generation (RAG) systems, centered on the problem of extracting hidden knowledge-base content through black-box interaction with a deployed RAG interface (Zhang et al., 7 May 2026). In this setting, the protected asset is the knowledge database, especially the retrieved chunks injected into the generator context during inference. LeakDojo models leakage attacks as attempts to induce near-verbatim reproduction of those chunks in final responses, and it evaluates them under a unified, modular methodology spanning attack strategy, RAG architecture, and defense configuration. Its main contribution is not a new single attack, but a benchmark structure that decomposes leakage into retrieval coverage and leak-triggering behavior, enabling controlled comparisons across six attacks, fourteen LLMs, four datasets, and multiple RAG pipelines (Zhang et al., 7 May 2026).

1. Conceptual scope and threat model

LeakDojo studies RAG leakage attacks, meaning attacks that try to steal content from the underlying RAG database by interacting only through the normal user interface (Zhang et al., 7 May 2026). The paper defines leakage operationally as reproduction of retrieved chunks—often near-verbatim—in the model’s final answer. This places the emphasis on extraction of hidden retrieved context rather than on model training-data memorization or system-prompt disclosure.

The threat model is explicitly black-box. The attacker can send queries and observe final responses, but does not know the exact RAG structure or configuration. The attacker’s goal is to maximize the number of unique chunks leaked from the knowledge database within a fixed interaction budget of $N$ rounds. The protected object is therefore the knowledge database $\mathcal{D}$ and, more concretely, the retrieved chunk sets $\mathcal{C}_i$ surfaced internally during inference (Zhang et al., 7 May 2026).

A central premise of the framework is that modern RAG systems are not limited to a simple retrieve-then-generate stack. They may include rewriters, rerankers, summarizers, and defense modules, while backend LLMs vary widely in instruction-following behavior. LeakDojo treats these components as first-class experimental variables rather than background implementation details. This makes the framework useful for auditing how architectural improvements interact with leakage risk, rather than merely asking whether leakage is possible in principle (Zhang et al., 7 May 2026).

A common misconception is that LeakDojo is primarily a prompt-injection benchmark. The paper situates leakage attacks within the broader prompt-injection literature, but its core security objective is narrower: extraction of retrieved document content from RAG databases. Another misconception is that it measures only the final output channel in the abstract. In practice, it measures the relationship between adversarial queries, retrieved chunk sets, and final responses, so its notion of leakage is specifically tied to content copied from retrieval context (Zhang et al., 7 May 2026).

2. System architecture and attack decomposition

LeakDojo is organized around a modular interaction loop between an attacker and a defensive RAG system. On the attacker side, each attack is factored into two components: a query generator and an adversarial instruction. The adversarial query at round $i$ is formalized as

$Q^{\text{adv}}_i = A_i \oplus I$

where $A_i$ is the anchor query and $I$ is a fixed adversarial instruction appended to it (Zhang et al., 7 May 2026).

This decomposition is one of the framework’s main analytical moves. The query generator governs how effectively the attacker probes the knowledge base and reaches fresh chunks, while the adversarial instruction governs how effectively the backend LLM is induced to reveal what was retrieved. The paper later shows that this separation is empirically meaningful rather than merely notational.

On the RAG side, a query can pass through several configurable stages. A query rewriter may first generate multiple rewritten queries,

$Q_n = \{Q_i\}_{i=1}^n$

after which a retriever produces initial candidate chunks, an optional reranker reorders them, an optional summarizer compresses or restructures them into final context $C_{\text{final}}$, and the generator LLM produces a response conditioned on that context and the query (Zhang et al., 7 May 2026). LeakDojo also models two defense points: an input-stage intent detector, which can block or clean suspicious queries, and an output-stage content detector, which blocks answers overly similar to retrieved context.

This modularization makes three kinds of comparison possible. First, the same attack can be run against multiple RAG architectures. Second, the same RAG system can be stressed with multiple attack components. Third, defenses can be inserted without changing the rest of the evaluation. This suggests a general benchmark philosophy in which leakage is treated as a property of a full pipeline rather than a single model call.

3. Attack taxonomy and benchmark configuration

LeakDojo benchmarks six representative leakage attacks from prior work: TGTB, PIDE (implemented as GEN-PIDE because the original query set was unavailable), DGEA, RAG-Thief, PoR, and IKEA (Zhang et al., 7 May 2026). The framework treats these not as incomparable bespoke systems, but as different instantiations of the common $A_i \oplus I$ attack structure.

Static attacks include TGTB and GEN-PIDE, whose queries do not depend on previous outputs. Interactive attacks include PoR, RAG-Thief, IKEA, and DGEA, which refine or adapt queries across rounds (Zhang et al., 7 May 2026). The distinction matters because one class emphasizes broad retrieval exploration and the other can exploit feedback from prior leakage.

Several attack profiles are especially important. DGEA uses a jailbreak-style instruction and often attains high leak-triggering ability but weak corpus coverage. PoR emphasizes adaptive keyword exploration and often performs strongly because it balances retrieval exploration with extraction prompting. RAG-Thief is feedback-guided and can outperform more explicit instruction-based attacks on some models. IKEA is associated with benign-looking or “silent” extraction queries, though in LeakDojo it is also re-expressed through the common instruction-augmented formulation (Zhang et al., 7 May 2026).

The experimental matrix spans fourteen LLMs, four datasets, and several RAG configurations. The four datasets are SciFact, NFcorpus, Enron Email, and FiQA. The benchmark uses bge-large-en-v1.5 as embedding model, Chroma as vector store, cosine similarity, and MMR retrieval rather than KNN, with retriever settings top_k = 10, fetch_k = 40, score_threshold = 0.75, and top_n = 5 (Zhang et al., 7 May 2026). The reranker is bge-reranker-large, while both rewriter and summarizer are LLM-based with gpt-4.1-mini as backend.

The standard attack budget is

$\mathcal{D}$0

which the paper justifies as large enough to differentiate attack effectiveness while remaining realistic for black-box extraction and computationally manageable (Zhang et al., 7 May 2026). An additional long-horizon PoR analysis up to $\mathcal{D}$1 shows saturation with diminishing returns.

One implementation choice is methodologically significant: the knowledge base is constructed from distinct, non-overlapping, naturally separated text segments, rather than sliding windows. This avoids overlap artifacts in leakage accounting and makes chunk-level leakage metrics more interpretable (Zhang et al., 7 May 2026).

4. Leakage metrics and measurement methodology

LeakDojo uses four core metrics. Their roles are complementary rather than redundant.

Metric	Definition	Measures
ARC	$\mathcal{D}$2	Retrieval coverage
SLT	$\mathcal{D}$3	Leak-triggering success
CCL	$\mathcal{D}$4	End-to-end cumulative leakage
CRR	Average contiguous-token recovery over successful queries	Verbatim recovery quality

The Adversarial Retrieval Coverage (ARC) metric is

$\mathcal{D}$5

and measures how many unique chunks the attacker manages to retrieve across all rounds (Zhang et al., 7 May 2026). It is therefore a query-generation metric: a high ARC means the attacker is exploring the corpus effectively.

The Successful Leak Trigger (SLT) metric is

$\mathcal{D}$6

with $\mathcal{D}$7 defined as ROUGE-L recall and threshold $\mathcal{D}$8 (Zhang et al., 7 May 2026). A query counts as a successful trigger if at least one retrieved chunk appears in the answer with sufficiently high recall.

The main end-to-end metric is Chunk Cumulative Leakage (CCL). First define the leaked chunk set for round $\mathcal{D}$9 as

$\mathcal{C}_i$0

Then

$\mathcal{C}_i$1

CCL is the fraction of unique chunks leaked across all attack rounds relative to the ideal maximum $\mathcal{C}_i$2 (Zhang et al., 7 May 2026). This is the paper’s primary measure of database extraction success.

The fourth metric, Chunk Recovery Rate (CRR), measures the quality of successful leakage using long contiguous token overlap with threshold $\mathcal{C}_i$3 tokens. For successful queries $\mathcal{C}_i$4,

$\mathcal{C}_i$5

The final CRR is the average over successful queries (Zhang et al., 7 May 2026). High CRR indicates that successful leaks are not merely paraphrases but often close to verbatim extraction.

The paper’s main methodological claim is that total leakage is well approximated by

$\mathcal{C}_i$6

This is presented as an empirical regularity rather than a theorem. It implies that retrieval coverage and leak-triggering compliance contribute largely independently: an attack that retrieves broadly but cannot coerce the generator will still leak little, and an attack with strong coercive instructions but poor retrieval exploration will likewise underperform (Zhang et al., 7 May 2026).

5. Principal empirical findings

The benchmark shows that attack effectiveness is highly unstable across models and datasets. A strong example is PoR on Enron Email: it achieves 88.3% CCL with Gemini-3-flash, but only 6.8% CCL with DeepSeek-V3. In that same DeepSeek-V3/Enron setting, RAG-Thief becomes the strongest attack with 64.4% CCL (Zhang et al., 7 May 2026). There is therefore no universally dominant attack.

The decomposition into ARC and SLT is strongly supported. On Enron under TGTB, retrieval coverage remains high across models, but leak-triggering varies dramatically. The paper highlights a comparison in which Gemini-3-flash attains 88.5% SLT, whereas Qwen-3-235B has 87.1% ARC but only 0.5% SLT, yielding minimal overall leakage despite broad retrieval (Zhang et al., 7 May 2026). This directly shows that retrieval success and instruction compliance are distinct axes.

The benchmark also finds that stronger instruction-following correlates with higher leakage risk. Using IFEval scores, the paper reports

$\mathcal{C}_i$7

between instruction-following capability and leakage risk (Zhang et al., 7 May 2026). This is a correlation result, not a causal proof, but it suggests that backend models optimized to follow user instructions are also more likely to comply with extraction-style prompts unless constrained by anti-leakage safeguards.

Another central result concerns RAG quality improvements. Query rewriting consistently increases ARC and reduces ARC variance, because it improves retrieval recall even for mediocre attacker queries (Zhang et al., 7 May 2026). This can make malicious probing easier. Summarization, by contrast, significantly reduces CCL, but the paper reports that it also harms faithfulness by disrupting context integrity and reducing the model’s use of retrieved chunks. Reranker effects are comparatively small and statistically non-significant, with reported Pearson correlations of $\mathcal{C}_i$8 for CCL, $\mathcal{C}_i$9 for ARC, and $i$0 for SLT, all with non-significant $i$1-values (Zhang et al., 7 May 2026).

The paper further reports a positive relationship between faithfulness and leakage risk. This suggests a structural privacy–utility tension: systems that reproduce retrieved context more faithfully may also be more likely to reproduce it too faithfully under adversarial prompting. The authors note that this relationship is specific to faithfulness; other utility measures such as answer relevancy and information density do not show the same strong positive association (Zhang et al., 7 May 2026).

A further misconception addressed by the results is that successful leakage is merely coarse extraction. In fact, CRR is often near 100% for successful attacks, indicating that when leakage occurs it is frequently close to verbatim chunk recovery rather than weak paraphrase (Zhang et al., 7 May 2026).

6. Defensive evaluation, limitations, and place within leakage benchmarking

LeakDojo includes a two-stage defense evaluation. The input intent detector, implemented with gpt-4.1-mini, classifies queries as safe, unsafe, or unknown; unsafe queries are cleaned by removing adversarial instructions, and unknown queries trigger fallback stripping of common override patterns (Zhang et al., 7 May 2026). The output content detector uses word-level ROUGE-L F-score with threshold 0.5 to block outputs overly similar to retrieved chunks.

Against the default attacks, the input detector is often very effective. The paper states that it can reduce the CCL of most existing attacks to below 1%. In one FiQA / DeepSeek-V3 / T2 case study, GEN-PIDE drops from 57.5% CCL to 0.2% CCL under input defense (Zhang et al., 7 May 2026). However, the framework also shows that these defenses are bypassable by stealthier instructions. The paper introduces RankerSet and CodeClaim, both of which conceal extraction intent behind plausible task framing. In the same defended FiQA / DeepSeek-V3 / T2 setting, GEN-PIDE + RankerSet reaches 47.8% CCL and GEN-PIDE + CodeClaim reaches 59.6% CCL under input defense; with both input and output defenses, CodeClaim + GEN-PIDE still yields 26.5% CCL (Zhang et al., 7 May 2026). The practical implication is that detectors tuned to overt “repeat the context” prompts are inadequate against semantically masked extraction.

LeakDojo’s limitations are clearly delimited. It focuses on effectiveness more than attack efficiency, although the appendix includes cost analyses. It evaluates only three common RAG enhancement modules—rewriter, reranker, summarizer—rather than the full space of possible RAG architectures. It uses a fixed standard budget of $i$2 for the main study. It is English-only. And it focuses specifically on leakage of retrieved chunks, not broader issues such as training-data memorization, retrieval poisoning, multimodal RAG leakage, or system-prompt extraction as a primary objective (Zhang et al., 7 May 2026).

Within the broader research landscape, LeakDojo belongs to a family of leakage-oriented benchmarking systems that make attack surfaces explicit and measurable. AgentLeak similarly emphasizes channel-specific privacy evaluation, but for multi-agent LLM systems rather than RAG, and extends leakage analysis beyond final outputs to internal coordination channels such as inter-agent messages and memory (Yagoubi et al., 12 Feb 2026). DeepLeak performs an analogous audit-and-harden workflow for post-hoc ML explanations, framing leakage as membership inference through explanation outputs and combining leakage profiling with mitigation search (Hmida et al., 6 Jan 2026). In software and ML-pipeline settings, ConDefects and LeakageDetector show the same methodological impulse toward controlled evaluation and operationalized leakage taxonomies, though they target benchmark contamination and data leakage bugs rather than RAG chunk extraction (Wu et al., 2023, AlOmar et al., 18 Mar 2025). This suggests that LeakDojo is best understood not as an isolated attack paper but as a benchmark framework within a broader move toward leakage-specific evaluation infrastructures.

Overall, LeakDojo’s distinctive contribution is to turn RAG leakage from an anecdotal attack phenomenon into a controlled experimental object. Its most durable findings are the factorization of leakage into query coverage and instruction-triggering, the correlation between instruction-following and leakage risk, and the evidence that architectural improvements such as rewriting and faithfulness enhancement can raise privacy risk unless evaluated jointly with defenses (Zhang et al., 7 May 2026).