RECAP: Auditing Memorization in LLMs

• RECAP is a system that audits memorization in LLMs by eliciting verbatim excerpts from undisclosed training data using an agentic, iterative process.
• It employs modules for segmentation, extraction, verification, jailbreaking, and feedback to overcome alignment refusals and enhance extraction fidelity.
• Empirical results on benchmarks like EchoTrace demonstrate significant ROUGE-L improvements, confirming its effectiveness in direct memorization auditing.

RECAP, short for "Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline," is a black-box-compatible agentic system for eliciting and verifying memorized passages from LLMs. It is designed for a setting in which training corpora are undisclosed and direct dataset inspection is infeasible, so memorization must be audited from model behavior rather than from provenance metadata. Its central claim is evidentiary rather than merely predictive: if a model can be induced to regenerate a target passage with high fidelity under minimal, high-level guidance, that constitutes strong direct evidence that the content was learned. RECAP operationalizes this view through iterative extraction, refusal detection, jailbreaking, discrepancy analysis, and contamination-averse feedback, and was evaluated on EchoTrace, a benchmark of 20 arXiv papers and 35 books. On copyrighted books, it raises average ROUGE-L from approximately $0.26$ under Dynamic Soft Prompting to approximately $0.46$, and improves GPT-4.1 from $0.379$ to $0.468$, a relative increase of about $24\%$ (Duarte et al., 29 Oct 2025).

1. Problem setting and evidentiary basis

RECAP addresses an auditing problem created by contemporary large-scale LLM training: model developers, regulators, and rightsholders often cannot inspect the original corpus, yet still need to determine whether a specific work was seen and memorized. The framework treats reproduction as the strongest available signal of memorization. This position is contrasted with statistical membership inference, dataset inference, and loss-based exposure signals, which are characterized as indirect and vulnerable to false positives arising from distribution shift or miscalibration (Duarte et al., 29 Oct 2025).

The paper identifies two practical obstacles to reproduction-based auditing. The first is alignment-driven refusal, especially for copyrighted content. The second is that a model frequently does not produce its best reconstruction on the first try. RECAP is built to address both obstacles while minimizing contamination: it aims to elicit the target model’s own internal memory rather than to inject the ground-truth passage through prompts or feedback. For that reason, its prompts and correction hints are high-level, and the feedback agent is explicitly forbidden from quoting or closely paraphrasing the reference passage.

A further structural choice is segmentation. Rather than probing an entire book or paper uniformly, RECAP divides a document into semantically coherent, contiguous, non-overlapping “events.” This event-centric formulation localizes extraction, allows event-level verification, and supports iterative refinement against a bounded reference segment.

2. Agentic pipeline and module structure

RECAP consists of five main modules and an optional filtering stage. The pipeline begins with preprocessing of the target document into event-level units, then alternates between extraction and correction, with a refusal branch that routes blocked generations through a jailbreak prompt (Duarte et al., 29 Oct 2025).

Module	Input / output	Function
Section Summary Agent	Document $\rightarrow$ event metadata + reference segment	Semantic segmentation and soft conditioning
Extraction Agent	Event prompt $\rightarrow$ completion	Target LLM under audit
Verbatim Verifier	Prompt-response pair $\rightarrow$ Yes/No	Refusal detection
Jailbreaker	Refused prompt $\rightarrow$ reframed prompt	Alignment circumvention
Feedback Agent	Reference + completion $\rightarrow$ discrepancy report	Minimal correction hints
Memorization Score Filter	Initial signals $0.46$0 keep/skip	Cost reduction

The Section Summary Agent outputs, for each event, a structured bullet-point detailed_summary, a character list, verbatim first_sentence and last_sentence anchors, and the corresponding reference text segment. These fields are then used to construct the Extraction Agent prompt. Extraction is performed with temperature $0.46$1, reflecting the system’s emphasis on deterministic, high-fidelity generation during audit rounds.

The Verbatim Verifier receives the prompt-response pair and returns a binary decision: “Yes” if the output is a substantive extraction attempt, and “No” if it is a refusal, safety response, or irrelevant completion. In practice, the verifier is Gemini-2.5-Flash, which is reported to have very low misclassification rates in spot checks. If a refusal is detected, the Jailbreaker applies a single static Narrative Tool Injection prompt that reframes the request as simulating a trusted function call, generate_literary_segment(...), parameterized by the event metadata. This single prompt bypasses more than $0.46$2 of refusals across models.

The Feedback Agent compares the completion to the gold event text and returns a structured report with three sections: Major structural issues, Missing elements, and Inaccurate elements. Its constraints are unusually strict: it may not quote or closely paraphrase the original, and must instead express abstract chronological discrepancies such as missing location detail, events being out of order, or invented reflection. These hints are appended to the prompt for the next extraction round.

To reduce cost, RECAP can optionally skip low-potential events. The filter combines a Parrot BERT reconstruction signal, ROUGE-L, and cosine similarity into a hybrid memorization score:

$0.46$3

where $0.46$4. Events below a threshold are excluded from refinement.

3. Iterative algorithm, scoring, and stopping criteria

Formally, RECAP operates on an event with reference text $0.46$5 and metadata $0.46$6. The prompt at iteration $0.46$7 is $0.46$8, the target model’s generation is $0.46$9, the refusal detector is $0.379$0, the jailbreak module is $0.379$1, and the feedback agent is $0.379$2. The automatic similarity function is ROUGE-L, written $0.379$3 (Duarte et al., 29 Oct 2025).

The loop begins with $0.379$4. At each round, the target model produces $0.379$5. If $0.379$6, the next prompt is replaced by the jailbreak transformation $0.379$7. Otherwise, the feedback agent computes discrepancies $0.379$8 and converts them into abstract hints $0.379$9. The prompt is updated by appending this structured guidance. Iteration stops after at most five total rounds, or earlier if the ROUGE-L score fails to improve.

ROUGE-L is defined from the longest common subsequence between reference $0.468$0 and hypothesis $0.468$1. With

$0.468$2

the ROUGE-L F-score is

$0.468$3

with $0.468$4 unless otherwise specified.

For books, RECAP uses a token-weighted average of passage-level ROUGE-L across a book’s passages. Group-level statistics are obtained by bootstrap sampling at the book level with $0.468$5 resamples. Passage counting uses 40-token segments: a segment is counted as memorized if it matches the gold segment with at most five token mismatches, approximately $0.468$6, to accommodate minor formatting variation. The paper also reports robustness down to zero-tolerance matching, where RECAP still extracts hundreds of exact matches.

This formalization makes explicit that RECAP is not simply a prompt template but an iterative control loop with verification, intervention, and bounded refinement. Its stopping rule is conservative: once automatic similarity ceases to improve, the system terminates rather than continuing to elaborate potentially contaminated or hallucinated text.

4. EchoTrace benchmark and empirical results

The primary evaluation benchmark is EchoTrace, which contains 20 arXiv research papers and 35 full-length books: 15 public-domain books, 15 copyrighted bestsellers, and 5 non-training controls released after known model cutoffs. The benchmark includes more than 70,000 40-token passages derived from semantically segmented events, each paired with high-level bullet summaries, character metadata, boundary anchors, and the gold reference text (Duarte et al., 29 Oct 2025).

The evaluated target models include large proprietary systems—GPT-4.1, Claude-3.7 Sonnet, DeepSeek-V3, Gemini-2.5 Pro—and smaller systems—GPT-4.1 Mini, GPT-4.1 Nano, Gemini-2.5 Flash, and Qwen-3 32B via vLLM. Baselines are Prefix-Probing, Dynamic Soft Prompting (DSP), and DSP plus the RECAP jailbreak. This separation allows refusal mitigation and iterative feedback to be measured independently.

Setting	Baseline	RECAP
20 arXiv papers, avg. ROUGE-L	DSP $0.468$7	$0.468$8
Large-model books, public domain	DSP $0.468$9	$24\%$0
Large-model books, copyrighted	DSP $24\%$1	$24\%$2
GPT-4.1, copyrighted books	DSP $24\%$3	$24\%$4

On the arXiv proof of concept, average ROUGE-L over 20 papers rises from $24\%$5 with Prefix-Probing to $24\%$6 with DSP and to $24\%$7 with RECAP. For Claude-3.7 on these papers, RECAP improves over DSP by $24\%$8, from $24\%$9 to $\rightarrow$0. On books, the public-domain average across large models rises from $\rightarrow$1 with Prefix-Probing to $\rightarrow$2 with DSP, $\rightarrow$3 with DSP + Jailbreak, and $\rightarrow$4 with RECAP. On copyrighted books, the corresponding trajectory is $\rightarrow$5.

These numbers show two separable effects. First, the jailbreak step is already decisive: on copyrighted books, average ROUGE-L rises from $\rightarrow$6 to $\rightarrow$7 when moving from DSP to DSP + Jailbreak. Second, discrepancy-driven iteration produces a further gain from $\rightarrow$8 to $\rightarrow$9. The frequently cited GPT-4.1 result is the move from $\rightarrow$0 to $\rightarrow$1, equivalent to a relative improvement of approximately $\rightarrow$2.

The benchmark also includes negative controls. Non-training books yield near-zero extractions, with at most a single 40-token passage per book-model pair. This is used to argue that the feedback process does not spuriously create verbatim content. An independent LLM judge further flags metadata and feedback as potentially too detailed in only $\rightarrow$3 and $\rightarrow$4 of cases, respectively.

Iteration depth and cost are explicitly measured. Most extractable gains occur in the first feedback round, and fewer than $\rightarrow$5 of events benefit from further rounds. Optional hybrid filtering reduces cost at modest performance loss; for example, DeepSeek-V3 drops from $\rightarrow$6 to $\rightarrow$7 ROUGE-L when fewer events are refined. On a large book audit with DeepSeek-V3 and Harry Potter Volume 1, approximate costs are $\rightarrow$8 for DSP, and $\rightarrow$9 for RECAP.

5. Mechanistic interpretation, contamination control, and relation to prior work

The paper attributes RECAP’s gains to localized, discrepancy-based guidance that scaffolds partial recollections into fuller reconstructions without revealing the answer. High-level chronological hints appear to improve longest-common-subsequence overlap by aligning the model’s output with the reference along dimensions such as who, where, when, and event order. The authors describe this as a form of “self-critique with externalized evaluator”: a second model diagnoses structural gaps, and the target model then produces a better continuation under focused constraints (Duarte et al., 29 Oct 2025).

Contamination control is central to that interpretation. The Section Summary Agent provides summaries and anchors rather than quoted passages. The Feedback Agent is prohibited from reproducing exact wording. The near-zero extraction rate on non-training controls is presented as evidence that RECAP is not merely constructing the answer through prompt leakage. A plausible implication is that the framework works best when the target model already contains partial verbatim or near-verbatim traces that can be reassembled under guided prompting.

The paper situates RECAP against several neighboring lines of work. Membership inference, dataset inference, and loss-based exposure analysis provide indirect signals of possible training exposure, but not direct reproduction. Prefix-Probing and Dynamic Soft Prompting show that memorized text can sometimes be continued from a carefully chosen prefix or summary, yet remain brittle and frequently blocked by alignment refusals. Jailbreaking and tiered red-teaming can overcome refusals, but do not by themselves provide a discrepancy-evaluation loop. RECAP’s distinct contribution is therefore the combination of refusal detection, a static low-cost jailbreak, iterative discrepancy extraction, and optional cost-aware filtering.

The empirical analysis also reports model-size and corpus-popularity effects. Larger models are more extractable than smaller variants of the same family, and copyrighted books with higher sales exhibit a weak-to-moderate positive trend in ROUGE-L under RECAP. The paper states that this is consistent with greater web prevalence.

6. Limitations, ethics, reproducibility, and governance implications

The framework has several explicit limitations. Feedback quality matters: weaker evaluators produce poorer hints, and larger feedback models such as GPT-4.1 outperform smaller ones. Some API-level or platform-level filters remain effective even after the jailbreak step, which helps explain why Gemini-2.5 Pro still lags in final ROUGE-L despite many refusals being bypassed. Long and complex passages remain vulnerable to hallucinated connective tissue, and the iterative pipeline is prompt-intensive enough that web-scale extraction remains nontrivial even with hybrid filtering. The benchmark is also biased toward popular works, so performance may be overestimated for obscure texts, and absence of extraction is not evidence of non-membership (Duarte et al., 29 Oct 2025).

The ethical and legal discussion is correspondingly strong. The same method that can audit memorization can also be misused to exfiltrate copyrighted content. For that reason, the authors do not release extracted copyrighted passages; they release only public-domain segments, event annotations, and model outputs. RECAP is positioned as an auditing tool for compliance and alignment research rather than as a distribution mechanism. The paper recommends defense-in-depth for deployers, including safeguards, logging, rate limits, and scope constraints, and advises researchers to avoid redistributing verbatim copyrighted outputs.

Reproducibility is unusually detailed. Code is released in a public repository, EchoTrace is hosted on Hugging Face with segmentations and metadata, and full prompt templates for all agents are provided in appendices. The default verifier is Gemini-2.5-Flash, the default feedback agent is GPT-4.1, summarization runs at temperature $\rightarrow$0, extraction and feedback rounds run at temperature $\rightarrow$1, and statistics use $\rightarrow$2 bootstrap resamples. White-box targets such as Qwen-3 32B are run via vLLM on A100 80GB GPUs, while proprietary targets are accessed through APIs. Exact replication may still vary because of API drift and external safety systems.

At a governance level, RECAP proposes reproduction as an operationally rich audit signal for training-time exposure. The paper suggests that such signals may inform fair-use disputes, licensing questions, risk assessment, and targeted mitigation of memorization in deployed systems. This suggests a shift from abstract exposure testing toward event-level, model-mediated evidence that is simultaneously more concrete and more legally sensitive.