Pre-Training Data Detection (PDD)

• Pre-Training Data Detection (PDD) is the process of determining if a text was included in an LLM’s pre-training corpus using membership inference techniques.
• It employs diverse methods—including likelihood, fine-tuning, representation, gradient, and active reconstruction approaches—to extract membership signals from model outputs.
• PDD has practical implications for copyright and privacy auditing, though its effectiveness relies on accessible model internals and robust benchmark designs.

Pre-Training Data Detection (PDD) is the task of inferring, from a trained model and a candidate text or corpus, whether that material was included in the model’s pre-training data. In the recent LLM literature, PDD is typically formulated as a membership inference problem over an unknown pre-training corpus, motivated by copyright auditing, privacy auditing, and benchmark contamination analysis. The term has also been extended to related questions such as estimating pre-training domain proportions from model outputs, while the acronym “PDD” is additionally used for unrelated concepts in adjacent literatures (Shi et al., 2023, Zhang et al., 2024, Liang et al., 2024).

1. Formalization and access models

In its standard LLM form, PDD asks for a detector that maps a text sequence $x$ and a target model $\mathcal{M}$ to a binary membership decision. A common formalization is

$\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$

where $1$ denotes “member” and $0$ denotes “non-member”. Score-based formulations then threshold a scalar statistic $s(x;\mathcal{M})$, for example

$$\text{prediction}(x,\mathcal{M})= \begin{cases} 1, & s(x;\mathcal{M})\ge \lambda\ 0, & s(x;\mathcal{M})< \lambda, \end{cases}$$

with the central design problem being the construction of a score that separates members from non-members as well as possible (Zhang et al., 2024, Tang et al., 22 Jul 2025).

The literature distinguishes several access regimes. In a grey-box setting, the auditor has only inference-time outputs such as per-token logits, per-token probabilities $p_\theta(x_t\mid x_{<t})$, and loss, but not model weights or gradients. In a black-box setting, some methods assume access only to the realized-token probabilities, while the strictest recent formulations assume only an input-output text interface with no probability exposure at all. White-box formulations instead assume access to model parameters, internal activations, and gradients, enabling detector construction from hidden states, neuron activations, or gradient profiles rather than output-layer statistics alone (Zhang et al., 2024, Liu et al., 2024, Lan et al., 6 Jun 2026).

These access assumptions are not merely operational details; they partition the methodology space. Probability-based PDD dominates grey-box work, activation- and gradient-based PDD dominate white-box work, and masked prompting with corpus-level statistical comparison has emerged as a response to the practical limitations of closed-source APIs (Zhang et al., 2024, Lan et al., 6 Jun 2026).

2. Methodological families

The field contains several distinct detector families, differentiated by what part of model behavior is treated as the membership signal.

Family	Representative methods	Primary signal
Likelihood-based	PPL, Zlib, Lowercase, Min-K%, Min-K%++, DC-PDD	Token probabilities or calibrated losses
Fine-tuning-based	FSD	Score deviation after fine-tuning on unseen in-domain data
Representation-based	Probe Attack, NA-PDD	Hidden states or neuron activation patterns
Gradient-based	GDS	Gradient magnitude, location, and concentration
Active reconstruction	ADRA, ADRA+	RL-induced suffix reconstructibility
Strict black-box corpus-level	MC-PDD	Masked-token hit-rate differences

Likelihood-based methods were the initial mainstream. Perplexity and related loss attacks use average next-token likelihood; Min-K% instead averages the lowest-probability tokens in a sequence; Min-K%++ replaces raw log-probability with a normalized token score,

$$\frac{\log p(x_t \mid x_{<t}) - \mu_{x_{<t}}}{\sigma_{x_{<t}}},$$

where $\mu_{x_{<t}}$ and $\mathcal{M}$0 are the mean and standard deviation of log-probabilities under the conditional categorical distribution at that position. DC-PDD adds divergence-based calibration against a token-frequency distribution from an external corpus, while SURP restricts attention to “surprising tokens,” defined by low entropy and low ground-truth probability, and averages only those token log-probabilities (Shi et al., 2023, Zhang et al., 2024, Zhang et al., 2024, Zhang et al., 2024).

A second family amplifies weak static signals by modifying the model. FSD fine-tunes on a small set of unseen in-domain non-members and measures score deviation before and after fine-tuning; non-members typically experience larger score decreases than members. Probe-based methods train linear classifiers on hidden activations extracted from a proxy contamination setup. NA-PDD replaces dense probes with a training-free neuron-activation procedure that identifies member-dominant and non-member-dominant neurons from reference corpora and scores a query by overlap ratios. GDS moves from activations to optimization behavior, computing gradient profiles over LoRA-inserted Attention and FFN modules and summarizing each gradient matrix by magnitude, eccentricity, sparsity, and concentration features before classification with a lightweight MLP (Zhang et al., 2024, Liu et al., 2024, Tang et al., 22 Jul 2025, Zhang et al., 5 Mar 2026).

A third family treats membership as reconstructibility rather than confidence. ADRA and ADRA+ split each candidate document into prefix $\mathcal{M}$1 and held-out suffix $\mathcal{M}$2, then use on-policy RL to elicit reconstruction of $\mathcal{M}$3 from $\mathcal{M}$4. Their core premise is that members are more reconstructible than non-members because RL can sharpen behaviors already encoded in the weights. At the opposite access extreme, MC-PDD abandons token probabilities entirely: it masks a highly specific TF-IDF-selected token in each document, queries the model for the ten most likely candidates, and decides corpus-level membership from whether the candidate corpus yields a statistically higher masked-token hit rate than a reference non-member corpus (Yin et al., 22 Feb 2026, Lan et al., 6 Jun 2026).

3. Benchmarks and evaluation regimes

Benchmark design is a central issue because PDD is highly sensitive to temporal leakage, distribution shift, and trivial metadata cues.

Benchmark	Domain	Distinguishing property
WikiMIA	Wikipedia events	Dynamic time-based member/non-member split
MIMIR	The Pile subsets	Minimal distribution shift, hard near-IID setting
ArxivMIA	arXiv abstracts	Technical long-form abstracts, 2,000 examples
PatentMIA	Chinese patents	10,000 Chinese 512-word patent snippets
CCNewsPDD	News articles	Same-month members and transformed non-members
Dolma-Book	Books	Head/middle/tail segmentation using released corpus
SteamMIA	Steam game metadata	Post-2023 benchmark for modern knowledge cutoffs

WikiMIA was introduced as a dynamic benchmark using data created before and after model training to support gold-truth detection; subsequent evaluations commonly report length-based splits such as 32, 64, and 128 tokens, and earlier descriptions also include 256-token truncations. MIMIR is harder by design: its train and test examples come from train versus test splits of the same The Pile subsets, reducing distribution shift and often driving AUROC values close to random guessing. ArxivMIA contributes 2,000 arXiv abstracts with average length 143 tokens, while PatentMIA extends PDD to Chinese with 10,000 512-word patent snippets (Shi et al., 2023, Zhang et al., 2024, Liu et al., 2024, Zhang et al., 2024).

Later work attacked benchmark bias directly. CCNewsPDD constructs members and non-members from the same August 2017 slice of CCNews, with non-members generated by back-translation, masking-and-fill, or LLM rewriting, thereby removing temporal drift as the main discriminant. Dolma-Book uses OLMo’s released pre-training corpus and Gutenberg books added after the Dolma collection date, then evaluates separately on head, middle, and tail segments of long books. MC-PDD introduces SteamMIA, a temporally split corpus of Steam game metadata with 468 member and 422 non-member samples, and also evaluates on BBC_News_Alltime and ArXiv_Alltime to test corpus-level black-box detection under rolling time cutoffs (Tang et al., 22 Jul 2025, Zhang et al., 2024, Lan et al., 6 Jun 2026).

Evaluation metrics remain dominated by AUROC and TPR at low FPR, especially TPR@5%FPR. Corpus-level methods additionally report hit-rate differences, $\mathcal{M}$5-values, and confidence-interval bounds for two-sample proportion comparisons. The coexistence of sample-level and corpus-level evaluation means that “performance” is not uniform across the literature: some methods optimize per-document ranking, whereas others test whether an entire dataset was likely present in pre-training (Zhang et al., 2024, Lan et al., 6 Jun 2026).

4. Empirical development of the field

The earliest strong reference-free baseline was Min-K%, which achieved an average AUC of $\mathcal{M}$6 on WikiMIA and improved on the best prior baseline by $\mathcal{M}$7. Min-K%++ then reframed PDD in terms of local maxima of the modeled distribution and reported new SOTA results: on WikiMIA it outperformed the runner-up by $\mathcal{M}$8, $\mathcal{M}$9, and $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$0 AUROC points for lengths 32, 64, and 128 respectively, averaged over five models; on MIMIR it improved over Min-K% by $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$1, $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$2, $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$3, and $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$4 for Pythia-1.4B through 12B, although absolute difficulty remained high (Shi et al., 2023, Zhang et al., 2024).

Calibration and model-modification methods then widened the gap. DC-PDD reported AUC $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$5 on BookMIA for GPT-3, exceeding Lowercase at $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$6 and Min-K% at $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$7, and on PatentMIA it reached $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$8 for Baichuan-13B and $\mathcal{A}(x,\mathcal{M}) \rightarrow \{0,1\},$9 for Qwen1.5-14B. FSD used score deviation after fine-tuning on small unseen in-domain data and reported striking gains; for example, on WikiMIA with OPT-6.7B and Min-k%, AUC increased from $1$0 to $1$1, and on ArXivTection with LLaMA-7B and Perplexity, AUC increased from $1$2 to $1$3 (Zhang et al., 2024, Zhang et al., 2024).

White-box methods produced another step change. Probe-based detection reached $1$4 AUC on WikiMIA for Pythia-2.8B and $1$5 on ArxivMIA for OpenLLaMA-13B. NA-PDD then reported $1$6 AUC on WikiMIA for Pythia-2.8B, $1$7 on ArxivMIA for OpenLLaMA-13B, and $1$8 on CCNewsPDD(prompt) for OPT-6.7B. GDS, using gradient-deviation profiles, reported $1$9 AUROC and $0$0 TPR@5%FPR on WikiMIA with LLaMA-7B, remained at $0$1 after timestamp deletion, and outperformed FSD in cross-dataset transfer settings with $0$2–$0$3 AUROC versus $0$4–$0$5 (Liu et al., 2024, Tang et al., 22 Jul 2025, Zhang et al., 5 Mar 2026).

Active reconstruction methods raised performance again in settings where passive scores saturate. ADRA and ADRA+ reported an average improvement of $0$6 over the previous runner-up across pre-training, post-training, and distillation detection. For pre-training specifically, ADRA+ reached $0$7 AUROC on BookMIA, $0$8 on WikiMIA$0$9 Hard, and $s(x;\mathcal{M})$0 on Dolma3 arXiv; on BookMIA it improved over Min-K%++ by $s(x;\mathcal{M})$1. In the strict black-box regime, MC-PDD reported clear and consistent hit-rate gaps between member and non-member corpora across open and closed models, and on LLaMA-3.1-8B-Instruct with SteamMIA it achieved accuracy $s(x;\mathcal{M})$2 and AUC $s(x;\mathcal{M})$3, comparable to several probability-based methods despite using only text outputs (Yin et al., 22 Feb 2026, Lan et al., 6 Jun 2026).

5. Limitations, confounds, and interpretive cautions

One persistent limitation is that PDD remains highly access-dependent. Min-K%++, DC-PDD, and related grey-box methods require per-token probabilities, and Min-K%++ in particular assumes access to the full conditional categorical distribution over the vocabulary. Closed APIs often expose at most limited token information or none at all. White-box methods such as Probe Attack, NA-PDD, and GDS require weights, activations, or gradients, and are therefore inapplicable to proprietary models served only through standard chat endpoints. FSD additionally requires the ability to fine-tune on small unseen in-domain corpora, which is a stronger assumption than ordinary API access (Zhang et al., 2024, Zhang et al., 2024, Zhang et al., 2024, Tang et al., 22 Jul 2025, Zhang et al., 5 Mar 2026).

A second issue is benchmark confounding. Time-based splits may reward temporal drift detection rather than genuine membership inference. CCNewsPDD was introduced precisely to remove this confound by pairing original members with transformed non-members from the same month. Likewise, deletion of timestamp tokens in WikiMIA degrades all methods, indicating that some apparent signal in earlier setups came from explicit temporal markers. Probe-based methods also show domain dependence: cross-domain transfer from WikiMIA to ArxivMIA is materially weaker than in-domain evaluation (Liu et al., 2024, Tang et al., 22 Jul 2025, Zhang et al., 5 Mar 2026).

A third limitation is that near-IID pre-training membership is intrinsically hard. MIMIR often yields AUROCs close to $s(x;\mathcal{M})$4, especially for smaller models and homogeneous subsets, even for strong recent methods. Corpus-level methods address variance by aggregation, but they trade away example-level resolution; MC-PDD detects whether a corpus is likely to have been used during pretraining, not whether each individual document is a member. It also depends on the availability of sufficiently specific words and on a well-matched non-member reference corpus (Zhang et al., 2024, Lan et al., 6 Jun 2026).

Finally, evidential status remains contested. The literature repeatedly treats PDD as a statistical auditing tool, not as a definitive proof of inclusion. One paper explicitly notes that MIAs alone cannot legally prove membership, even when they substantially increase the evidential weight of a claim. In practice, copyright, privacy, and contamination analysis therefore combine PDD with provenance information, temporal metadata, corpus construction details, or additional extraction-style evidence (Tang et al., 22 Jul 2025).

6. Broader variants and terminological scope

The phrase “pre-training data detection” has broadened beyond single-sample membership inference. One extension is “data proportion detection,” which seeks estimates $s(x;\mathcal{M})$5 for the proportions of different pre-training domains from model outputs alone. Under a Data Mixing Law,

$s(x;\mathcal{M})$6

the paper derives

$s(x;\mathcal{M})$7

with $s(x;\mathcal{M})$8, linking generated-domain proportions $s(x;\mathcal{M})$9 to latent training-mixture weights (Liang et al., 2024).

The acronym “PDD” is also overloaded in unrelated literatures. In computer vision, “Part Detector Discovery” denotes a method that analyzes gradient maps in a pre-trained CNN to discover implicit object-part detectors without retraining on the target fine-grained dataset (Simon et al., 2014). In SSL data curation for vision foundation models, “Poisoned Data Detector” denotes an active defense that uses ImageBind embeddings with classical classifiers such as RF, KNN, NB, and SVM to filter poisoned images before pre-training, with SVM-PDD reported as strongest on both in-distribution and out-of-distribution tests (Gupta et al., 8 Jun 2026). Another adjacent but distinct line, “Detection-Aware Pre-training,” uses weakly localized pseudo boxes from classification data to pre-train object detectors, making the model location-aware; it concerns task alignment for detection rather than inference about whether a given sample was used during pre-training (Zhong et al., 2021).

Taken together, the contemporary meaning of Pre-Training Data Detection in LLM research is relatively specific: it is a membership-inference-style audit of opaque pretraining corpora, now spanning probability-based, representation-based, gradient-based, active-reconstruction, and strict black-box corpus-level methods. At the same time, the broader literature shows a systematic expansion from single-example membership decisions to corpus-level auditing, domain-mixture estimation, and pre-training data integrity analysis across modalities (Liang et al., 2024, Lan et al., 6 Jun 2026, Gupta et al., 8 Jun 2026).