MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models

Abstract: Pretraining is fundamental to the development of LLMs, yet the opacity of pretraining data complicates model analysis and raises ethical, legal, and fairness concerns. Detecting whether specific datasets were used during pretraining is, therefore, critical. Existing state-of-the-art methods typically rely on access to model probability distributions, making them unsuitable for closed-source LLMs that provide only input-output interfaces. To address this limitation, we introduce Masked Corpus-level Pretraining Data Detection (MC-PDD), a novel method inspired by the masked language modeling paradigm. MC-PDD masks highly specific tokens in each text and prompts the LLM to predict the missing content. It then assesses whether the difference in prediction hit rates between a candidate corpus and a reference non-member corpus is statistically significant. Based on this comparison, MC-PDD determines whether the candidate texts were likely included in the model's pretraining data. Experimental results demonstrate clear and consistent differences in prediction hit rates between pretrained and unseen data across three datasets, for both open-source and closed-source LLMs. Despite operating under a stricter black-box setting, MC-PDD achieves performance comparable to existing detection methods. Our approach enables practical applications such as model auditing and data copyright verification using only standard API access. Upon acceptance, we will publicly release the code and datasets.

• The paper introduces MC-PDD, a black-box method for detecting pretraining data exposure by masking TF-IDF-selected tokens and analyzing hit rates.
• The methodology leverages masked token prediction and statistical hit rate comparisons, achieving significant discrimination with a hit rate gap over 9% and robust p-values.
• Practical experiments demonstrate MC-PDD’s utility for model auditing, copyright verification, and contamination detection across diverse LLMs.

MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box LLMs

Motivation and Problem Setting

LLMs are typically trained on enormous, opaque text corpora. Lack of transparency in pretraining datasets introduces critical issues, particularly concerning data contamination and privacy risks. Existing corpus-level pretraining data detection methods generally require access to model internals, rendering them unsuitable for closed-source models that only provide input-output API access. The MC-PDD framework addresses this gap by enabling corpus-level detection in a strict black-box setting—relying exclusively on observable model outputs, without any internal probability distributions or weights.

Methodology

MC-PDD is inspired by the masked language modeling paradigm. The core procedure involves applying a TF-IDF algorithm to identify highly specific tokens in each text, which are then masked and replaced with a <blank> token. The masked text is submitted to the model via a standardized prompt, requesting the model to predict the missing token. The effectiveness of prediction is quantified by calculating the average "hit rate"—the rate at which the correct masked token appears among the model's top-N generated candidates. Membership inference is facilitated by statistically comparing hit rates across candidate and reference (non-member) corpora; significantly elevated hit rates for the candidate corpus indicate its probable inclusion in the model's pretraining data.

Figure 1: MC-PDD utilizes masked token prediction via the model's API to elicit memorized content, demonstrating direct probe of pretraining exposure.

The TF-IDF selection is critical: masking highly distinctive tokens (i.e., those rare in the corpus but frequent in the text) increases sensitivity to memorization rather than generic language modeling ability. The prompt design incorporates entity name and introduction to maximize memory activation and minimize generalization confounds.

Experimental Validation

Robust benchmarking was conducted across six state-of-the-art LLMs (instruction-tuned open-source and API-based closed-source alike). Three datasets were used: SteamMIA (game metadata stratified by release dates), BBC News, and ArXiv paper abstracts. Models included LLaMA-3.1-8B-Instruct, Claude-3-Haiku, DeepSeek-V3-Chat, GPT-4o-Mini, LLaMA-3-70B, and Gemini-1.5-Flash, with model knowledge cutoff dates strictly controlling member/non-member categorization.

Results show that MC-PDD achieves corpus-level discrimination with a minimum hit rate gap exceeding 9% for all models with relevant cutoff dates, and $p$-values consistently below 0.05. For reference, LLaMA-3.1-8B-Instruct yields a member hit rate of 48.50%, non-member rate of 36.26%, and a difference of -12.24% ($p=0.0001$). All models but Gemini-1.5-Flash (due to cutoff overlap) manifest strong detection efficacy.

Minimum Sample Size Analysis

Detection robustness to sample size was empirically evaluated. Bootstrapped resampling over decreasing sample sizes confirmed that MC-PDD maintains statistically significant hit rate gaps down to a corpus of 100 samples. Below this threshold, statistical power drops precipitously for most models.

Figure 2: Mean hit rate difference and $p$-value remain robust for MC-PDD down to a sample size of 100, validating practical deployment feasibility.

Ablation and Control Experiments

Several ablations and controls further delineate the desiderata for reliable detection:

• Prompt Specificity: Removing entity names sharply reduces member hit rates (from 48.50% to 32.91%), confirming the necessity of highly specific context for memory activation. The hit rate gap persists ($p=0.0116$) but at reduced magnitude.
• Word Selection Strategy: Masking random tokens negates discriminative power, collapsing hit rates and eliminating significant difference—empirically validating TF-IDF as the optimal selection.
• Expansion to Less Specific Words: Increasing the number of masked tokens per document (decreasing specificity) systematically reduces hit rate difference and statistical significance, highlighting the sensitivity of MC-PDD to term selection.
• Data Contamination Simulation: Instruction fine-tuning LLaMA-3.1-8B-Instruct with contaminated non-member samples increases hit rates for these samples, confirming MC-PDD's ability to detect memorization arising from post-pretraining contamination.

  Figure 3: Training epochs show distinct hit rate profiles for member, non-member, and contaminated non-member samples, confirming MC-PDD's memory detection.

Comparative Performance

MC-PDD is benchmarked against prior black-box methods that rely on perplexity, token probability, and reference models. Despite operating under a stricter interface constraint, MC-PDD achieves comparable or superior accuracy and AUC scores, evidenced in extensive experiments.

Practical and Theoretical Implications

MC-PDD offers clear utility for:

• Model Auditing: Enables practical detection of pretraining corpus exposure in commercial, API-restricted environments.
• Copyright Verification: Supports provenance analysis for copyrighted texts.
• Benchmark Reliability: Aids detection and quantification of test set contamination, essential for maintaining valid AI evaluations.
• MIA Extension: Broadens membership inference attack applicability by aligning corpus-level detection with stricter adversarial settings.

On the theoretical side, MC-PDD reveals bounds on LLM memorization and generalization, paving the way for informed study of privacy risks and regulatory compliance in LLM deployment. The approach further establishes statistical detection thresholds relevant for audit and governance scenarios.

Limitations and Future Directions

MC-PDD's effectiveness depends on the presence of sufficiently specific tokens within candidate corpora. TF-IDF's reliance on lexical frequency misses nuances of semantic specificity; future research should explore semantically-informed or syntactic word selection. Scalability to larger-scale models and more diverse datasets remains untested. Document-level detection and enhanced specificity extraction merit further investigation for improved robustness and precision.

Conclusion

MC-PDD constitutes a rigorous, black-box corpus-level method for pretraining data detection, validated across open- and closed-source models, with minimal sample requirements for statistically reliable deployment. Its reliance on masked token prediction, strategic TF-IDF masking, and statistical comparison of hit rates ensures practical and robust detection in domains where pretraining transparency is unattainable. The approach advances practical auditability, contamination monitoring, and privacy assessment in large-scale AI systems, laying foundational groundwork for subsequent advances in black-box corpus-level detection methodologies.