Training Data Detection (TDD) Overview

• Training Data Detection (TDD) is a method to determine if a data sample was used in training a machine learning model using score thresholds and membership signals.
• The field spans diverse paradigms—including metric-based, learning-based, model-based, and query-based methods—tailored to different access scenarios and modalities.
• TDD is applied across text, image, and code domains to support privacy audits, copyright risk assessments, and mitigation of training data contamination.

Training Data Detection (TDD) is the task of deciding whether a specific data instance was used to train a machine learning model. In the computer security literature it is also referred to as Membership Inference Attack (MIA), and in recent LLM work it is often specialized as Pre-training Data Detection (PDD) when the target is an LLM’s pre-training corpus (Zhu et al., 2024, Tang et al., 22 Jul 2025). Across these settings, TDD is motivated by privacy and copyright concerns, by the need to assess training-data breaches and benchmark contamination, and by the use of membership signals as evidence in model unlearning and audit workflows (Zhu et al., 2024, Zhang et al., 5 Mar 2026). Recent research spans broad black-box benchmarking across image, tabular, text, and LLM settings, code-specific evaluation for autoregressive CodeLLMs, and white-box approaches based on gradients or internal activations (Zhu et al., 2024, Li et al., 23 Jul 2025, Zhang et al., 5 Mar 2026, Tang et al., 22 Jul 2025).

1. Formal task definitions and threat models

At its most general, TDD asks: given a trained target model $f_\theta$ and a data point $x$ (with label $y$ when available), determine whether $x$ was part of the model’s training set. TDDBench formalizes this as a decision rule $A(x)\in\{\text{member},\text{non-member}\}$ based on a score $s(x)$ and a threshold $\tau$ (Zhu et al., 2024). In practice, this framing covers both discriminative models, where the detector may use losses, confidences, or logits, and generative models, where the detector often uses token-level likelihood signals.

For autoregressive CodeLLMs, TDD is formulated as membership inference over source code. Given a tokenized code sequence $x=x_1,\dots,x_N$ and a pretrained autoregressive model $M$ trained on dataset $D_t$, the task is to infer whether $x$0, using a scoring function $x$1 and a threshold $x$2. The study on CodeLLMs uses the average log generation probability,

$x$3

which underlies several black-box detectors (Li et al., 23 Jul 2025).

Threat models divide the literature. TDDBench evaluates TDD primarily in the black-box setting, distinguishing score-based access, label-only access, and query-based black-box access, while noting that standard white-box signals include losses, gradients, and intermediate features even though white-box methods are not benchmarked there (Zhu et al., 2024). By contrast, GDS is explicitly white-box: it requires per-sample gradients obtained through a single backward pass on a frozen open-source LLM with attached LoRA adapters (Zhang et al., 5 Mar 2026). NA-PDD is likewise white-box, requiring access to internal FFN neuron activations captured by hooks during inference (Tang et al., 22 Jul 2025).

A central conceptual issue concerns what counts as “membership.” Much natural-language TDD work implicitly assumes verbatim membership: a sample is a member only if it appears exactly as in training. The CodeLLM study argues that this assumption is inadequate for code, where equivalence depends on clone types rather than exact textual identity. It therefore evaluates robustness using the Type-1 to Type-4 code clone taxonomy, ranging from formatting-only variants to semantic equivalence (Li et al., 23 Jul 2025). This shift from string identity to program-level equivalence is one of the clearest domain-specific refinements of the TDD problem.

2. Detection paradigms and scoring principles

The broadest methodological taxonomy in the provided literature is TDDBench’s four-paradigm classification of 21 methods: metric-based methods such as Metric-loss, Metric-conf, Metric-corr, Metric-ent, and Metric-ment; learning-based methods such as Learn-original, Learn-top3, Learn-sorted, Learn-label, and Learn-merge; model-based methods such as Model-loss, Model-calibration, Model-lira, Model-fpr, and Model-robust; and query-based methods such as Query-augment, Query-transfer, Query-adv, Query-neighbor, Query-qrm, and Query-ref (Zhu et al., 2024). These categories differ primarily in access assumptions and auxiliary requirements. Metric-based detectors require only target outputs; learning-based methods train a meta-classifier from shadow-model features; model-based methods compare the target model’s behavior to reference models trained with or without the sample; query-based methods perturb or augment the input and study response patterns.

For CodeLLMs, seven state-of-the-art TDD methods originally developed for LLMs were adapted to code and used entirely in black-box mode: Perplexity (PPL), zlib calibration, Min-K%, Min-K%++, Reference (Ref), Neighbor, and ReCaLL (Li et al., 23 Jul 2025). PPL thresholds the sample’s average log-likelihood. zlib calibration divides that likelihood by compression length to control for textual complexity. Min-K% and Min-K%++ focus on low-probability tokens, under the assumption that token outliers carry membership signal. Ref compares the target model to a smaller reference model through a loss gap. Neighbor contrasts the original sample with masked or infilled variants. ReCaLL measures relative conditional log-likelihood under non-member prefixes. All seven require token log-likelihoods; Ref additionally requires a second model, Neighbor uses external T5-large for neighbor generation, and ReCaLL requires multiple non-member prefixes (Li et al., 23 Jul 2025).

White-box LLM-specific detectors replace output statistics with internal-state measurements. GDS treats pre-training membership as an optimization-stage signal. It extracts per-sample gradient profiles from LoRA_B matrices after a single backward pass on the causal language-modeling loss and computes eight features across magnitude, location, and concentration: Abs_Mean, Row_Mean_Max, Row_Ecc, Col_Ecc, 10p_Ratio, Sparsity, Std, and Row_Mean_Std. These are aggregated into a feature tensor $x$4 and fed to a lightweight MLP that outputs a membership probability (Zhang et al., 5 Mar 2026). NA-PDD instead builds a reference-based detector from differential neuron activation patterns. It identifies “member neurons” and “non-member neurons” using activation frequencies on small labeled reference sets, selects the most discriminative layers via $x$5, and scores a candidate by the ratio

$x$6

where the numerator and denominator are averaged coincidence rates with member and non-member neuron sets over the selected layers (Tang et al., 22 Jul 2025).

Across these approaches, a recurrent distinction is between absolute and relative signals. PPL and entropy-style methods use absolute scores. Reference, ReCaLL, LiRA-style methods, and NA-PDD introduce relative comparisons against baselines, reference models, prefixes, or neuron classes. The empirical literature repeatedly associates these relative constructions with better robustness under calibration shifts, domain shifts, or structured perturbations (Zhu et al., 2024, Li et al., 23 Jul 2025, Tang et al., 22 Jul 2025).

3. Benchmarks, datasets, and evaluation protocols

TDDBench is the largest cross-modality benchmark in the provided material. It consists of 13 datasets across image, tabular, and text modalities and evaluates 21 TDD methods on 41 target models (Zhu et al., 2024). The benchmark uses balanced member/non-member labels constructed by splitting each dataset 50:50 into target and auxiliary sets and then splitting the target set into train and holdout halves. It reports multiple operating metrics, including ROC-AUC, Average Precision, membership advantage $x$7, accuracy under prior $x$8, and operating-point measures such as TPR@1%FPR and TPR@10%FPR. Multiple random partitions are used, with averages over five independent runs (Zhu et al., 2024).

For source code, the dedicated benchmark is CodeSnitch, a function-level dataset comprising 9,000 code samples across Python, Java, and C++ (Li et al., 23 Jul 2025). Each sample is labeled as “member” or “non-member” with balance enforced within each language split. Member data are curated to be highly likely in CodeLLM pretraining, filtered from training sources such as The Stack and DeepSeek-Coder sources and constrained to predate model releases. Non-member data come from highly starred GitHub repositories after January 1, 2024, including functions using 2024 APIs and new commits (Li et al., 23 Jul 2025). The evaluation covers eight CodeLLMs: SantaCoder, StarCoder2-3B, StarCoder2-7B, CodeLlama-7B, CodeLlama-13B, DeepSeekCoder-1.3B, DeepSeekCoder-6.7B, and DeepSeekCoder-33B.

The CodeSnitch study extends beyond static benchmarking by defining three robustness settings grounded in code-clone theory. Setting 1 mutates only members and leaves non-members unchanged. Setting 2 makes membership clone-type-aware, so that only mutations up to a chosen clone level remain members while stronger mutations are reclassified as non-members. Setting 3 applies the same mutation to both members and non-members to isolate the mutation’s direct effect (Li et al., 23 Jul 2025). The mutation suite covers formatting changes (Type-1), identifier renaming and adversarial obfuscation (Type-2), semantics-preserving statement substitutions at several similarity thresholds (Type-3), and a Hybrid setting combining syntactic and lexical modifications.

White-box LLM studies introduce their own benchmarks. GDS evaluates five public datasets—WikiMIA, BookMIA, ArxivTection, BookTection, and MIMIR with seven subsets—on Neo-2.7B, GPT-J-6B, OPT-6.7B, Pythia-6.9B, and LLaMA-7B, reporting AUROC and TPR@5%FPR (Zhang et al., 5 Mar 2026). NA-PDD evaluates on ArxivMIA, WikiMIA, and a new benchmark, CCNewsPDD (Tang et al., 22 Jul 2025). CCNewsPDD is designed to remove temporal bias by fixing both positives and negatives to August 2017 CCNews articles and constructing non-members through three transformations: back-translation, random masking with BERT infilling, and BART-based rewriting. The benchmark table reports 1,200 examples, average length about 309 tokens, and an equal member/non-member split, evaluated on Pythia-2.8B and OPT-6.7B (Tang et al., 22 Jul 2025).

4. Empirical performance patterns

A central result of TDDBench is that overall performance is not satisfactory. Across datasets and architectures, many methods fall in typical AUROC ranges between 0.5 and 0.6, while model-based methods perform best on average yet often reach only about 0.65 to 0.75 except in heavily overfitted cases (Zhu et al., 2024). The reported paradigm-level averages are AUROC 0.626 for metric-based methods, 0.646 for learning-based methods, 0.706 for model-based methods, and 0.604 for query-based methods. The best single-method peak in that benchmark comes from the query-based family, but with extreme computational cost (Zhu et al., 2024).

TDDBench also makes clear that TDD effectiveness depends strongly on model memorization. On Adult, where the train–test gap is about 0.001, most methods remain near chance at roughly 0.50–0.51 AUROC. On CIFAR-100, where the gap is about 0.417, many methods exceed 0.85 AUROC (Zhu et al., 2024). Architecture knowledge matters as well: when shadow or reference models mismatch the target architecture, learning-based and model-based performance degrades, as illustrated by Learn-original on CIFAR-10 dropping from 0.631 with matched shadows to 0.578 with ResNet18 shadows (Zhu et al., 2024).

In the code-specific setting, the results are notably more differentiated. On original CodeSnitch for Python, ReCaLL is the clear leader with average AUC about 0.796 across models and per-model AUCs between 0.741 and 0.830; PPL, Min-30%, zlib, Min-30%++, Ref, and Neighbor all remain much lower, in the approximate 0.560–0.612 range (Li et al., 23 Jul 2025). Larger models improve AUC for most methods. Cross-language evaluation on StarCoder2-7B shows Setting 1 detector averages of about 0.53 for C++, 0.59 for Python, and 0.63 for Java, with ReCaLL remaining the top method in all languages (Li et al., 23 Jul 2025).

White-box methods in open LLMs achieve much higher absolute numbers on their own benchmarks. GDS reports, for LLaMA-7B, AUROC 0.96 on WikiMIA, 0.97 on ArxivTection, 0.98 on BookTection, and 0.99 on BookMIA, with corresponding TPR@5%FPR substantially above strong likelihood baselines (Zhang et al., 5 Mar 2026). NA-PDD likewise outperforms nine baselines on all tested datasets and models, with especially large gains on CCNewsPDD variants: 92.4/92.1 AUROC on CCNewsPDD(trans), 98.8/95.2 on CCNewsPDD(mask), and 99.6/99.7 on CCNewsPDD(prompt) for Pythia/OPT (Tang et al., 22 Jul 2025). These figures are not directly comparable to TDDBench or CodeSnitch because the datasets, access assumptions, and benchmark constructions differ, but they show that white-box signals can be substantially more separable in open-model auditing scenarios.

5. Robustness, confounders, and contested assumptions

The most explicit robustness analysis in the provided literature is the CodeSnitch clone-based study. Under member-only mutation, all detectors degrade, but degradation depends strongly on mutation type: formatting-only Type-1 changes cause minor drops; Type-2 identifier renaming causes significant declines for token-probability methods; adversarial identifier obfuscation with random 8-character strings is worst-case and drives many methods toward random guessing; Type-3 syntactic changes further degrade PPL, zlib, and Min-K variants; and Hybrid mutations push nearly all detectors close to chance, though ReCaLL remains strongest (Li et al., 23 Jul 2025). Under clone-type-aware relabeling, AUCs are higher and more stable because the evaluation definition aligns better with program equivalence. This directly challenges the assumption that exact string membership is the only meaningful notion of inclusion for code.

Several failure modes recur across modalities. In code, grammar tokens are highly predictable and therefore weakly discriminative, while identifier declarations often receive low probability regardless of membership, which confounds Min-K-style methods that focus on low-probability tokens (Li et al., 23 Jul 2025). zlib-style calibration does not capture code structure or semantics well. Neighbor generation is unreliable because masking and infilling often produce syntactically invalid or semantically implausible code variants (Li et al., 23 Jul 2025).

In text pre-training detection, GDS argues that likelihood-based methods are susceptible to word-frequency bias, rare-word effects, short-text instability, and nonadaptive local-token heuristics, while fine-tuning-enhanced heuristics such as FSD improve in-domain performance but degrade under cross-dataset transfer (Zhang et al., 5 Mar 2026). NA-PDD’s benchmark design responds to a different confounder: temporal drift. CCNewsPDD fixes members and non-members to the same August 2017 distribution and constructs negatives through transformations rather than post-cutoff sampling, reducing the risk that detectors merely separate old from new text rather than training from non-training data (Tang et al., 22 Jul 2025).

A common misconception is that TDD provides definitive proof of inclusion. The deployment guidance for NA-PDD states that PDD provides evidence, not legal proof (Tang et al., 22 Jul 2025). Another misconception is that TDD is equivalent to reconstructing training data. GDS explicitly states that it is not designed to reconstruct or expose training samples, but rather to reveal membership signals for auditing, copyright risk assessment, and contamination checks (Zhang et al., 5 Mar 2026). A plausible implication is that robust governance requires both detection methods and careful interpretation of what those detections can and cannot establish.

6. Practical deployment and research directions

Method choice depends first on access. Under minimal black-box access with scores and limited resources, TDDBench recommends starting with simple metric-based methods such as loss, confidence, or entropy, while recognizing that their AUROC is often modest unless the target model is overfit (Zhu et al., 2024). With auxiliary data and the ability to train references, model-based methods such as LiRA-style detectors and debiased-loss variants are the most reliable general-purpose black-box option in TDDBench, especially when false-positive constraints matter (Zhu et al., 2024).

For CodeLLMs, the practical guidance is more specific. ReCaLL is recommended as the primary detector because it consistently delivers the best AUC on original CodeSnitch and remains relatively robust under clone-aware and mutation-heavy settings (Li et al., 23 Jul 2025). Thresholds should be derived by ROC analysis on a held-out calibration split of code-like data, with $x$9 chosen to meet operational FPR targets. zlib-style calibration is not recommended for code. If multiple signals are needed, an ensemble combining ReCaLL with PPL is suggested, with Ref added when a compatible smaller sibling model is available for greater resilience under Type-3-like paraphrases (Li et al., 23 Jul 2025). The main operational cost is that ReCaLL uses about 13 forward passes per sample when 12 prefixes are used.

For white-box open LLMs, GDS and NA-PDD offer different trade-offs. GDS requires one forward and one backward pass per sample with LoRA adapters, uses no fine-tuning, and identifies informative gradient signals particularly in low layers and Attention sub-modules (Zhang et al., 5 Mar 2026). NA-PDD requires access to FFN activations and a small labeled reference set—100 member and 100 non-member samples were sufficient in the reported experiments—to build discriminative neuron sets and layer rankings (Tang et al., 22 Jul 2025). In both cases, open-weight access is decisive: neither method applies directly to closed API-only systems.

The mitigation agenda extends beyond detection itself. The CodeLLM study recommends aggressive corpus deduplication, including near-duplicate and clone-aware deduplication across Type-1, Type-2, and Type-3 code; restriction to permissively licensed sources; provenance tracking; retrieval filters; refusal policies for regurgitation; and watermarking or content-provenance tools (Li et al., 23 Jul 2025). TDDBench identifies broader methodological bottlenecks: poor performance in low-memorization regimes, sensitivity to architecture mismatch, and large time and memory costs for strong model-based and query-based methods (Zhu et al., 2024). Future work proposed across the papers includes code-aware neighbor generation, clone-aware calibration, prefix-selection strategies optimized for code, efficient ReCaLL variants with fewer prefixes, architecture-agnostic references, broader language and domain coverage, and privacy-preserving training methods for code and LLMs (Li et al., 23 Jul 2025, Zhu et al., 2024, Zhang et al., 5 Mar 2026).

Taken together, these studies depict TDD as a technically heterogeneous field rather than a single settled technique. In broad black-box benchmarking, performance is often limited and highly conditional on memorization. In source code, membership definitions must account for clone structure rather than verbatim identity. In open LLMs, white-box gradients and neuron activations provide substantially stronger signals than output-level heuristics on the studied benchmarks. This suggests that the future of TDD will likely depend on tighter alignment between detector design, access assumptions, and the semantic structure of the underlying data modality.