CCNewsPDD: Temporally Unbiased PDD Benchmark

• CCNewsPDD is a CC-News-based benchmark for pre-training data detection that fixes the publication period to August 2017 to eliminate temporal drift as a confounding variable.
• It employs controlled text transformations such as back translation, token masking with BERT, and prompt-based rewriting to generate non-training examples while preserving semantic content.
• Evaluation with NA-PDD on models like Pythia-2.8B and OPT-6.7B yields high AUC scores, showcasing the benchmark’s efficacy in isolating memorization signals over time-based cues.

Searching arXiv for the cited paper and related PDD benchmarks. CCNewsPDD is a benchmark for pre-training data detection (PDD) introduced to evaluate whether a LLM can be tested for membership of specific data in its pre-training corpus under a temporally unbiased setting. It is constructed from the CC-News corpus and is described as a time-drift-free benchmark because it avoids the common practice of labeling older data as “training” and newer data as “non-training,” a practice that can confound PDD evaluation with publication-date effects rather than actual memorization or training-set membership (Tang et al., 22 Jul 2025). In the benchmark’s design, both classes are drawn from the same publication period—August 2017—and non-training instances are generated through controlled text transformations, so that temporal distribution is held constant while surface form is altered (Tang et al., 22 Jul 2025).

1. Definition and motivation

CCNewsPDD is a CC-News-based benchmark for PDD, proposed in the context of a broader effort to identify whether specific data was included in an LLM’s pre-training corpus (Tang et al., 22 Jul 2025). The benchmark addresses a central evaluation problem in prior PDD settings: many “training versus non-training” splits are inadvertently driven by time. When training examples are sampled from older public data and non-training examples from later data, a detector may succeed by exploiting temporal distribution shift rather than by identifying signatures of memorization or membership (Tang et al., 22 Jul 2025).

The benchmark was therefore introduced as a temporally unbiased alternative. Its core premise is that publication date should not determine class membership. Instead, time is fixed and text form is manipulated. This makes CCNewsPDD a benchmark intended to support a more faithful evaluation of PDD methods, especially methods that claim to detect pre-training membership rather than merely distinguish old news from new news (Tang et al., 22 Jul 2025).

The source corpus is the CC-News dataset, cited in the underlying work to Hamborg et al. (2017). The benchmark is additionally motivated by the fact that CC-News was part of the training data for major LLMs such as Pythia and OPT, which makes the setting relevant to real pre-training-data detection rather than a purely synthetic proxy (Tang et al., 22 Jul 2025).

2. Benchmark construction and temporal control

The defining construction principle of CCNewsPDD is temporal alignment. To eliminate time drift, the benchmark uses news articles published in August 2017, selecting a single time slice rather than mixing earlier and later documents (Tang et al., 22 Jul 2025). Both the training and non-training sets are thus drawn from the same publication period.

Non-training examples are not created by choosing later articles. Instead, half of the CCNews corpus is transformed so that the resulting texts preserve semantics while differing sufficiently in surface realization that they are unlikely to have appeared in exactly that form during pre-training (Tang et al., 22 Jul 2025). This design keeps the temporal distribution constant while generating positive and negative examples through content transformation.

Three benchmark variants are defined through distinct transformations:

Variant	Transformation	Stated effect
CCNewsPDD(trans)	Back translation: English → French → English using MarianMT models	Introduces syntactic variation while preserving meaning
CCNewsPDD(mask)	Randomly masks 15% of tokens and uses BERT to fill in plausible substitutions	Causes localized perturbation while preserving overall structure
CCNewsPDD(prompt)	Uses explicit instruction prompting with BART to rewrite or reformulate the original text	Produces a more comprehensive paraphrase at the discourse level

The benchmark design is explicitly intended to preserve the original distribution as much as possible while generating texts that are meaningfully different from the originals (Tang et al., 22 Jul 2025). A plausible implication is that the benchmark operationalizes “non-training” as transformed, same-time data rather than as data with a different publication history. The underlying work also states the associated caveat directly: non-training status is a benchmark construction decision made under the realistic constraint that exact pre-training logs are unavailable, not a direct ground-truth label extracted from a model’s training pipeline (Tang et al., 22 Jul 2025).

3. Relation to earlier PDD benchmarks

CCNewsPDD is contrasted with prior PDD benchmarks including WikiMIA, based on Wikipedia, and ArxivMIA, based on arXiv abstracts (Tang et al., 22 Jul 2025). In standard usage, those benchmarks can rely on release-date-based splits, which create time drift. The resulting evaluation risk is that a detector learns differences associated with recency, topic evolution, editorial conventions, or other date-correlated effects, rather than actual training-set inclusion (Tang et al., 22 Jul 2025).

Against that background, CCNewsPDD’s distinguishing feature is not merely its corpus choice but its split construction. It is explicitly designed to remove temporal bias by using data from the same time period and then transforming part of that data to form the non-training set (Tang et al., 22 Jul 2025). This means the benchmark asks a narrower and more controlled question: whether a method can discriminate original pre-training instances from semantically related but reformulated instances without using publication time as a shortcut.

The benchmark also differs from earlier settings in that it is built from a corpus known to have been included in the pre-training corpora of major LLMs, specifically Pythia and OPT (Tang et al., 22 Jul 2025). This makes the benchmark particularly suitable for open or partially documented pre-training settings. The work is correspondingly careful not to universalize beyond that scope, noting that the benchmark’s assumptions may not transfer unchanged to closed-source models or models with unknown data mixtures (Tang et al., 22 Jul 2025).

4. Evaluation protocol and NA-PDD context

CCNewsPDD is introduced together with NA-PDD, a neuron activation-based PDD algorithm that analyzes differential neuron activation patterns between training and non-training data (Tang et al., 22 Jul 2025). Although the benchmark itself is a dataset construction and evaluation setting rather than a detection algorithm, its reported results are embedded in the NA-PDD evaluation pipeline.

For CCNewsPDD, the models evaluated are Pythia-2.8B and OPT-6.7B, chosen because their pretraining corpora are known to include CC-News (Tang et al., 22 Jul 2025). NA-PDD is compared against nine established PDD methods, divided into reference-free and reference-based baselines:

• Reference-free methods: Loss Attack, Neighbor Attack, Min-K% Prob, Min-K%++ Prob, DC-PDD.
• Reference-based methods: Zlib Compression, Lowercased Text, Smaller Model, Probe Attack.

The evaluation metric is AUC, described as threshold-independent and robust to class imbalance (Tang et al., 22 Jul 2025). The appendix-level setup reported for CCNewsPDD includes reference activation patterns constructed on 200 samples, hyperparameter tuning on a 200-sample validation set, and final evaluation on an 800-sample test set (Tang et al., 22 Jul 2025). Probe Attack uses 200 non-member samples as its training pool, with half of those, 100 samples, used to fine-tune the model; it validates on 400 samples and is reported on the same 800-sample test set (Tang et al., 22 Jul 2025).

The activation-based machinery used in this context records neuron activations by attaching hooks to the FFN layers in Transformer models, and a neuron is considered active if its post-activation output exceeds a threshold $\tau$ (Tang et al., 22 Jul 2025). The activation rule is defined as

$$I(x,n) = \begin{cases} 1, & \text{if } a_n(x) > \tau,\ 0, & \text{otherwise}. \end{cases}$$

Training and non-training activation frequencies are then defined over reference sets: $$f_{train}(n) = \frac{1}{|\mathcal{D}_{train}|} \sum_{x \in \mathcal{D}_{train}} I(x,n), \qquad f_{non}(n) = \frac{1}{|\mathcal{D}_{non}|} \sum_{x \in \mathcal{D}_{non}} I(x,n).$$

Member and non-member neurons are selected by a dominance threshold $\alpha$: $$\mathcal{N}_{mem} = \{n \in \mathcal{N} \mid f_{train}(n) > \alpha \cdot f_{non}(n)\},$$

$$\mathcal{N}_{non} = \{n \in \mathcal{N} \mid f_{non}(n) > \alpha \cdot f_{train}(n)\}.$$

For an input text $x$, similarity scores are computed as overlaps with these neuron sets: $$s_{mem}(x) = \frac{|\mathcal{N}(x) \cap \mathcal{N}_{mem}|}{|\mathcal{N}_{mem}|}, \qquad s_{non}(x) = \frac{|\mathcal{N}(x) \cap \mathcal{N}_{non}|}{|\mathcal{N}_{non}|}.$$

A layer-level discriminative score is defined by

$$S_\ell = |\mathcal{N}^\ell_{mem}| - |\mathcal{N}^\ell_{non}|,$$

and the top-$K$ layers under this score are selected for final scoring. The final ratio is

$$I(x,n) = \begin{cases} 1, & \text{if } a_n(x) > \tau,\ 0, & \text{otherwise}. \end{cases}$$0

with membership predicted by

$$I(x,n) = \begin{cases} 1, & \text{if } a_n(x) > \tau,\ 0, & \text{otherwise}. \end{cases}$$1

These definitions are specific to NA-PDD, but they form the experimental context in which CCNewsPDD is used (Tang et al., 22 Jul 2025).

The reported algorithmic flow is: forward the text through the model and record activations; threshold activations to determine active neurons; compute activation frequencies on training and non-training reference sets; identify member and non-member neurons; rank layers by discriminative score; compute similarity scores on selected layers; and apply a threshold to classify membership (Tang et al., 22 Jul 2025).

5. Quantitative results

The main results on CCNewsPDD are reported in AUC and show NA-PDD as the best method on every CCNewsPDD variant for both evaluated models (Tang et al., 22 Jul 2025). The reported values are as follows.

Benchmark variant	Pythia-2.8B	OPT-6.7B
CCNewsPDD(trans)	92.4%	92.1%
CCNewsPDD(mask)	98.8%	95.2%
CCNewsPDD(prompt)	99.6%	99.7%

For CCNewsPDD(trans), the strongest baseline is Min-K%++ Prob, with 71.0% on Pythia and 76.6% on OPT (Tang et al., 22 Jul 2025). For CCNewsPDD(mask), the strongest baseline is Probe Attack, with 81.3% on Pythia and 82.3% on OPT (Tang et al., 22 Jul 2025). For CCNewsPDD(prompt), the strongest baseline is again Probe Attack, with 90.7% on Pythia and 90.1% on OPT (Tang et al., 22 Jul 2025).

The underlying work states that on OPT with CCNewsPDD(prompt), NA-PDD improves from 90.1% to 99.7% AUC, described in the paper’s wording as a gain of 27.9 AUC points over the second best method (Tang et al., 22 Jul 2025). More generally, the paper identifies CCNewsPDD as the setting in which NA-PDD shows its strongest gains over prior methods.

Several comparative observations are emphasized. On CCNewsPDD(trans), traditional probability-based methods are markedly weaker while NA-PDD reaches the low 90s (Tang et al., 22 Jul 2025). On CCNewsPDD(mask), performance is higher for all methods, but NA-PDD still exceeds the best baseline by roughly 13–17 points (Tang et al., 22 Jul 2025). On CCNewsPDD(prompt), NA-PDD nearly saturates performance, reaching approximately 99.6–99.7% (Tang et al., 22 Jul 2025). The paper also notes that Probe Attack is the strongest non-NA-PDD baseline overall on CCNewsPDD, while reference-based methods are not consistently better than reference-free ones; in some settings, Min-K%++ exceeds Probe Attack (Tang et al., 22 Jul 2025).

The reported figures further indicate that prompt and mask variants are easier for NA-PDD than trans, although all three variants are solved extremely well by the proposed method (Tang et al., 22 Jul 2025). Additional robustness analyses in the paper are not benchmark construction results per se, but they show stability across model size, amount of reference data, activation threshold, dominance threshold, and number of layers, which the authors use to support the benchmark’s utility as an evaluation setting (Tang et al., 22 Jul 2025).

6. Assumptions, caveats, and interpretive boundaries

CCNewsPDD is designed to reduce temporal drift, not to eliminate every possible confounder. The benchmark assumes that same-time data is a good proxy for unbiased evaluation, but the work explicitly notes that this may not remove all distribution differences (Tang et al., 22 Jul 2025). Consequently, temporal alignment should be understood as a targeted control rather than as a guarantee of perfect distributional equivalence.

The benchmark’s transformed non-training examples are intended to preserve semantics, but the transformations do not necessarily preserve naturalness perfectly (Tang et al., 22 Jul 2025). Back-translation, masking plus BERT infilling, and BART-based rewriting can introduce artifacts that affect detectability. This is an important qualification because a detector might, in principle, exploit regularities induced by the transformation process itself. The benchmark’s rationale is that the transformations preserve the original distribution as much as possible while making exact text reuse unlikely, but this is still a constructed rather than observational notion of non-membership (Tang et al., 22 Jul 2025).

Another limitation is epistemic rather than statistical: non-training status is inferred from timing and transformation, not verified against pre-training logs (Tang et al., 22 Jul 2025). The benchmark therefore does not provide direct ground truth from the model’s training pipeline. A plausible implication is that CCNewsPDD evaluates operational PDD under realistic documentation constraints rather than under exhaustive provenance knowledge.

Finally, the benchmark is explicitly tied to open-source or known-corpus settings, since it relies on a corpus known to have appeared in the training data for Pythia and OPT (Tang et al., 22 Jul 2025). It is not presented as a universal benchmark for all LLMs, particularly not for closed-source models with opaque data mixtures (Tang et al., 22 Jul 2025). This scope condition is central to interpreting both the benchmark and the very high AUC values reported on it.

7. Significance within pre-training data detection

Within the paper’s overall argument, CCNewsPDD serves as a corrective to a major flaw in prior PDD evaluation: temporal bias (Tang et al., 22 Jul 2025). Its importance lies less in corpus scale or task novelty than in the methodological claim that credible PDD evaluation requires controlling for publication time. By fixing the time period and generating non-training examples through transformation, the benchmark makes successful detection more plausibly attributable to memorization-related or data-specific signatures rather than to recency effects (Tang et al., 22 Jul 2025).

The benchmark also sharpens the distinction between surface-level heuristics and representation-level signals. The paper uses CCNewsPDD to argue that neuron activation patterns provide a stronger signal for PDD than likelihood-, confidence-, or perplexity-based indicators, since NA-PDD substantially outperforms both reference-free and reference-based baselines across all three variants (Tang et al., 22 Jul 2025). This does not by itself prove that activation-based detection isolates memorization in a fully causal sense, but it does suggest that temporally controlled benchmarks can expose weaknesses in older evaluation protocols and can discriminate more effectively among competing PDD methods.

CCNewsPDD is therefore best understood as a benchmark that redefines what constitutes a rigorous PDD testbed in settings where exact pre-training provenance is unavailable. It combines a known pre-training corpus, a fixed publication window, semantically preserving transformations, and AUC-based evaluation on Pythia-2.8B and OPT-6.7B (Tang et al., 22 Jul 2025). In that configuration, it functions as both an experimental control for time drift and a substantive benchmark for comparing PDD algorithms under a more constrained and arguably more faithful notion of membership detection.