WikiMIA: Pretraining Data Detection Benchmark

• WikiMIA is a benchmark that operationalizes pretraining data detection using temporally split Wikipedia event pages as gold-standard member and non-member examples.
• It employs diverse evaluation settings—verbatim, paraphrase, and varied token lengths—to robustly assess membership inference methods such as Min-K%, ReCaLL, and EM-MIA.
• The benchmark’s scalable, reproducible design underpins its application in auditing LLMs for copyrighted content, privacy risks, and training data contamination.

WikiMIA is a dynamic benchmark for pretraining data detection in LLMs, introduced in “Detecting Pretraining Data from LLMs” (Shi et al., 2023). It operationalizes a specific membership inference setting: given a text sequence and black-box access to an LLM, determine whether the model was trained on that text, despite not knowing the pretraining corpus. Its defining feature is a temporally grounded construction over Wikipedia event pages, using pages created before and after a model’s pretraining cutoff to provide gold-truth member and non-member examples. In the LLM privacy and auditing literature, WikiMIA has become a recurrent evaluation substrate for reference-free, reference-based, label-only, text-only, gray-box, and white-box membership inference attacks (Shi et al., 2023).

1. Problem setting and motivation

WikiMIA was created to address the pretraining data detection problem for LLMs: whether a target text was part of a model’s pretraining data, given only model access and not the corpus itself. The motivating concern is that LLM pretraining data can reach up to trillions of tokens and is rarely disclosed, yet it may contain copyrighted materials, personally identifiable information, and benchmark test data (Shi et al., 2023).

The benchmark was proposed against a backdrop in which existing MIAs were poorly matched to modern LLM pretraining. Earlier approaches either assumed access to reference models trained on data similar to the original pretraining distribution or relied on exact duplication, which the original paper characterizes as largely infeasible for massive LLM corpora (Shi et al., 2023). WikiMIA therefore addresses both an epistemic problem—lack of visibility into pretraining corpora—and an evaluation problem—lack of verifiable ground truth for membership labels.

This framing also fixed the benchmark’s role in downstream auditing. The original study explicitly positioned the task as relevant to auditing for copyrighted, private, or test data in commercial and research LLMs, and later papers continued to use WikiMIA for privacy, contamination, and compliance analyses across increasingly restrictive access regimes (Shi et al., 2023).

2. Temporal construction and gold-truth membership

The core design principle of WikiMIA is temporal labeling through timestamped Wikipedia event pages. Member data are Wikipedia event pages created before a model’s pretraining cutoff, with the original construction using pages created before 2017 as likely members. Non-member data are event pages created after the cutoff, with the original benchmark using pages created after January 1, 2023 as guaranteed unseen for models pretrained earlier (Shi et al., 2023).

The construction pipeline is automated. Non-member pages are collected through the Wikipedia API from event categories after the cutoff date; member pages are selected from a much earlier period; and low-information pages such as “Timeline of ...” and “List of ...” are filtered out. For fairness, the original benchmark formation randomly selected an equal number of member and non-member pages, exemplified by 394 of each (Shi et al., 2023).

WikiMIA includes multiple evaluation settings. In the verbatim setting, the benchmark uses the original article text. In the paraphrase setting, ChatGPT paraphrases the originals to test robustness beyond exact string match. It also uses length buckets, truncating texts to 32, 64, 128, and 256 tokens to expose sequence-length effects (Shi et al., 2023).

The principal methodological claim of the benchmark is that this temporal split provides “perfect ground truth” for non-membership: an event article created after model pretraining cannot have appeared in the pretraining corpus. The member label is probabilistic rather than formally guaranteed, but the original design argues that very early event pages have high likelihood of inclusion in any Wikipedia-containing LLM corpus (Shi et al., 2023). This suggests that WikiMIA is best understood as a high-confidence temporal benchmark with an especially strong guarantee on the non-member side.

3. Benchmark properties and evaluation protocol

The original work characterizes WikiMIA as accurate, general, dynamic, versatile, and scalable. It is accurate because recency prevents contamination of the non-member set; general because it applies to any model using Wikipedia data; dynamic because it can be updated as new event pages appear; versatile because it supports verbatim, paraphrase, and varying-length settings; and scalable because the pipeline is automated (Shi et al., 2023).

Evaluation is usually reported with AUC and a low-FPR operating metric such as TPR@5%FPR in the original paper, while later work also standardized TPR@1%FPR for more stringent auditing regimes (Shi et al., 2023, Xie et al., 2024). The benchmark has been used across LLaMA, GPT-NeoX, Pythia, OPT, and Mamba families, among others (Shi et al., 2023, Xie et al., 2024).

Several empirical regularities were established early. In the original study, detection improved with longer texts and larger models, and the benchmark remained meaningful under paraphrasing, indicating that some methods were not restricted to exact duplication signals (Shi et al., 2023). Later work retained the short-length variants—especially WikiMIA-32, WikiMIA-64, and WikiMIA-128—as standardized stress settings for sequence-level and token-level membership analysis (Xie et al., 2024).

These properties explain why WikiMIA was rapidly reused. Its value is not only the temporal split itself, but also the fact that the split is reproducible, automatically refreshable, and compatible with black-box or gray-box evaluation protocols.

4. Detection methodologies evaluated on WikiMIA

The benchmark and its originating paper introduced Min-K% Prob, a reference-free score built on the hypothesis that unseen texts contain a few outlier tokens with very low probability under the LLM, whereas seen texts are less likely to contain such outliers. For a sequence $x=(x_1,\ldots,x_N)$, the score averages token log-likelihoods over the bottom $k\%$ of tokens:

$$Min\text{-}k(x)=\frac{1}{E}\sum_{x_i\in \text{Min-K\%}(x)} \log p(x_i\mid x_1,\ldots,x_{i-1}).$$

The original experiments reported that Min-K% achieved a 7.4% improvement on WikiMIA over previous methods and illustrated an average AUC of 0.72 versus 0.67 for PPL in the example table (Shi et al., 2023).

“Min-K%++: Improved Baseline for Detecting Pre-Training Data from LLMs” reformulated the signal in terms of local maxima and per-prefix calibrated likelihood. Instead of raw log-probability, it normalizes each token by the conditional mean and standard deviation of the next-token distribution. On WikiMIA, Min-K%++ reported average AUROC values of 78.3, 78.6, and 77.2 for lengths 32, 64, and 128, and outperformed the runner-up by 6.2% to 10.5% in detection AUROC averaged over five models (Zhang et al., 2024).

“ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods” introduced a context-conditioned score based on non-member prefixes:

$\text{ReCaLL}(x)=\frac{LL(x\mid P)}{LL(x)}.$

Its empirical claim was that conditioning member data on non-member prefixes induces a larger decrease in log-likelihood than conditioning non-member data the same way. On WikiMIA, ReCaLL reported average AUCs of 90.1, 91.3, and 90.0 for lengths 32, 64, and 128, exceeding Min-K++% by 14.8–15.4 points (Xie et al., 2024).

“Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding” extended this line by contrasting member and non-member contexts rather than using only non-member prefixes. Its score is

$$s(x,\mathcal{M})=\frac{LL(x\mid P_{\text{non-member}})-\gamma\cdot LL(x\mid P_{\text{member}})}{LL(x)}.$$

On WikiMIA-32, it reported an AUC of 95.8 and TPR@5%FPR of 75.1, and the paper further reported strong robustness under random deletion, synonym substitution, and paraphrasing (Wang et al., 2024).

“Detecting Training Data of LLMs via Expectation Maximization” introduced EM-MIA, which iteratively refines membership scores and prefix scores through an EM procedure. On WikiMIA it reported more than 96% AUC-ROC across all settings, including 99.0% on OPT-66B at lengths 32 and 64 (Kim et al., 2024). “Probing LLMs for Pre-training Data Detection” took a different route by training a linear probe over internal activations and reported an AUC of 69.8 on WikiMIA, outperforming its baselines by about 5 points (Liu et al., 2024). In the label-only setting, “Towards Label-Only Membership Inference Attack against Pre-trained LLMs” proposed PETAL, which uses per-token semantic similarity to approximate perplexity and reported WikiMIA AUCs of 0.64 for Pythia, 0.62 for OPT, 0.58 for LLaMA2, and 0.60 for Falcon (He et al., 26 Feb 2025).

Taken together, these results show that WikiMIA became not merely a dataset, but a comparative protocol around which several distinct MIA paradigms—log-likelihood tails, calibrated likelihood, conditional likelihood ratios, contrastive conditioning, EM refinement, activation probing, and text-only semantic approximation—were developed and benchmarked.

5. Benchmark evolution: WIKIMIA-24 and WikiMIA-25

As model release cycles accelerated, several papers argued that the original temporal cutoff was no longer adequate for newly released LLMs. This led to updated benchmark variants.

Benchmark	Temporal construction	Intended target
WikiMIA	Member pages created before 2017; non-member event pages created after January 1, 2023; verbatim, paraphrase, and 32/64/128/256-token settings (Shi et al., 2023)	Gold-truth pretraining data detection for Wikipedia-containing LLMs
WIKIMIA-24	Same sampling and cleaning pipeline, with cutoff moved to March 1, 2024; member articles before March 1, 2024 and non-members after March 2024, while member data are restricted to pages created before 2017 (Fu et al., 2024)	Contemporary aligned and unaligned LLMs released before March 2024
WikiMIA-25	Built from 2025 Wikipedia dumps; members from January 2015 to December 2016; non-members from March 2025 to September 2025; 32/64/128-word buckets (Yi et al., 16 Jan 2026)	Modern proprietary and open-source LLMs in black-box and gray-box settings

WIKIMIA-24 was introduced by “MIA-Tuner: Adapting LLMs as Pre-training Text Detector,” which argued that the original WikiMIA could not fairly benchmark models whose plausible pretraining data extended beyond January 2023. On WIKIMIA-24, MIA-Tuner reported average AUCs of 0.97 for aligned LLMs and 0.98 for unaligned LLMs, compared with prior averages around 0.63 and 0.78, respectively (Fu et al., 2024).

WikiMIA-25 was introduced by “Membership Inference on LLMs in the Wild” for black-box and gray-box evaluation on modern proprietary APIs. The benchmark includes 759 pairs at 32 words, 875 pairs at 64 words, and 529 pairs at 128 words, with smaller balanced subsets sometimes used due to API cost. On this benchmark, the text-only SimMIA method substantially outperformed the black-box baseline SaMIA, for example reporting AUC 86.7 versus 62.6 on Claude-4.5-Haiku and 75.7 versus 65.7 on GPT-5-Chat (Yi et al., 16 Jan 2026).

The benchmark family also supports white-box internal-signal studies. “AttenMIA: LLM Membership Inference Attack through Attention Signals” used WikiMIA-32 and reported, for Llama2-13b, ROC AUC 0.996 and TPR@1%FPR 87.9% with perturbation-based attention features, far above RECALL’s 0.922 AUC and 13.3% TPR@1%FPR in the cited comparison (Zaree et al., 26 Jan 2026). This suggests that the WikiMIA lineage now functions as a benchmark suite spanning black-box text-only, gray-box log-probability, and white-box representation-level attacks.

6. Interpretation, limitations, applications, and name ambiguity

WikiMIA’s main limitation, as articulated most explicitly by EM-MIA, is that it may be distribution-shift friendly rather than a pure memorization benchmark. Because members and non-members are temporally separated, strong performance can arise from detecting temporal or topical distributional shifts instead of bona fide membership. EM-MIA reports that the most discriminative prefixes on WikiMIA are often non-members and argues that the benchmark may therefore overstate the difficulty of membership inference “in the wild”; on random-split settings such as MIMIR, the same paper states that methods fall close to random guessing (Kim et al., 2024).

That critique does not negate the benchmark’s utility. The original paper used WikiMIA and Min-K% in three practical scenarios—copyrighted book detection, contaminated downstream example detection, and privacy auditing of machine unlearning—and presented the benchmark as a fair, reproducible basis for auditing opaque LLMs (Shi et al., 2023). Later work extended this operational role. For example, “Reconstruction of Differentially Private Text Sanitization via LLMs” used WikiMIA to test recovery from DP-sanitized text and reported black-box word-level reconstruction success rates at $\epsilon=12$ of 72.18% on LLaMA-2 (70B), 82.39% on LLaMA-3 (70B), 91.20% on ChatGPT-4o, and 94.01% on Claude-3.5 (Sonnet) (Pang et al., 2024). This suggests that WikiMIA functions not only as a benchmark for MIAs, but also as an experimental substrate for privacy-risk studies that depend on realistic, temporally grounded text.

A final point of encyclopedic clarification is terminological. In contemporary LLM research, “WikiMIA” refers to the Wikipedia-based benchmark for pretraining data detection introduced in 2023 (Shi et al., 2023). An unrelated 2012 neuroimaging paper, “Online open neuroimaging mass meta-analysis,” also used “WikiMIA” to denote a wiki-based mass meta-analysis system built on MediaWiki and a computational web service (Nielsen et al., 2012). The two usages are historically independent, and current NLP literature overwhelmingly uses the term in the benchmark sense.