---
title: WikiMIA-32 Benchmark for LLM Membership Inference
url: https://www.emergentmind.com/topics/wikimia-32-benchmark
type: topic
---

# WikiMIA-32 Benchmark for LLM Membership Inference

WikiMIA-32 is a temporally-stratified, model-agnostic benchmark for evaluating membership inference attacks on large language models (LLMs). It was introduced to facilitate systematic and reliable detection of whether specific textual snippets were included in an LLM’s pretraining corpus, a question of central importance for auditing model behaviors in contexts involving copyright, privacy, or benchmark contamination. WikiMIA-32 provides gold-standard labels, rigorous partitioning to avoid contamination, and enables evaluation with only black-box access to models, leveraging a carefully-designed decision rule—Min–k% Prob—that exploits the statistical behavior of outlier token probabilities for robust inference [2310.16789].

## 1. Dataset Construction and Partitioning

WikiMIA-32 is derived exclusively from Wikipedia “Event” category pages, with construction designed to provide true positive (“member”) and true negative (“non-member”) examples. 

- **Temporal Split**: Pages created before 2017 are designated as “members” (assumed to be safely included in pretraining for LLMs such as LLaMA, GPT-NeoX, OPT), while those created after January 1, 2023 are designated as “non-members,” guaranteeing absence from any LLM pretraining set.
- **Filtering**: Pages lacking narrative text—such as “List of …” or “Timeline of …”—are removed to ensure high-quality, content-rich examples.
- **Sampling**: 394 event pages post-2023 were collected as non-members, and 394 pre-2016 pages were randomly sampled as members, yielding a total of 788 distinct pages.
- **Length Buckets**: For each page, the first $N$ tokens (tokenized using the model’s native tokenizer) are extracted, forming length-specific cohorts named “WikiMIA-$N$.” WikiMIA-32 comprises all 788 snippets truncated to 32 tokens ($N=32$).
- **Paraphrase Augmentation**: Each 32-token snippet is also semantically paraphrased using ChatGPT, resulting in 788 original and 788 paraphrased examples per bucket.
- **Gold-Truth Labeling**: Membership labels are determined automatically by construction—no additional human annotation is necessary.

## 2. Evaluation Protocol and Inference Task

The core evaluation task in WikiMIA-32 is membership inference—determining, given an input snippet $x$ (32 tokens), whether $x$ appeared in a model’s pretraining data, under black-box access to the model $f_\theta$ which returns per-token probabilities $p(x_i \mid x_1, \dots, x_{i-1})$.

- **Label Assignment**: $h(x, f_\theta) \in \{0, 1\}$, with $1$ indicating “in-pretraining” (member) and $0$ “not in pretraining” (non-member).
- **Metrics**:
    - **True Positive Rate (TPR)**: $\mathrm{TPR} = \frac{TP}{TP + FN}$
    - **False Positive Rate (FPR)**: $\mathrm{FPR} = \frac{FP}{FP + TN}$
    - **Area Under the ROC Curve (AUC)**: $\mathrm{AUC} = \int_{0}^{1} \mathrm{TPR}(\mathrm{FPR}^{-1}(t))\,dt$
    - **$TPR@5\%FPR$**: TPR measured at a fixed $\mathrm{FPR} = 5\%$
- **Input Variants**: Metrics are reported for both original and paraphrased examples, averaged where appropriate.

## 3. Min–k% Prob: Reference-Free Statistical Detection

WikiMIA-32 supports, and was instrumental in validating, the Min–k% Prob method for membership inference:
- **Negative Log-Likelihoods**: For $x = x_1 \dots x_N$, compute per-token losses $\ell_i = -\log p(x_i \mid x_1, \dots, x_{i-1})$.
- **Outlier Amplification**: For a fixed $k\%$, calculate the average of the top $\lceil k\% \cdot N \rceil$ highest $\ell_i$ values, denoted $Min\textrm{–}k(x) = \frac{1}{E} \sum_{i \in S} \ell_i$ where $S$ indexes the largest losses.
- **Decision Rule**: Membership is predicted if $Min\textrm{–}k(x) \leq \epsilon$ (for a threshold $\epsilon$); $\epsilon$ is swept to generate a full ROC curve, so no fixed threshold is required.
- **Rationale**: Unseen (non-member) snippets frequently contain outlier tokens with anomalously low token probability, inflating $Min\textrm{–}k$ and improving class separation.

## 4. Baseline Methods and Comparative Performance

Five reference-free baselines are implemented for comparative evaluation on WikiMIA-32 (original and paraphrased text), all using only black-box access to model token probabilities:

| Method           | Description                                          | Avg. AUC |
|------------------|-----------------------------------------------------|----------|
| Neighbor         | Neighbourhood curvature attack (DetectGPT)          | 0.65     |
| PPL (LOSS)       | Standard perplexity/loss attack                     | 0.67     |
| Zlib             | Perplexity compared to zlib compression entropy     | 0.65     |
| Lowercase        | Perplexity difference after input lowercasing       | 0.61     |
| SmallerRef       | Perplexity under a smaller reference model          | 0.66     |
| Min–k (ours)     | Min–$k\%$ Prob, top 20% losses averaged             | 0.72     |

- **Performance**: On WikiMIA-32 (original and paraphrased, averaged over 5 LLMs), Min–k achieves an average AUC of 0.72 versus the best baseline at 0.67, representing a 0.05 absolute and 7.4% relative improvement.
- **Interpretation**: This suggests that Min–k’s focus on extremal token losses captures model uncertainty arising from data absence more robustly than mean probability or other heuristics.

## 5. Experimental Details and Models

- **Evaluated Models**: Pythia-2.8B, GPT-NeoX-20B, LLaMA-30B, LLaMA-65B, OPT-66B, all open-source and representative of contemporary LLM architectures pretrained on Wikipedia.
- **Hyperparameter Selection**: $k$ varied in $\{10, 20, 30, 40, 50\}$, optimized on a held-out LLaMA-65B validation split; $k=20\%$ selected.
- **Data Preprocessing**:
    - *Tokenization*: Model-native BPE or tokenizer.
    - *Snippet Extraction*: First $N$ tokens from each example, $N=32$ for WikiMIA-32.
    - *Paraphrasing*: ChatGPT, fixed “paraphrase this in English” prompt.
- **Prompting**: No template prompt; left-to-right token likelihoods are queried with the bare snippet.

## 6. Design Principles and Update Strategy

- **Accuracy**: Temporal cutoff ensures that non-member examples cannot have been present in any model pretraining data up to the 2023 cutoff.
- **Model-Agnosticism**: Evaluation is supported for any LLM with Wikipedia-pretraining, with no LLM-specific adaptation required.
- **Dynamic Growth**: Ongoing, automated data collection via the Wikipedia public API enables continual expansion as new events are added.
- **Generalizability**: The paradigm is expected to remain valid for future models and datasets, provided strict temporal delineation is maintained.

## 7. Significance and Broader Implications

WikiMIA-32 defines a new standard for rigorous, scalable, and reliable membership inference benchmarking for LLMs. Its temporally-segmented, updatable construction, along with gold-standard labeling and support for both paraphrased and verbatim detection, establishes a strong empirical foundation for pretraining data auditing. The Min–k method substantiated on WikiMIA-32 demonstrates robust performance improvement, particularly in identifying model uncertainty associated with out-of-distribution tokens, with direct applications to privacy auditing, copyright detection, and integrity assurance for deployed generative models [2310.16789].

Source: https://www.emergentmind.com/topics/wikimia-32-benchmark