Tab-MIA: Benchmark for Tabular MIAs
- Tab-MIA is a benchmark dataset that evaluates membership inference attacks on large language models fine-tuned with serialized tabular data.
- It comprises five diverse table collections across three domains and six encoding formats, highlighting variations in privacy leakage.
- Results demonstrate rapid leakage emergence and different attack success rates depending on fine-tuning epochs, encoding formats, and model types.
Tab-MIA is a benchmark dataset for evaluating membership inference attacks (MIAs) on tabular data in LLMs. It was introduced to study privacy risks that arise when LLMs are fine-tuned on serialized tables, where personally identifiable information (PII), explicit attributes, recurring schemas, heterogeneous data types, and column-level semantics can make structured data leak differently from free-form text. The benchmark comprises five data collections, each represented in six encoding formats, and is used to evaluate black-box likelihood-based MIAs on LLMs fine-tuned with tabular data as well as on pretrained models exposed to Wikipedia-derived tables (German et al., 23 Jul 2025).
1. Conceptual scope and motivation
Tab-MIA was proposed to fill a specific gap: earlier MIA benchmarks such as BookMIA, WikiMIA, and MIMIR addressed textual corpora, but there was no benchmark specifically for tabular data in LLMs, and no systematic study of how table encoding format changes membership leakage (German et al., 23 Jul 2025). The motivating concern is practical rather than merely methodological. LLMs are increasingly adapted to ingest structured data such as healthcare records, financial spreadsheets, census data, and web tables, typically by serializing them into text-like forms such as JSON, HTML, Markdown, or CSV-like strings. In that setting, an attacker may try to infer whether a specific table or row chunk was part of training.
The benchmark rests on the claim that tabular data differs materially from unstructured text. Rows correspond to entities, columns correspond to attributes, and records may combine numeric, categorical, textual, date, or identifier fields. Even when individual fields are common, combinations across columns can uniquely identify a record. The content can be low-entropy in the sense of being short and repetitive, yet still highly explicit. In addition, serialization is not neutral: the encoding format can change tokenization, schema overhead, and the concentration of content tokens, which in turn changes what an LLM memorizes and what an MIA can detect (German et al., 23 Jul 2025).
A plausible implication is that tabular privacy auditing for LLMs requires different instrumentation from text-only auditing. Tab-MIA operationalizes that view by holding the underlying table content fixed while varying its serialization and evaluating attacks under a consistent black-box scoring regime.
2. Benchmark composition and construction
Tab-MIA contains five table collections spanning three source domains: Wikipedia-derived tables, Census, and Housing. The benchmark mixes short-context and long-context sources. For the short-context collections, the original question-answering or fact-verification tasks are stripped down to the tables themselves so that the benchmark isolates memorization of the table rather than memorization of an associated question (German et al., 23 Jul 2025).
| Collection | Source domain | Filtered records / features |
|---|---|---|
| WikiTableQuestions (WTQ) | Wikipedia | 1,290; at least 5 features |
| WikiSQL | Wikipedia | 17,900; at least 5 features |
| TabFact | Wikipedia | 13,100; at least 5 features |
| Adult (Census Income) | Census | 2,440; 15 features |
| California Housing | Housing | 1,030; 10 features |
The construction pipeline is explicitly designed to prevent inflated attack success due to accidental duplicates. Each table appears only once in the benchmark. For short-context tables, any table whose Line-Separated serialization exceeds 10,000 characters is removed. For long-context collections, the tables are split into row-wise chunks of 20 records each so that they fit within the model’s context window while preserving structure. As a result, the benchmark unit for MIA is either a full table, in the short-context case, or a 20-row chunk, in the long-context case (German et al., 23 Jul 2025).
Membership labels are defined cleanly. For each dataset, 50% of the tables or chunks are designated as members and used in fine-tuning, while the remaining 50% serve as non-members for MIA evaluation. All encoded variants are saved as JSONL files, which supports reproducible evaluation under identical train/non-train partitions (German et al., 23 Jul 2025).
This design matters because the benchmark does not treat tabular leakage as a single-dataset phenomenon. Instead, it distributes evaluation across collections that differ in length, schema regularity, and domain semantics, making it possible to compare membership inference across multiple table regimes rather than along one narrow axis.
3. Encodings, attack setting, and evaluation task
A defining feature of Tab-MIA is that every table collection is represented in six serialization formats: JSON, HTML, Markdown, Key-Value Pair, Key-is-Value, and Line-Separated (German et al., 23 Jul 2025). These formats vary systematically in structural overhead and content density.
JSON represents a table as an array of row objects and preserves row/column structure clearly while adding punctuation and structural tokens. HTML uses <table>, <tr>, <th>, and <td> markup and therefore introduces substantial structural redundancy. Markdown preserves headers and separators with moderate verbosity. Key-Value Pair is row-centric and content-dense, with rows written as flat Column: value sequences. Key-is-Value turns cells into short natural-language statements such as “Name is Alice.” Line-Separated is a minimal CSV-like plain-text form and is especially concentrated on raw row content (German et al., 23 Jul 2025).
The benchmark hypothesis is that serialization changes how much model capacity is spent on schema tokens versus actual values, how dispersed the sequence is, and how easy it becomes for likelihood-based attacks to separate members from non-members. Tab-MIA therefore treats encoding format as a first-class experimental variable rather than a preprocessing convenience.
The threat model is black-box. The attacker has query access sufficient to obtain token probabilities or log-likelihood-based scores on candidate serialized tables, does not inspect model internals, and decides whether a given serialized table was part of fine-tuning. The evaluated attacks are reference-free black-box methods rather than shadow-model or reference-based attacks such as LiRA. The benchmark evaluates three state-of-the-art MIAs: LOSS (PPL), Min-K%, and Min-K%++. The experiments use 20.0% for Min-K and Min-K++, and the paper reports AUROC and TPR@FPR = 5% as the primary metrics (German et al., 23 Jul 2025).
Formally, a sample is a member if it was included in the fine-tuning set for the target model. For long-context data, “sample” means a 20-row chunk rather than the entire original table. This distinction is operationally important because it places membership inference at the level of the serialized unit actually seen by the model.
4. Experimental protocol and model suite
The benchmark fine-tunes four open-weight pretrained LLMs: LLaMA-3.1 8B, LLaMA-3.2 3B, Gemma-3 4B, and Mistral 7B (German et al., 23 Jul 2025). These models were selected to span different model families, parameter scales, tokenizers, and pretraining characteristics.
Fine-tuning is performed with QLoRA using 4-bit quantization, batch size 2, learning rate , optimizer paged_adamw_8bit, warmup steps 20, random seed 42, and HuggingFace Transformers plus PEFT on a single RTX 6000 GPU with 48GB VRAM. The default fine-tuning horizon is 3 epochs, and the paper also studies 1 and 2 epochs to measure how quickly tabular memorization emerges (German et al., 23 Jul 2025).
The attacks themselves are not redesigned for structured data. Instead, a table or row chunk is serialized into one of the six formats, the resulting token sequence is scored by the target LLM, and the attack statistic is computed on that sequence. This methodological choice is central to the benchmark: it isolates the effect of serialization on leakage while keeping the MIA machinery constant.
The paper also includes a pretrained-model setting without fine-tuning. In that experiment, WTQ tables are treated as candidate members under the assumption that some Wikipedia tables may already have appeared in pretraining corpora, while non-members are synthetic tables generated with GPT-4o mini. The attack used there is Min-K++ 20.0%, and the aim is to probe pretraining-set membership rather than fine-tuning leakage (German et al., 23 Jul 2025).
5. Main empirical findings
The central empirical conclusion is that LLMs do memorize tabular data enough for effective membership inference, and that this vulnerability depends strongly on fine-tuning duration, serialization format, model family, and dataset type (German et al., 23 Jul 2025).
The epoch study is the clearest demonstration of rapid leakage emergence. Using Line-Separated encoding with Min-K++ 20.0%, AUROC increases sharply from epoch 1 to epoch 3. On WTQ, LLaMA-3.1 8B rises from 61.6 to 80.8 to 93.6, LLaMA-3.2 3B from 57.6 to 74.8 to 89.7, Mistral 7B from 69.7 to 88.4 to 97.7, and Gemma-3 4B from 59.3 to 77.0 to 89.6. Comparable monotone or near-monotone increases appear on WikiSQL and TabFact. Adult is more moderate, while California Housing reaches high vulnerability by 3 epochs, with LLaMA-3.1 8B at 87.8 and Mistral 7B at 86.8 (German et al., 23 Jul 2025).
At stricter operating points, the leakage remains substantial. On WTQ, Mistral 7B with Min-K++ after 3 epochs achieves TPR@FPR = 5% of 88.2, and LLaMA-3.1 8B reaches 66.4. On California Housing, Mistral 7B reaches 56.8 and LLaMA-3.1 8B reaches 52.5. These numbers show that the signal is not confined to threshold-averaged AUROC (German et al., 23 Jul 2025).
Across attacks, Min-K++ 20.0% is usually the strongest. On WikiSQL with Line-Separated encoding after 3 epochs, LLaMA-3.1 8B yields 76.5 for PPL, 78.1 for Min-K, and 88.9 for Min-K++; Mistral 7B yields 75.1, 77.6, and 87.8, respectively. This indicates that focusing on the lowest-probability tokens is more effective than full-sequence loss for serialized tables (German et al., 23 Jul 2025).
Encoding format is one of the benchmark’s most consequential findings. Across many settings, Line-Separated and Key-Value Pair are the most vulnerable formats, whereas JSON and HTML are often lower-risk. For Mistral 7B on WTQ with Min-K++, Markdown reaches 94.2, JSON 82.7, HTML 92.9, Key-Value Pair 94.9, Key-is-Value 89.7, and Line-Separated 97.7. The California Housing results are even more stark: for the same model and attack, Markdown is 80.0, JSON 54.5, HTML 50.6, Key-Value Pair 92.6, Key-is-Value 74.9, and Line-Separated 86.8. In the paper’s interpretation, JSON and HTML inject structural redundancy, while flat encodings concentrate actual table content (German et al., 23 Jul 2025).
Dataset type matters as well. The short-context Wikipedia-derived collections—WTQ, WikiSQL, and TabFact—are the most vulnerable overall, reaching values up to 97.7, 88.9, and 89.9. Adult is the hardest dataset for MIAs in the benchmark, with best AUROC around 71.5, whereas California Housing is more vulnerable and reaches 87.8 and 92.6 in some model/encoding combinations (German et al., 23 Jul 2025).
Cross-format transfer is partial but nontrivial. In a cross-format generalization experiment using Gemma-3 4B and Min-K++ 20.0%, the best attack performance is generally on the diagonal, where training and attack encodings match. Yet transfer remains meaningfully above chance. The paper gives the example that training and evaluating on Markdown yields 85.2 AUROC, while training on Markdown and attacking with Key-Value Pair yields 68.9 and with Line-Separated 69.4. The experiment is described in the text as TabFact, though the figure caption says WTQ (German et al., 23 Jul 2025).
Pretrained models without fine-tuning also show moderate leakage on WTQ. For LLaMA-3.1 8B, the reported Min-K++ scores are 69.3 for Markdown, 62.4 for JSON, 66.7 for HTML, 72.0 for Key-Value Pair, 71.6 for Key-is-Value, and 71.5 for Line-Separated. For LLaMA-3.2 3B, Mistral 7B, and Gemma-3 4B, the best scores are also with Key-Value Pair, reaching 66.2, 66.9, and 63.4. These are lower than fine-tuned results but still above chance, which suggests some pretraining memorization of public tabular content (German et al., 23 Jul 2025).
6. Position within the broader MIA literature
Within the literature, Tab-MIA occupies a specific niche: it is a benchmark for MIAs on LLMs trained on serialized tables, rather than a new attack or a general synthetic-data audit framework (German et al., 23 Jul 2025). That distinction is useful because tabular membership inference now spans several partially separate research lines.
One adjacent line concerns text MIAs that use selective scoring rather than full-sequence aggregation. Tag&Tab is a black-box membership inference attack for LLM pretraining data detection that first tags semantically important, high-entropy keywords and then computes membership scores from the model’s probabilities on those keywords rather than on all tokens. The paper explicitly notes that this aligns with the idea of a tabulated keyword-based MIA, even though its operational setting is document-level text rather than structured tables (Antebi et al., 14 Jan 2025). This suggests that “Tab-MIA” can denote either a specific tabular-data benchmark or, in a different usage, a tabulation-style MIA statistic.
A second line concerns synthetic tabular data rather than LLM fine-tuning on serialized tables. TAMIS targets differentially private graphical-model synthesizers such as MST and PrivBayes by recovering the graphical structure from the released synthetic dataset and computing a factorized density ratio ; for MST, it achieves exact graph recovery and stronger membership inference than MAMA-MIA in the reported experiments (Andrey et al., 1 Apr 2025). Synth-MIA broadens that perspective into a model-agnostic auditing framework for tabular synthetic data, implementing 13 attack methods through a Scikit-Learn-like API and advocating worst-case leakage estimation by taking the maximum across attacks; it also reports weak correlation between DCR-Prop and Max-AUC, with and (Ward et al., 22 Sep 2025). A further branch studies diffusion-based tabular generators directly: step-wise error comparison attacks yield AUC 97% on Shoppers, 83% on Default, and 99% on Default′ for TabDDPM, while TabSyn remains below 51% AUC under the tested attacks (Cheng et al., 16 Oct 2025).
In that broader context, Tab-MIA’s distinctive contribution is not that it exhausts tabular MIA, but that it makes one overlooked threat measurable: membership leakage in LLMs fine-tuned on serialized tabular inputs, with encoding format treated as a controlled privacy variable. The paper also states several limitations. It includes five datasets from only three broad source domains, evaluates only three black-box reference-free attacks, studies serialized tables rather than all table-model architectures, chunks long tables into 20-row segments, and does not deeply explore defenses such as differential privacy, regularization, or redaction (German et al., 23 Jul 2025).
The benchmark’s main implication is therefore methodological. Serialization choice is not merely a downstream engineering decision; it is a privacy-relevant design choice. Flat, content-dense encodings such as Line-Separated and Key-Value Pair often maximize leakage, while structured encodings such as JSON and HTML often reduce it, though they do not eliminate it. Even limited fine-tuning can produce high-vulnerability regimes, and exact secrecy of the serialization format is not a reliable defense because cross-format attacks retain useful signal (German et al., 23 Jul 2025).