Synth-MIA: Privacy Auditing for Synthetic Data

Updated 12 July 2026

Synth-MIA is a framework and Python library that audits privacy leakage in tabular synthetic data using diverse membership inference attack methods.
It couples a formal auditing procedure with a Scikit-Learn-like testbed to measure attack metrics such as AUC-ROC and TPR at fixed FPR.
Empirical benchmarks show that higher synthetic data quality may coincide with increased privacy leakage, emphasizing adversary-aware evaluation.

Synth-MIA is a framework and Python library for auditing privacy leakage in tabular synthetic data via membership inference attacks (MIAs) under model-agnostic threat models. Introduced in "Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis" (Ward et al., 22 Sep 2025), it couples a formal auditing procedure with a Scikit-Learn-like testbed implementing 13 attack methods, and it was used in what the paper describes as the largest tabular synthesis privacy benchmark to date. Its central premise is adversary-aware evaluation: higher synthetic data quality can coincide with greater privacy leakage, similarity-based privacy metrics can correlate only weakly with MIA results, and even differentially private generators such as PATEGAN can fail to preserve privacy under attack (Ward et al., 22 Sep 2025).

1. Conceptual basis and motivation

Tabular generative models are increasingly used for privacy-preserving data release, data augmentation, and imputation. Synth-MIA is motivated by the observation that such models can memorize parts of the training data or overfit the training distribution, so synthetic records may become close to, or effectively identical to, training records. In that setting, privacy risk is not adequately characterized by non-adversarial similarity statistics alone (Ward et al., 22 Sep 2025).

The dominant privacy proxy in the tabular synthesis literature is Distance to Closest Record–Proportion (DCR-Prop). For each training record, DCR-Prop compares its nearest-neighbor distance in the synthetic dataset with its nearest-neighbor distance in a reference dataset, and reports the proportion of training records whose nearest neighbor is synthetic rather than reference. Synth-MIA treats this family of metrics as insufficient because they lack an explicit adversary model and do not directly quantify whether an attacker can infer training membership. In the benchmark associated with Synth-MIA, DCR-Prop shows only weak correlation with MIA-based leakage, with $r \approx 0.225$ and $p \approx 5.9 \times 10^{-13}$ (Ward et al., 22 Sep 2025).

Membership inference attacks recast privacy as an attack-performance problem. Rather than asking whether synthetic samples look close to real ones, they ask whether an adversary can determine whether a target record was part of the generator’s training set. This shifts evaluation from geometric resemblance to operational leakage. The framework therefore measures privacy primarily through attack metrics such as AUC-ROC and TPR at fixed FPR, especially low-FPR operating points (Ward et al., 22 Sep 2025).

2. Threat model and formal auditing objective

Synth-MIA standardizes two threat models. In the no-box setting, the attacker has access only to the synthetic dataset $S$ . In the no-box-calibrated setting, the attacker has access to $S$ and to a reference dataset $R$ sampled independently from the same population as the training set $T$ , but disjoint from $T$ . The attacker does not know the generator architecture, hyperparameters, or training procedure (Ward et al., 22 Sep 2025).

An attack is formalized as a decision rule

$\mathcal{A}(x^{\star}) = \mathbb{I}\left[f(x^{\star}) > \gamma\right],$

where $f(x^\star)$ is an attack-specific scoring function and $\gamma$ is a threshold. Attack performance is then summarized by standard binary metrics. Synth-MIA emphasizes

$p \approx 5.9 \times 10^{-13}$ 0

and

$p \approx 5.9 \times 10^{-13}$ 1

with $p \approx 5.9 \times 10^{-13}$ 2 denoting an attack score and $p \approx 5.9 \times 10^{-13}$ 3 the operating threshold (Ward et al., 22 Sep 2025).

The framework is explicitly ensemble-based. Given a collection of attacks $p \approx 5.9 \times 10^{-13}$ 4, empirical privacy leakage is estimated through the strongest observed attack, in the sense of the supremum of $p \approx 5.9 \times 10^{-13}$ 5 subject to an FPR constraint. This follows an empirical-differential-privacy viewpoint: privacy is not summarized by the performance of a single attack, but by the maximum leakage observed across a structured attack set. The same formulation also motivates an effective empirical epsilon through inequalities of the form

$p \approx 5.9 \times 10^{-13}$ 6

which Synth-MIA uses as an interpretive device rather than as a replacement for formal differential privacy (Ward et al., 22 Sep 2025).

3. Testbed architecture and attack taxonomy

Synth-MIA is organized into three modules: preprocessing, attack execution, and evaluation. The preprocessing module receives training, holdout, synthetic, and reference datasets; performs type casting, scaling, and encoding; and, importantly, fits encoders on synthetic or reference data rather than on training data, so that the attacker is not granted unrealistic access to the private training distribution. The attack module implements MIAs as subclasses of a BaseAttacker class with a Scikit-Learn-like API. The evaluation module computes standardized privacy metrics, including AUC-ROC, TPR@FPR, accuracy, precision, recall, advantage, privacy gain, empirical epsilon, and Brier score (Ward et al., 22 Sep 2025).

The 13 implemented attacks span multiple leakage mechanisms rather than a single adversarial doctrine.

Family	Representative methods	Primary signal
Distance-based / memorization-focused	DCR, DCR-Diff, Local Neighborhood, Monte Carlo	Nearest-neighbor proximity or local occupancy
Density-based / overfitting-focused	DOMIAS, Density Estimator, Gen-LRA, classifier-based density ratio	$p \approx 5.9 \times 10^{-13}$ 7, $p \approx 5.9 \times 10^{-13}$ 8, or local likelihood ratios
Local neighborhood / mixed signal	DPI	Synthetic-to-reference density in a local neighborhood
Classifier-based	LOGAN, classifier attack	Discriminator or classifier output

Several scoring functions illustrate the heterogeneity of the attack surface. The DCR score is

$p \approx 5.9 \times 10^{-13}$ 9

so closer synthetic neighbors imply stronger membership evidence. DOMIAS uses a density ratio,

$S$ 0

which targets distributional overfitting. DPI computes a local synthetic-to-reference ratio,

$S$ 1

and is explicitly oriented toward local plagiarism-like behavior. Gen-LRA uses a local likelihood-ratio construction based on density estimation and the effect of adding a candidate record to the reference set (Ward et al., 22 Sep 2025).

This diversity is not incidental. The benchmark shows that different attacks exploit different failure modes—direct memorization, local density distortion, or broader distributional overfitting—so a single attack family cannot be assumed to dominate across datasets and model classes (Ward et al., 22 Sep 2025).

4. Benchmark design and empirical findings

The benchmark underpinning Synth-MIA uses 48 datasets from the OpenML-CC18 curated classification suite after excluding extremely high-dimensional cases. Each dataset is split 80/20 into train and test, synthetic datasets are generated at $S$ 2, $S$ 3, and $S$ 4 the training size, and up to 1000 training samples and 1000 holdout samples are used as evaluation points. The test set is divided equally into holdout and reference subsets (Ward et al., 22 Sep 2025).

Nine tabular generators are evaluated: CT-GAN, AdsGAN, PATEGAN, TVAE, N-Flows, Adversarial Random Forest (ARF), Tab-DDPM, TabSyn, and REaLTabFormer. Utility is measured with XGBoost classifier AUC on real holdout data. Fidelity is measured with MMD and JSD. Privacy is measured by running the full attack suite and reporting both per-attack metrics and the maximum leakage across attacks (Ward et al., 22 Sep 2025).

The benchmark’s principal empirical finding is that higher synthetic data quality corresponds to greater privacy leakage. Models with stronger utility and fidelity scores also tend to exhibit higher Max-AUC and Max-TPR@FPR. Among the reported model means, REaLTabFormer attains Max-AUC $S$ 5 and TPR@FPR $S$ 6 $S$ 7, while TabSyn attains Max-AUC $S$ 8 and TPR@FPR $S$ 9 $S$ 0. At stricter operating points, TPR@FPR $S$ 1 values around $S$ 2 to $S$ 3 indicate that a non-trivial fraction of training records can be identified with zero false positives (Ward et al., 22 Sep 2025).

No single attack is uniformly best. Gen-LRA, DPI, and DCR-Diff are the strongest attacks on average, and Gen-LRA is best in roughly $S$ 4 to $S$ 5 of synthetic datasets depending on whether one ranks by AUC or TPR@FPR $S$ 6. Yet every implemented attack is the best attack for some generator-dataset combinations. This is the core empirical justification for Synth-MIA’s ensemble design: privacy leakage is routinely underestimated if auditing is restricted to one attack family (Ward et al., 22 Sep 2025).

5. Similarity metrics, differential privacy, and interpretation of leakage

Synth-MIA’s most direct methodological intervention is its argument against equating privacy with similarity control. DCR-Prop can remain near values conventionally regarded as benign while MIA leakage is materially elevated. TabSyn and REaLTabFormer illustrate this mismatch: they have strong synthetic-data quality and strong MIA leakage even when non-adversarial similarity metrics appear acceptable. This suggests that privacy assessment based on nearest-neighbor similarity alone is structurally incomplete (Ward et al., 22 Sep 2025).

The framework also stresses that formal differential privacy claims should still be audited empirically. PATEGAN, a differentially private GAN using teacher aggregation, has lower mean leakage than several non-DP models when aggregated over all datasets, but its worst-case behavior remains substantial. For the ten datasets with highest leakage, PATEGAN at $S$ 7 reaches AUC $S$ 8, TPR@FPR $S$ 9 $R$ 0, and TPR@FPR $R$ 1 $R$ 2. A plausible implication is that implementation details, dataset structure, and worst-case instances remain decisive even when a generator is marketed as differentially private (Ward et al., 22 Sep 2025).

The framework’s practical reading of attack metrics is operational rather than abstract. In its usage discussion, AUC near $R$ 3 is treated as little leakage; TPR@FPR $R$ 4 near $R$ 5 corresponds to few perfectly detectable records; and TPR@FPR $R$ 6 is interpreted as a substantial fraction of the training population being identifiable given $R$ 7 false positives. The system is therefore intended not merely for post hoc reporting, but for model selection, privacy tuning, and validation of privacy claims in deployment pipelines (Ward et al., 22 Sep 2025).

Synth-MIA belongs to a rapidly expanding line of work on adversary-aware auditing of synthetic-data generators. A closely related example is "MIA-EPT: Membership Inference Attack via Error Prediction for Tabular Data" (German et al., 16 Sep 2025), a black-box attack targeting tabular diffusion models through error-based feature vectors obtained by masking and reconstructing attributes of target records. MIA-EPT reports AUC-ROC up to $R$ 8 and TPR@10\% FPR values of $R$ 9, and it achieved second place in the MIDST 2025 Black-box Multi-Table track (German et al., 16 Sep 2025). Relative to that attack-specific line, Synth-MIA is a testbed and threat framework rather than a single attack.

Outside tabular synthesis, the broader MIA literature has moved toward architecture-aware attacks and defenses. "MIA-Tuner: Adapting LLMs as Pre-training Text Detector" (Fu et al., 2024) turns aligned and unaligned LLMs into pre-training data detectors through instruction-based tuning, while "Mitigating Membership Inference in Intermediate Representations via Layer-wise MIA-risk-aware DP-SGD" (Meng et al., 26 Feb 2026) allocates privacy protection across layers according to IR-level MIA risk. These works are not tabular-synthesis frameworks, but they reinforce the same general conclusion: privacy leakage is highly mechanism-specific, and effective auditing or mitigation requires attack models matched to the exposed interface (Fu et al., 2024, Meng et al., 26 Feb 2026).

The label "Synth-MIA" is also contextually adjacent to several synthesis-oriented systems in other domains, including a variational parametric audio synthesizer built around pitch-conditioned spectral-envelope modeling (Subramani et al., 2020), a score-informed resynthesis framework based on Music Interpretation Analysis (Simonetta, 2022), medical image synthesis frameworks such as DDMM-Synth for MRI-guided CT generation (Li et al., 2023) and GaNDLF-Synth for biomedical imaging workflows (Pati et al., 2024), and the physics-based T-SYNTH breast-imaging dataset (Wiedeman et al., 5 Jul 2025). Those systems concern synthesis itself rather than tabular privacy auditing. The explicit software framework named Synth-MIA, however, denotes the tabular MIA testbed centered on ensemble privacy auditing and maximum empirical leakage estimation (Ward et al., 22 Sep 2025).