---
title: 'Synth-MIA: Privacy Auditing for Synthetic Data'
url: https://www.emergentmind.com/topics/synth-mia
type: topic
---

# Synth-MIA: Privacy Auditing for Synthetic Data

Synth-MIA is a framework and Python library for auditing privacy leakage in tabular synthetic data via membership inference attacks (MIAs) under model-agnostic threat models. Introduced in "Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis" [2509.18014], it couples a formal auditing procedure with a Scikit-Learn-like testbed implementing 13 attack methods, and it was used in what the paper describes as the largest tabular synthesis privacy benchmark to date. Its central premise is adversary-aware evaluation: higher synthetic data quality can coincide with greater privacy leakage, similarity-based privacy metrics can correlate only weakly with MIA results, and even differentially private generators such as PATEGAN can fail to preserve privacy under attack [2509.18014].

## 1. Conceptual basis and motivation

Tabular generative models are increasingly used for privacy-preserving data release, data augmentation, and imputation. Synth-MIA is motivated by the observation that such models can memorize parts of the training data or overfit the training distribution, so synthetic records may become close to, or effectively identical to, training records. In that setting, privacy risk is not adequately characterized by non-adversarial similarity statistics alone [2509.18014].

The dominant privacy proxy in the tabular synthesis literature is Distance to Closest Record–Proportion (DCR-Prop). For each training record, DCR-Prop compares its nearest-neighbor distance in the synthetic dataset with its nearest-neighbor distance in a reference dataset, and reports the proportion of training records whose nearest neighbor is synthetic rather than reference. Synth-MIA treats this family of metrics as insufficient because they lack an explicit adversary model and do not directly quantify whether an attacker can infer training membership. In the benchmark associated with Synth-MIA, DCR-Prop shows only weak correlation with MIA-based leakage, with \(r \approx 0.225\) and \(p \approx 5.9 \times 10^{-13}\) [2509.18014].

Membership inference attacks recast privacy as an attack-performance problem. Rather than asking whether synthetic samples look close to real ones, they ask whether an adversary can determine whether a target record was part of the generator’s training set. This shifts evaluation from geometric resemblance to operational leakage. The framework therefore measures privacy primarily through attack metrics such as AUC-ROC and TPR at fixed FPR, especially low-FPR operating points [2509.18014].

## 2. Threat model and formal auditing objective

Synth-MIA standardizes two threat models. In the **no-box** setting, the attacker has access only to the synthetic dataset \(S\). In the **no-box-calibrated** setting, the attacker has access to \(S\) and to a reference dataset \(R\) sampled independently from the same population as the training set \(T\), but disjoint from \(T\). The attacker does not know the generator architecture, hyperparameters, or training procedure [2509.18014].

An attack is formalized as a decision rule
\[
\mathcal{A}(x^{\star}) = \mathbb{I}\left[f(x^{\star}) > \gamma\right],
\]
where \(f(x^\star)\) is an attack-specific scoring function and \(\gamma\) is a threshold. Attack performance is then summarized by standard binary metrics. Synth-MIA emphasizes
\[
\text{TPR} = \Pr\big[\Lambda(x) \geq \tau \mid x \in T\big]
\]
and
\[
\text{FPR} = \Pr\big[\Lambda(x) \geq \tau \mid x \in R\big],
\]
with \(\Lambda\) denoting an attack score and \(\tau\) the operating threshold [2509.18014].

The framework is explicitly ensemble-based. Given a collection of attacks \(\mathcal{A} = \{A_1,\dots,A_n\}\), empirical privacy leakage is estimated through the strongest observed attack, in the sense of the supremum of \(\text{TPR}_A(\tau)\) subject to an FPR constraint. This follows an empirical-differential-privacy viewpoint: privacy is not summarized by the performance of a single attack, but by the maximum leakage observed across a structured attack set. The same formulation also motivates an effective empirical epsilon through inequalities of the form
\[
\text{TPR}(\tau) \le e^\varepsilon \cdot \text{FPR}(\tau) + \delta,
\]
which Synth-MIA uses as an interpretive device rather than as a replacement for formal differential privacy [2509.18014].

## 3. Testbed architecture and attack taxonomy

Synth-MIA is organized into three modules: preprocessing, attack execution, and evaluation. The preprocessing module receives training, holdout, synthetic, and reference datasets; performs type casting, scaling, and encoding; and, importantly, fits encoders on synthetic or reference data rather than on training data, so that the attacker is not granted unrealistic access to the private training distribution. The attack module implements MIAs as subclasses of a `BaseAttacker` class with a Scikit-Learn-like API. The evaluation module computes standardized privacy metrics, including AUC-ROC, TPR@FPR, accuracy, precision, recall, advantage, privacy gain, empirical epsilon, and Brier score [2509.18014].

The 13 implemented attacks span multiple leakage mechanisms rather than a single adversarial doctrine.

| Family | Representative methods | Primary signal |
|---|---|---|
| Distance-based / memorization-focused | DCR, DCR-Diff, Local Neighborhood, Monte Carlo | Nearest-neighbor proximity or local occupancy |
| Density-based / overfitting-focused | DOMIAS, Density Estimator, Gen-LRA, classifier-based density ratio | \(p_S(x)\), \(p_R(x)\), or local likelihood ratios |
| Local neighborhood / mixed signal | DPI | Synthetic-to-reference density in a local neighborhood |
| Classifier-based | LOGAN, classifier attack | Discriminator or classifier output |

Several scoring functions illustrate the heterogeneity of the attack surface. The DCR score is
\[
f_{\text{DCR}}(x^\star, S) = - \min_{x \in S} d(x^\star, x),
\]
so closer synthetic neighbors imply stronger membership evidence. DOMIAS uses a density ratio,
\[
f_{\text{DOMIAS}}(x^\star, S, R) = \frac{p_S(x^\star)}{p_R(x^\star)},
\]
which targets distributional overfitting. DPI computes a local synthetic-to-reference ratio,
\[
f_{\text{DPI}}(x^\star, S, R) =
\frac{\sum_{\mathbf{z} \in D(x^\star)} \mathbb{I}(\mathbf{z} \in S)}
{\sum_{\mathbf{z} \in D(x^\star)} \mathbb{I}(\mathbf{z} \in R)},
\]
and is explicitly oriented toward local plagiarism-like behavior. Gen-LRA uses a local likelihood-ratio construction based on density estimation and the effect of adding a candidate record to the reference set [2509.18014].

This diversity is not incidental. The benchmark shows that different attacks exploit different failure modes—direct memorization, local density distortion, or broader distributional overfitting—so a single attack family cannot be assumed to dominate across datasets and model classes [2509.18014].

## 4. Benchmark design and empirical findings

The benchmark underpinning Synth-MIA uses 48 datasets from the OpenML-CC18 curated classification suite after excluding extremely high-dimensional cases. Each dataset is split 80/20 into train and test, synthetic datasets are generated at \(1\times\), \(2\times\), and \(3\times\) the training size, and up to 1000 training samples and 1000 holdout samples are used as evaluation points. The test set is divided equally into holdout and reference subsets [2509.18014].

Nine tabular generators are evaluated: CT-GAN, AdsGAN, PATEGAN, TVAE, N-Flows, Adversarial Random Forest (ARF), Tab-DDPM, TabSyn, and REaLTabFormer. Utility is measured with XGBoost classifier AUC on real holdout data. Fidelity is measured with MMD and JSD. Privacy is measured by running the full attack suite and reporting both per-attack metrics and the maximum leakage across attacks [2509.18014].

The benchmark’s principal empirical finding is that higher synthetic data quality corresponds to greater privacy leakage. Models with stronger utility and fidelity scores also tend to exhibit higher Max-AUC and Max-TPR@FPR. Among the reported model means, REaLTabFormer attains Max-AUC \(= 0.594\) and TPR@FPR\(=0.1\) \(= 0.224\), while TabSyn attains Max-AUC \(= 0.569\) and TPR@FPR\(=0.1\) \(= 0.186\). At stricter operating points, TPR@FPR\(=0\) values around \(0.03\) to \(0.05\) indicate that a non-trivial fraction of training records can be identified with zero false positives [2509.18014].

No single attack is uniformly best. Gen-LRA, DPI, and DCR-Diff are the strongest attacks on average, and Gen-LRA is best in roughly \(25\%\) to \(29\%\) of synthetic datasets depending on whether one ranks by AUC or TPR@FPR\(=0.1\). Yet every implemented attack is the best attack for some generator-dataset combinations. This is the core empirical justification for Synth-MIA’s ensemble design: privacy leakage is routinely underestimated if auditing is restricted to one attack family [2509.18014].

## 5. Similarity metrics, differential privacy, and interpretation of leakage

Synth-MIA’s most direct methodological intervention is its argument against equating privacy with similarity control. DCR-Prop can remain near values conventionally regarded as benign while MIA leakage is materially elevated. TabSyn and REaLTabFormer illustrate this mismatch: they have strong synthetic-data quality and strong MIA leakage even when non-adversarial similarity metrics appear acceptable. This suggests that privacy assessment based on nearest-neighbor similarity alone is structurally incomplete [2509.18014].

The framework also stresses that formal differential privacy claims should still be audited empirically. PATEGAN, a differentially private GAN using teacher aggregation, has lower mean leakage than several non-DP models when aggregated over all datasets, but its worst-case behavior remains substantial. For the ten datasets with highest leakage, PATEGAN at \(\epsilon = 1\) reaches AUC \(= 0.628\), TPR@FPR\(=0\) \(= 0.190\), and TPR@FPR\(=0.1\) \(= 0.298\). A plausible implication is that implementation details, dataset structure, and worst-case instances remain decisive even when a generator is marketed as differentially private [2509.18014].

The framework’s practical reading of attack metrics is operational rather than abstract. In its usage discussion, AUC near \(0.5\) is treated as little leakage; TPR@FPR\(=0\) near \(0\) corresponds to few perfectly detectable records; and TPR@FPR\(=0.1 > 0.2\) is interpreted as a substantial fraction of the training population being identifiable given \(10\%\) false positives. The system is therefore intended not merely for post hoc reporting, but for model selection, privacy tuning, and validation of privacy claims in deployment pipelines [2509.18014].

## 6. Related developments and terminological scope

Synth-MIA belongs to a rapidly expanding line of work on adversary-aware auditing of synthetic-data generators. A closely related example is "MIA-EPT: Membership Inference Attack via Error Prediction for Tabular Data" [2509.13046], a black-box attack targeting tabular diffusion models through error-based feature vectors obtained by masking and reconstructing attributes of target records. MIA-EPT reports AUC-ROC up to \(0.599\) and TPR@10\% FPR values of \(22.0\%\), and it achieved second place in the MIDST 2025 Black-box Multi-Table track [2509.13046]. Relative to that attack-specific line, Synth-MIA is a testbed and threat framework rather than a single attack.

Outside tabular synthesis, the broader MIA literature has moved toward architecture-aware attacks and defenses. "MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector" [2408.08661] turns aligned and unaligned LLMs into pre-training data detectors through instruction-based tuning, while "Mitigating Membership Inference in Intermediate Representations via Layer-wise MIA-risk-aware DP-SGD" [2602.22611] allocates privacy protection across layers according to IR-level MIA risk. These works are not tabular-synthesis frameworks, but they reinforce the same general conclusion: privacy leakage is highly mechanism-specific, and effective auditing or mitigation requires attack models matched to the exposed interface [2408.08661] [2602.22611].

The label "Synth-MIA" is also contextually adjacent to several synthesis-oriented systems in other domains, including a variational parametric audio synthesizer built around pitch-conditioned spectral-envelope modeling [2004.00001], a score-informed resynthesis framework based on Music Interpretation Analysis [2205.00941], medical image synthesis frameworks such as DDMM-Synth for MRI-guided CT generation [2303.15770] and GaNDLF-Synth for biomedical imaging workflows [2410.00173], and the physics-based T-SYNTH breast-imaging dataset [2507.04038]. Those systems concern synthesis itself rather than tabular privacy auditing. The explicit software framework named Synth-MIA, however, denotes the tabular MIA testbed centered on ensemble privacy auditing and maximum empirical leakage estimation [2509.18014].

Source: https://www.emergentmind.com/topics/synth-mia