Synthetic Utility Gap Explained

• Synthetic Utility Gap is a measure of the difference in statistical properties and predictive performance between synthetic and real data.
• Key metrics such as moment matching, propensity-score MSE, and classification accuracy score are used to evaluate this gap.
• Post-processing methods like exponential tilting and dual stochastic optimization are applied to minimize the gap while preserving privacy.

A synthetic utility gap is the quantitative deficit in statistical or task utility incurred by replacing real data with synthetic data. This concept formalizes the extent to which synthetic data fails to reproduce the desired statistical properties, inferential results, or downstream task performance characteristic of authentic data. The gap may refer to general distributional similarity, specific analytic results, or application-dependent task measures (e.g., downstream classifier accuracy, statistical moment matching, or utility metrics on domain-specific aggregates). There is no universal metric, but the gap is conventionally defined as the difference of a utility measure between the synthetic and the real data, or—when ground-truth is inaccessible—as a divergence from a differentially private statistical summary.

1. Formal Definitions of the Synthetic Utility Gap

Across the literature, the synthetic utility gap is typically defined with respect to a prespecified set of query statistics $Q = \{q_1, \ldots, q_K\}$, model performance, or a general distributional criterion. The gap quantifies the failure of the synthetic empirical law $P_{\text{syn}}$ to match the private real distribution $P_{\text{real}}$ with respect to these summaries. The general form is: $$G(P_{\text{syn}}; Q) = \| q(P_{\text{syn}}) - q(P_{\text{real}}) \|_1 \qquad \text{where} \qquad q(P) = (q_1(P), ..., q_K(P)), \quad q_k(P) = \mathbb{E}_{X \sim P}[q_k(X)]$$ This gap is typically measured in practice via a noisy (differentially private) estimate $a$ of $q(P_{\text{real}})$: $$\text{Gap}_{\text{syn}} = \| q(P_{\text{syn}}) - a \|_1, \quad \text{Gap}_{\text{post}} = \| q(P_{\text{post}}) - a \|_1$$ where $P_{\text{post}}$ denotes a post-processed (tilted/reweighted) synthetic distribution.

In predictive modeling, the gap may be formulated as a difference in performance metrics: $$\Delta_{\text{utility}} = U_{\text{real}} - U_{\text{syn}}$$ for utility $U$ defined as, e.g., F1 score on a test set, mean squared error, or classification accuracy.

For information-theoretic distances, density ratio or $f$-divergence frameworks give: $$D_f(P_{\text{real}} \| P_{\text{syn}}) = \int f(r(x))\,P_{\text{syn}}(x)\,dx, \quad r(x) = \frac{P_{\text{real}}(x)}{P_{\text{syn}}(x)}$$ and directly interpret the gap as divergence from the real distribution in hypothesis testing or density estimation senses.

2. Representative Metrics and Benchmarks

A broad range of utility gap metrics are established and widely adopted:

• Moment/Query Matching ($\|\cdot\|_1$ or $\|\cdot\|_2$ gap): Applied to feature means, correlations, or other task-relevant low-order moments.
• Propensity-score mean-squared error (pMSE): For global indistinguishability, the pMSE between real and synthetic is scaled by its exact finite-sample null expectation, with $$S_{\text{pMSE}} = pMSE_{\text{obs}} / \mathbb{E}_{\text{null}}[pMSE]$$; values near 1 indicate synthetic data "looks real" under the test (Raab et al., 2021).
• Classification Accuracy Score (CAS): In image generation or tabular prediction, CAS is the accuracy of a model trained on synthetic data and evaluated on real test cases; $$\Delta = \text{Acc}_{\text{real}} - \text{Acc}_{\text{syn}}$$ (Lampis et al., 2023).
• Density-ratio divergence: $D_{\text{PE}}$ (Pearson), $D_{\text{KL}}$ (KL); nonparametric density ratio estimators (e.g., uLSIF) are used for sensitivite, dimension-agnostic scoring (Volker et al., 23 Aug 2024).
• Interval-overlap (CIO): For statistical inferences, the confidence-interval overlap or standardized difference in parameter estimates between real and synthetic analyses quantifies the specific utility gap (Snoke et al., 2016, Little et al., 2022).
• Task-specific errors (e.g., maximum RMSE on term-deposit yield curves, Frobenius norm of matrix-valued summaries, etc.): Domain-specific as in financial usage indices or transition matrices (Caceres et al., 29 Oct 2024).
• Composite or multi-dimensional metrics: Aggregations via PCA (upca score) or by principal-component regression of multiple (possibly conflicting) utility measures (Dankar et al., 2022).

An illustrative table of utility gap metrics and settings:

Metric	Mathematical Formulation	Context/Interpretation
$\ell_1$ query gap	$\|q(P_{\text{syn}}) - q(P_{\text{real}})\|_1$	General summary-statistics matching
pMSE (propensity)	$\frac{1}{N}\sum_i(\hat e_i - c)^2$	Global distributional similarity
$\Delta$CAS	$\text{Acc}_{\text{real}} - \text{Acc}_{\text{syn}}$	Downstream predictive performance
$f$-divergence	$D_f(P_{\text{real}}\|P_{\text{syn}})$	Distributional distance via density ratios (Volker et al., 23 Aug 2024)
CI-overlap (CIO)	Pct. overlap of 95% CI intervals	Inferential/parameter-preservation

3. Statistical and Algorithmic Approaches to Gap Minimization

A central focus of recent work is not only measuring but directly minimizing the synthetic utility gap—either at data generation, via model tuning, or by post-processing:

• Post-processing via KL-projection: The optimal synthetic distribution (over the empirical support of $P_{\text{syn}}$) minimizing $D_{\text{KL}}(P \| P_{\text{syn}})$ subject to matching queries $a_k$ up to tolerance $\gamma$ (Wang et al., 2023) is achieved via exponential tilting:

$$P^*_{\text{post}}(x_i) \propto p_{\text{syn}}(x_i) \exp\left(-\sum_{k=1}^{K} \lambda_k^* [q_k(x_i)-a_k]\right)$$

Dual variables $\lambda^*$ are solved by minimizing

$$\min_{\lambda \in \mathbb{R}^K} F(\lambda) + \gamma \|\lambda\|_1, \qquad F(\lambda) = \log \left(\frac{1}{n} \sum_{i=1}^n \exp(-\sum_k \lambda_k [q_k(x_i)-a_k])\right)$$

using a stochastic first-order (proximal-gradient) method, with batch-based compositionally biased gradient steps ensuring computational scalability.

• Diagnostic and iterative improvement: Interactive visualization and heatmap-based diagnostics (e.g., in synthpop (Raab et al., 2021)) allow identification and targeted correction of marginal or interaction gaps in synthetic data, leading to iterative model refinement that "closes" the utility gap for low-order marginals.
• Downstream-fidelity-oriented pipelines: Techniques such as Dynamic Sample Filtering, Dataset Recycle, and Expansion Trick (as orchestrated in GaFi (Lampis et al., 2023)) systematically filter, resample, and diversify synthetic samples post-generation to empirically minimize downstream utility gaps (in classifier accuracy), achieving error reductions of 2–4% on standard benchmarks.
• Domain-specific gap reduction: In financial and microdata applications, utility gaps are minimized by favoring marginal-based tabular synthesis (AIM, MST) with optimized binning, as this ensures sharp improvement in domain-aggregate fidelity under privacy constraints (Caceres et al., 29 Oct 2024).

4. Empirical Behavior and Benchmark Results

Empirical studies demonstrate that post-processing or design choices can reduce the synthetic utility gap by factors of 2–10, often to the noise level of the differentially private queries themselves:

• Post-hoc tilting: On UCI and Home–Credit tabular datasets (Wang et al., 2023), post-processing cut average $\ell_1$-correlation gap and marginal JS divergence by 2–10$\times$, sometimes to (numerically) zero, while achieving logistic-regression F1-scores close to (or exceeding) unprocessed synthetic data; on large-scale datasets (over $10^5$ rows, $10^2$ columns), GPU-based dual-solve with resampling took under 4 minutes per pass.
• Deep image generative models: GaFi (Lampis et al., 2023) closed the accuracy gap from 7.9% to 1.78% (CIFAR-10), with similar results on other datasets. Each modular technique (filtering, recycle, expansion) contributed independently, with additive improvements. On multi-generator pipelines, the final error matched or outperformed contemporary state-of-the-art synthetic-to-real cross-domain baselines.
• Limitations and failure modes: Any approach limited to reweighting (as opposed to augmenting) synthetic data cannot recover support lost in the initial generative phase; rare but critical configurations absent from $P_{\text{syn}}$ remain unrecoverable (Wang et al., 2023). Furthermore, only those query statistics expressible as bounded-sensitivity functions can be effectively corrected; combinatorial statistics or complex pipelines may resist low-distortion gap minimization.
• Trade-off with privacy: All DP-conforming post-processing must treat access to real data as a one-time, noise-adding operation (either for synthetic data generation or separate DP query on $Q$), after which only deterministic transformations are applied (post-processing immunity) (Wang et al., 2023). The overall privacy guarantee compounds additively.

5. Theoretical and Practical Significance

Synthetic utility gaps have both theoretical and operational implications:

• Guarantees and limits: Any minimization of the gap is bounded by what is represented (with sufficient frequency or diversity) in the underlying $P_{\text{syn}}$. Without comprehensive support, no post-processing can induce missing structure (Wang et al., 2023).
• Robustness: Empirical evidence and stochastic dual optimization results confirm that the convex-constrained tilting or reweighting approach converges reliably to the utility-optimal synthetic law, given small batch stochasticity and standard learning rate schedules, extending to high-dimensional $Q$ via parallelization.
• Privacy integrity: By requiring no further accesses to real data (except for the DP summaries), the approach explicitly fits within the differential privacy framework through post-processing invariance, making all utility improvements "free" from a privacy cost perspective (beyond the required DP budget for the post-processed queries).
• Scalability: The convex program solved (either via exponential tilt or dual stochastic gradient) retains quadratic-per-batch cost in data size and query dimension, and is practical on modern hardware for datasets with hundreds of thousands of synthetic records and moderate ($<100$) $K$ queries per user-defined utility target.

6. Limitations and Future Directions

Key limitations include:

• Expressivity of $Q$: Only summary statistics with well-defined bounded sensitivity are feasible for post-processing correction. Highly nonlinear, structured, or combinatorial utilities may not admit small-gap solutions within practical distortion bounds.
• Support limitations: Post-processing cannot "invent" synthetic points absent from the initial empirical support; thus, the reweighted synthetic distribution remains confined to the convex hull of $P_{\text{syn}}$.
• Balance of constraints: Introducing new penalties (e.g., additional moment-matching) or further restricting the KL ball tightens control but raises computational and statistical challenges, particularly in high-dimensional, highly-interactive data regimes.
• Extension to non-tabular/relational data: Application to time-series, graphs, and multi-table/relational settings demands alternate DP metrics (event-level or user-level privacy) and richer constrained-projection or generative processes, a topic of ongoing theoretical development.
• Downstream complexity: For highly nonlinear or non-convex downstream tasks, query-based gap minimization may not suffice to close utility gaps without altering the generative model architecture itself (i.e., pure post-processing may be inadequate).

By systematically applying formal post-processing, diagnostics-aware synthesis, or utility-driven pipeline search, synthetic data producers can sharply close the synthetic utility gap for a wide range of analytic tasks and statistical summaries, while respecting strong privacy budgets and practical deployment constraints. Maturation of these methods directly advances the real-world fitness and trustworthiness of synthetic data releases in privacy-sensitive and utility-demanding domains.