---
title: Synthetic Dataset Quality Metric (SDQM)
url: https://www.emergentmind.com/topics/synthetic-dataset-quality-metric-sdqm
type: topic
---

# Synthetic Dataset Quality Metric (SDQM)

Searching arXiv for the cited SDQM/SRA papers and adjacent evaluation frameworks to ground the encyclopedia entry.
to=arxiv_search.search  天天中彩票被json
{"query":"all:\"Synthetic Ranking Agreement\" OR id:1806.11345 OR ti:\"Measuring the quality of Synthetic data for use in competitions\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}ดลองใช้ฟรี
Also checking broader synthetic-data quality evaluation frameworks that contextualize SDQM/SRA.
to=arxiv_search.search  天天中彩票足球json
{"query":"id:2004.07740 OR ti:\"Really Useful Synthetic Data\" OR ti:\"quality assessment framework for synthetic tabular data\" OR id:2401.13716","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}Japgolly
Retrieving additional evaluation-framework papers on holdout-based and structured synthetic data assessment for comparison.
to=arxiv_search.search  天天中彩票官网်json
{"query":"id:2104.00635 OR id:2403.10424 OR ti:\"Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data\" OR ti:\"Structured Evaluation of Synthetic Tabular Data\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} pc蛋蛋
Synthetic Dataset Quality Metric (SDQM) denotes a quantitative criterion, or family of criteria, for assessing whether a synthetic dataset preserves the properties required by an intended use-case. The literature does not define a single universal SDQM. In competition-oriented work, the concept maps directly to Synthetic Ranking Agreement (SRA), which measures whether the pairwise ordering of algorithms on synthetic data matches their ordering on original data [1806.11345]. Subsequent work broadens the notion to multidimensional evaluation over training-sample similarity, population generalization, inference validity, prediction performance, privacy, fairness, carbon footprint, reporting quality, and task-specific transferability, which indicates that SDQM is best understood as a use-conditioned evaluation framework rather than a fixed scalar invariant across domains [2004.07740].

## 1. Conceptual scope and major formulations

Across the arXiv literature, SDQM appears in at least six distinct senses. Some papers define a single metric tailored to a narrow task; others define a structured dashboard or weighted aggregate over multiple dimensions. This divergence is explicit in work that states there is no consensus or universally accepted standard framework for synthetic-data evaluation [2604.15961].

| Formulation | Core criterion | Typical output |
|---|---|---|
| Competition fidelity | Pairwise algorithm ordering is preserved | SRA score |
| Four-quadrant utility framework | Training vs population; general vs specific utility | Multi-metric report or weighted SDQM |
| Holdout-based fidelity/privacy | Synthetic should resemble training no more than holdout does | Fidelity/privacy vector |
| Structured tabular objective | $Q=P$ and $S \neq X$ | Decomposed structural scores |
| Healthcare governance framework | Similarity, Usability, Privacy, Fairness, Carbon footprint | Weighted dimension score |
| Medical scorecard framework | Correctness, Coverage, Constraint, Completeness, Compliance, Comprehension, Consistency | Optional overall rating |

The competition formulation is the most literal antecedent of SDQM as a ranking-preservation metric. The general utility formulation in "Really Useful Synthetic Data" divides evaluation by reference target—training data versus underlying population—and by utility type—general similarity versus specific inference or prediction—yielding four quadrants of assessment [2004.07740]. "Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data" instead anchors quality to a holdout split and treats privacy as a co-equal axis, while "Structured Evaluation of Synthetic Tabular Data" proposes a single mathematical objective, namely that synthetic data should be drawn from the same distribution as the observed data, formalized as $Q=P$ with the additional condition $S \neq X$ [2104.00635] [2403.10424].

In regulated settings, the concept is expanded further. "Can I trust my fake data" defines a MECE framework centered on Similarity, Usability, Privacy, Fairness, and Carbon footprint, with Representativity as a superset covering Similarity and Usability [2401.13716]. "Scorecards for Synthetic Medical Data Evaluation and Reporting" defines the "Seven Cs"—Correctness, Coverage, Constraint, Completeness, Compliance, Comprehension, and Consistency—and allows, but does not prescribe, an overall score [2406.11143]. This suggests that SDQM is not a single settled object in the literature, but a problem-specific operationalization of synthetic-data quality.

## 2. Ranking-preserving SDQM: Synthetic Ranking Agreement

The canonical competition-oriented SDQM is SRA. Let $\mathcal{D}=\{\mathbf{u}_i\}_{i=1}^n$ be a dataset of $n$ i.i.d. samples, let $\mathcal{T}$ be a task, let $\mathcal{D}_1,\mathcal{D}_2$ denote train and test splits of the real data, let $\mathcal{A}_1,\ldots,\mathcal{A}_k$ be machine-learning methods, let $m_\mathcal{T}$ be the task metric, and let $G$ generate a synthetic dataset $\mathcal{D}^G$ with synthetic splits $\mathcal{D}_1^G,\mathcal{D}_2^G$. Define real and synthetic performance scores
$$
R_i = m_\mathcal{T}(\mathcal{A}_i(\mathcal{D}_1), \mathcal{D}_2), \qquad
S_i = m_\mathcal{T}(\mathcal{A}_i(\mathcal{D}_1^G), \mathcal{D}_2^G).
$$
The target property is that if one algorithm beats another on synthetic data, the same pairwise ordering should hold on real data. SRA measures the fraction of ordered algorithm pairs for which the sign of the performance difference is concordant:
$$
SRA(G) = \frac{1}{k(k-1)} \sum_{i=1}^k \sum_{j \neq i} \mathbb{I}\Big((R_i-R_j)(S_i-S_j) > 0\Big).
$$
Under the notation $s_i^R:=R_i$ and $s_i^S:=S_i$, the same quantity is an SDQM in the sense
$$
SDQM_{\text{pair}}(G) \equiv SRA(G) = \frac{1}{m(m-1)} \sum_{i=1}^m \sum_{j \neq i} \mathbb{I}\Big((s_i^R-s_j^R)(s_i^S-s_j^S) > 0\Big).
$$
Ties are not counted as correct, because a zero difference yields indicator value $0$. The paper contrasts SRA with TSTR,
$$
TSTR(G)=\frac{1}{k}\sum_{i=1}^k m_\mathcal{T}(\mathcal{A}_i(\mathcal{D}_1^G), \mathcal{D}_2),
$$
which measures average predictive performance on real test data after training on synthetic data, but does not measure rank preservation [1806.11345].

The experimental illustration uses the MAGGIC heart-failure dataset, binary prediction, AUROC, and 12 algorithms: Logistic Regression, Random Forests, Gaussian Naive Bayes, Bernoulli Naive Bayes, Linear SVM, Decision Tree, LDA, AdaBoost, Bagging, GBM, Multi-layer Perceptron, and XGBoost. Rather than benchmark a specific generator, the paper injects label noise by flipping binary labels with probability $p \in [0,0.5)$, thereby constructing a privacy-utility trade-off example. TSTR decreases as $p$ increases, but SRA is not strictly decreasing; the paper explicitly notes an increase from $p=0.10$ to $p=0.15$. The interpretation is narrow but important: if the objective is algorithm selection in a competition, relative ordering may remain useful even when absolute predictive performance degrades. The paper also notes three limitations that remain central in later SDQM work: SRA depends on the chosen algorithm set, multiple variants of the same method family can bias the score, and no explicit statistical inference, confidence intervals, or tie handling beyond the indicator definition are provided [1806.11345].

## 3. Reference targets, utility types, and holdout-based assessment

A broader SDQM literature departs from competition ranking and asks what synthetic data should approximate: the observed training sample, or the underlying population. "Really Useful Synthetic Data" formalizes this with two orthogonal dimensions. The first is the reference target: training data similarity versus generalization similarity to a population. The second is utility type: general distributional similarity versus specific task performance. This yields four quadrants—training/general, training/specific, population/general, and population/specific—and the paper specifies metrics for each. General similarity is measured with the propensity score mean squared error ratio and with a 1-way Wasserstein distance ratio under a randomization test. Specific inference and prediction are measured with percent bias, variance ratio, 90% confidence interval coverage, interval width, and RMSE. For multiple synthetic datasets, the paper recommends Rubin’s rules,
$$
\bar{\theta}=\frac{1}{m}\sum_{i=1}^m \hat{\theta}_i^*, \quad
\bar{\omega}=\frac{1}{m}\sum_{i=1}^m V(\hat{\theta}_i^*), \quad
b=\frac{1}{m-1}\sum_{i=1}^m(\hat{\theta}_i^*-\bar{\theta})^2, \quad
u=\bar{\omega}+\left(1+\frac{1}{m}\right)b,
$$
and, under uncongeniality, the conservative variance inflation $u_{uc}=2u$. The paper does not prescribe a single scalar SDQM, but explicitly states that one can be constructed by normalizing and weighting quadrant-specific metrics [2004.07740].

A related but stricter holdout-based perspective appears in "Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data". There the real data are split into $D_{\text{train}}$ and $D_{\text{holdout}}$, the synthesizer is trained only on $D_{\text{train}}$, and fidelity is computed by total variation distance on discretized lower-dimensional marginals. For each $k$-way subset of columns, one averages
$$
TVD(P,Q)=\frac{1}{2}\sum_{\omega} |P(\omega)-Q(\omega)|
$$
between synthetic and training empirical marginals, then compares this to the corresponding holdout-versus-training baseline. Privacy is assessed by the distance to closest record under Hamming distance after discretization. For each synthetic record, one computes whether it is closer to a training record than to a holdout record, with ties split equally; a share near $0.5$ is interpreted as strong evidence that synthetic records are just as close to training as to holdout. The paper recommends reporting fidelity for $k=1,2,3$ and privacy jointly rather than collapsing them into a single scalar [2104.00635].

"Structured Evaluation of Synthetic Tabular Data" goes further by arguing that the single objective is $Q=P$ together with $S \neq X$. It then decomposes this objective structurally into marginals, pairwise distributions, leave-one-out conditionals, and the full joint. A key statement is that $Q=P$ if and only if
$$
Q(c_j \mid \{c_{-j}\}) = P(c_j \mid \{c_{-j}\})
$$
for all columns $j$, which makes leave-one-out conditionals a complete characterization of the joint. The paper uses this decomposition to classify existing metrics as necessary or incomplete surrogates, to motivate model-free baselines such as self, perm, and half, and to propose a spectrum of new metrics including MutualInformationSimilarity and PCC-based scores [2403.10424].

## 4. Task-specific SDQM formulations

Recent work increasingly defines SDQM as task-aware rather than task-agnostic. In object detection, "SDQM: Synthetic Data Quality Metric for Object Detection Dataset Evaluation" defines an explicit composite score
$$
SDQM(D;R)=\sum_{k=1}^{K} w_k \,\tilde{q}_k(D,R), \qquad \sum_{k=1}^{K} w_k=1,
$$
or, in implementation, a regressor $f$ mapping normalized sub-metrics to predicted mAP on real data. The sub-metrics span Pixel Intensity Match, Spatial Distribution Difference, Bounding Box Match, Label Overlap, embedding-space $\beta$-Recall and $\alpha$-Precision, Dataset Separability, and a model-family interpretability term called V-Information,
$$
I_\mathcal{V}(X \to Y)=H_\mathcal{V}(Y)-H_\mathcal{V}(Y\mid X).
$$
In experiments on RarePlanes, DIMO, and WASABI with YOLOv11n, the random-forest SDQM correlates strongly with mAP50, with Pearson $r=0.8719$, outperforming $\alpha$-Precision, $\beta$-Recall, Authenticity, and Mauve. Feature importance is highest for Pixel Intensity Match, $\beta$-Recall, and V-Information, while Label Overlap contributes least [2510.06596].

In Earth observation, "Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation" argues that common fidelity metrics rooted in ImageNet-pretrained features are unreliable indicators for geospatial data. The paper reports a stark misalignment: semantics-preserving perturbations such as rotation can inflate FID from $2.09$ to $36.51$ on ARAS-Test, while human recognition accuracy under rotation remains $94.05\%$. It therefore proposes a candidate SDQM that combines downstream performance gain on the target task, human realism or recognizability, invariance robustness to transformations such as rotation and flip, and task-specific distribution mismatch in an EO encoder space. The practical conclusion is that automatic quality evaluation should be grounded in downstream task performance and human evaluation rather than generic visual fidelity metrics alone [2606.25128].

A different task-level design appears in "Synthetic Dataset Evaluation Based on Generalized Cross Validation". Let $O$ be the synthetic dataset, $\{D_i\}_{i=1}^N$ real reference datasets, and $P_{ij}$ the performance of a model trained on dataset $i$ and evaluated on dataset $j$. After row-wise normalization,
$$
R_{ij}=\frac{P_{ij}}{P_{ii}},
$$
the paper defines simulation quality
$$
A_o = \sum_{i=1}^N w_i R_{oi}, \qquad
w_i = \frac{R_{io}}{\sum_{j=1}^N R_{jo}},
$$
and transfer quality
$$
S_o = \sum_{i=1}^N v_i R_{oi}, \qquad
C_{ij}=\frac{R_{ij}}{R_{ij}+R_{ji}}, \qquad
v_i=\frac{2}{N-1}\sum_{j \neq i} C_{ij}.
$$
On Virtual KITTI with KITTI and BDD100K as references, YOLOv5s, and AP50, the paper reports $A_o=0.49$ and $S_o=0.45$ [2509.11273].

Two further variants emphasize no-reference realism and annotation-free selection. "A Style-Based Metric for Quantifying the Synthetic-to-Real Gap in Autonomous Driving Image Datasets" defines Style Embedding Distribution Discrepancy (SEDD) from Gram-matrix style embeddings, with
$$
SEDD1=\|c_{\text{new}}-c_{\text{real}}\|_2
$$
and a Gaussian-kernel MMD as SEDD2; the method ranks VKITTI2 as closer to real than VKITTI and tracks improvements after sim-to-real enhancement [2510.10203]. "SynQuE: Estimating Synthetic Dataset Quality Without Annotations" defines the synthetic dataset quality estimation problem when only limited unlabeled real data are available, using representation-based proxies such as MMD, Proxy-A-Distance, Mean Distance to Medoid, and MAUVE, together with the LLM-based LENS score. On Text2SQL, selecting the top-3 synthetic datasets via SynQuE proxies raises accuracy from $30.4$ to $38.4$ on average relative to indiscriminate selection [2511.03928].

## 5. Regulated-domain frameworks, scorecards, and accountability

In healthcare and other regulated domains, SDQM is often embedded in a governance process rather than treated as a stand-alone metric. "Can I trust my fake data" defines a framework centered on Similarity, Usability, Privacy, Fairness, and Carbon footprint and computational complexity, organized across stages from priorities and context through training-data evaluation, generation-process evaluation, synthetic-data evaluation, analytical validation, clinical validation, and monitoring. The framework explicitly includes utility transfer tests such as TRTR, TSTR, TRTS, and TSTS; privacy tests such as membership inference, nearest-neighbor memorization, k-anonymity, $l$-diversity, $t$-closeness, and differential privacy; fairness metrics such as Demographic Parity and Equalized Odds; and carbon or energy reporting. It also proposes an illustrative composite score
$$
SDQM = \sum_{d=1}^{D} w_d s_d, \qquad \sum_{d=1}^{D} w_d = 1,
$$
with possible thresholding for deployment in clinical contexts [2401.13716].

"Scorecards for Synthetic Medical Data Evaluation and Reporting" reorients the same problem into a standardized SMD Card. Its quantitative core is the Seven Cs: Correctness, Coverage, Constraint, Completeness, Compliance, Comprehension, and Consistency. The paper names modality-specific metrics such as FID, cosine similarity, BLEU, constraint satisfaction scores, differential privacy parameters, and subgroup variability measures, and allows an optional overall quality assessment derived from selected metrics. The emphasis is not solely on fidelity, but on auditability, intended use, limitations, and regulatory transparency [2406.11143].

DAISYnt provides a more operational suite for regulated domains. It evaluates statistical similarity through correlation similarity, Information Value similarity, univariate and multivariate distribution tests, and discriminator-based detectability; utility through AUC gap, prediction-vector cosine similarity, and hidden-representation similarity; and privacy through cloned rows, close rows, distance-based linkability, model-based linkability, and inference risk. The paper does not define a built-in single SDQM, but it explicitly motivates a modular, test-based standard for model selection, auditing, and fine-tuning in finance, healthcare, insurance, and education [2204.06297].

A complementary accountability perspective appears in "DataRubrics for Automated Quality Metrics and Accountability". Here the quality object is the dataset paper and its documentation rather than the distribution alone. DataRubrics evaluates dimensions such as Data Sources, Data Annotators, Data Novelty, Task Utility, Annotation Guidelines, Quality Assurance, Human Language Coverage, Non-Human Language Coverage, and Reproducibility via structured JSON schemas and LLM-as-a-judge protocols [2506.01789]. At a broader survey level, "The LLM Data Auditor" proposes modality-agnostic intrinsic evaluation under two pillars—Quality and Trustworthiness—and defines a composite harmonic coupling between a geometric-mean quality index $Q$ and trustworthiness index $T$,
$$
SDQM = \frac{2}{\frac{1}{Q}+\frac{1}{T}},
$$
thereby making explicit that a dataset can score well only if both pillars are strong [2601.17717].

## 6. Limitations, controversies, and open questions

The main limitation of SDQM as a concept is that it lacks a universal target. Several papers explicitly state that there is no consensus on concise metrics or universally accepted standard frameworks, and some deliberately avoid defining a single score at all [2604.15961] [2509.11273]. Even when a scalar exists, it is often task-conditioned, model-conditioned, or benchmark-conditioned. This is obvious in SRA, which depends on the selected portfolio of algorithms and can be biased by over-representing a single method family, and in object-detection SDQM, whose strongest component is V-Information instantiated with YOLOv11n [1806.11345] [2510.06596].

A second limitation is the tension between fidelity and actual use. The EO evidence shows that FID, KID, IS, LPIPS, and SSIM can react strongly to semantics-preserving perturbations while downstream segmentation and human realism remain stable, which undermines any claim that generic fidelity scores are universal quality surrogates [2606.25128]. More generally, the four-quadrant framework makes explicit that quality is subjective and use-dependent: training-target descriptive replication, population inference, and predictive transfer can favor different datasets and different metric weightings [2004.07740].

A third limitation concerns uncertainty. The original SRA paper reports no bootstrapping, confidence intervals, or formal hypothesis tests. Holdout-based and Monte Carlo frameworks partially address this, but uncertainty handling remains uneven across the literature [1806.11345] [2004.07740]. A plausible implication is that SDQM reporting is most defensible when it includes repeated synthetic draws, bootstrap intervals, or explicit sensitivity analyses over algorithms, discriminators, seeds, and reference sets.

Finally, privacy remains structurally entangled with quality. DAISYnt, the holdout-based framework, and healthcare scorecards all treat privacy as a first-class dimension rather than an afterthought, and several papers emphasize that stronger privacy mechanisms or stronger perturbation can reduce utility while not necessarily destroying the quantities relevant to a specific application [2104.00635] [2204.06297]. The open questions recur across papers: how to choose balanced algorithm portfolios, how to calibrate and weight heterogeneous metrics, how to standardize score reporting across modalities and domains, and how to align automated metrics with human judgment and real downstream performance. The literature therefore converges less on a universal SDQM formula than on a methodological principle: synthetic data quality must be defined against a reference target, a task, and an explicit risk model.

Source: https://www.emergentmind.com/topics/synthetic-dataset-quality-metric-sdqm