SynthEval Framework Overview

• SynthEval is a modular framework that quantitatively assesses synthetic data across fidelity, utility, and privacy dimensions.
• It employs methods such as TSTR for utility, statistical tests for fidelity, and membership inference to evaluate privacy risks.
• The system offers extensibility through JSON-driven configurations and plug-in metrics to integrate new evaluation criteria.

SynthEval Framework

The SynthEval framework refers to a suite of open-source methodologies and software tools for comprehensive, quantitative evaluation of synthetic data generated by machine learning models, with specific focus on LLMs and tabular data generative models. SynthEval variants have been designed to assess synthetic data across fidelity, utility, and privacy axes, with implementations supporting statistical and downstream ML benchmarking, compatibility with mixed-type tabular data, and principled risk assessment for privacy leakage. This article provides an integrated perspective synthesizing core methods and experimental results from leading SynthEval frameworks, particularly as elaborated in "A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by LLMs" (Yuan et al., 2024), "SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data" (Lautrup et al., 2024), and related works.

1. Architectural Principles and Workflow

SynthEval is organized as a modular, end-to-end pipeline for structured synthetic data evaluation. Its architecture features:

• Data Ingestion & Preprocessing: Real and synthetic datasets (comprising both categorical/numerical tabular attributes and free-form text, e.g., product reviews) are loaded and harmonized. Categorical fields are typically label-encoded, with automatic type detection and normalization. Both train/test splits for utility analysis and participant labeling for privacy metrics are generated here (Yuan et al., 2024, Lautrup et al., 2024).
• Multi-faceted Evaluation Engine: Three main evaluation modules are invoked sequentially or in combination:
  1. Fidelity assessment: Measures how well synthetic data matches the real data in statistical and structural attributes.
  2. Utility assessment: Quantifies downstream task performance, routinely using a Train-Synthetic-Test-Real (TSTR) protocol for classifiers.
  3. Privacy risk assessment: Empirically estimates the risk of privacy leakage under scenarios such as membership inference attacks.

• Reporting & Visualization: Outputs are consolidated as dashboards or tables, with optional visualizations (e.g., distribution overlays, ROC curves) to facilitate interpretability and comparative analysis across models (Lautrup et al., 2024).

• Extensibility: The core framework exposes a modular metrics API and configuration system (JSON- or YAML-driven) that enables custom evaluation bundles and rapid integration of new metrics or privacy tests.

2. Evaluation Metrics and Mathematical Formulations

2.1 Fidelity Metrics

Fidelity measures the congruence between synthetic and reference data distributions and structures:

• Structure Preserving Score (SPS):

$\mathrm{SPS} = \frac{|r \cap s|}{|r \cup s|}$

where $r$ and $s$ are sets of column names in real and synthetic tables (Yuan et al., 2024).

• Integrity Score (IS):

$\mathrm{IS} = \frac{|\{x \in s: x \in C_r\}|}{|s|}$

with $C_r$ the set of real category values or continuous range bounds.

• Column Shape Similarity:
  • Kolmogorov–Smirnov for continuous columns:

  $D_{KS} = \sup_x \left| F_r(x) - F_s(x) \right|$ - Total Variation Distance (TVD) for categoricals:

  $$\mathrm{TVD}(P, Q) = \frac{1}{2} \sum_x |P(x) - Q(x)|$$ - The average of $(1-D_{KS})$ or $(1-\mathrm{TVD})$ is used as a summary shape score (Yuan et al., 2024, Lautrup et al., 2024).

• Textual Review Fidelity: Comparison of sentiment distributions, keyword overlap, and average token length between synthetic and real free-text reviews (Yuan et al., 2024).

2.2 Utility Metrics

• Classification Utility (TSTR): Downstream task accuracy and mean absolute error (MAE):

$$\mathrm{Accuracy} = \frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} + \mathrm{TN} + \mathrm{FP} + \mathrm{FN}} \qquad \mathrm{MAE} = \frac{1}{N} \sum_i |\hat{y}_i - y_i|$$

Here, a logistic regression classifier is trained on synthetic data and tested on real data (Yuan et al., 2024).

• Comprehensive Utility Metrics: Extended variants of SynthEval support dimension-wise mean difference, overlap of confidence intervals, differences in correlation or mutual information matrices, and cross-classifier consistency metrics, including differences in AUROC and micro-F1 between real and synthetic (Lautrup et al., 2024).

2.3 Privacy Metrics

• Membership Inference Attack (MIA) Rate: A random forest classifier is trained to predict whether samples belong to the original real training set or arose from a synthetic set. The MIA success rate (recall) is:

$r$0

Higher recall signals increased privacy leakage risk (Yuan et al., 2024, Lautrup et al., 2024).

• ε-Identifiability, Attribute Disclosure: Additional privacy metrics in the extended framework include:

  • $r$1-risk: Proportion of real samples closest (by Gower’s distance) to a synthetic point than to any other real point, entropy-weighted to emphasize rare records.
  • Attribute Disclosure Risk (ADR): Macro-F1 of a random forest classifier predicting real record attributes using nearest synthetic neighbors.
• Hit Rate, NN Ratios: Fraction of real records exactly matched or median synthetic-to-real nearest neighbor distance ratio (Lautrup et al., 2024).

3. Unified Evaluation across Mixed-Type and Textual Data

SynthEval's methodological innovation is the seamless treatment of numerical and categorical variables without imposing strict preprocessing regimes (e.g., one-hot encoding). Core practices include:

• Gower’s Distance: For any nearest-neighbor or distance-based metric, Gower’s composite distance is used, handling numerical and categorical types uniformly:

$r$2

with $r$3 appropriately normalized or binarized per variable type (Lautrup et al., 2024).

• "Mixed" Correlation Matrices: Synthesis of statistical dependencies using Pearson (numeric-numeric), Cramer’s V (categorical-categorical), and the correlation ratio $r$4 (numeric-categorical).
• Permutation-Based p-values: Empirical statistical significance is computed via permutations, avoiding reliance on model-based assumptions.
• Text Handling: Textual columns (e.g., product reviews) analyzed through sentiment, n-gram statistics, and entity frequency, augmenting table-centric metrics for broader data modalities (Yuan et al., 2024).

4. Implementation and Extensibility

SynthEval is implemented in Python 3, leveraging common data science libraries:

• pandas, numpy for data manipulation,
• scikit-learn, scipy for statistical testing and machine learning (KS tests, random forests, logistic regression),
• matplotlib/seaborn for visualization,
• NLTK or TextBlob for sentiment and keyword extraction where text is evaluated (Yuan et al., 2024).

A metrics registry auto-discovers per-metric classes, and extensibility is provided by a plug-in mechanism: new metric modules can be integrated by conforming to a prescribed evaluate interface. Configuration supports both preset and custom JSON-based profiles, with code-level exposure of distance metrics, cross-validation folds, and permutation parameters (Lautrup et al., 2024).

The reporting subsystem produces tables and plots for interactive or batch analysis, and benchmarking mode aggregates normalized metric values according to ranking strategies (linear, normal, quantile) for robust model comparison. The framework can be invoked via both importable Python API and command-line interface (Lautrup et al., 2024).

5. Empirical Results and Use Cases

In the reference implementation, SynthEval was applied to synthetic product review data generated by ChatGPT-3.5, Claude 3 Opus, and Llama 2 13B, using Amazon reviews as ground truth. Distinct outcomes include:

• Fidelity: All models preserved schema structure (SPS=100%), but varied in data integrity (Claude 98.4%, ChatGPT 93.9%, Llama 87.6%) and column shape similarity (Claude/ChatGPT ≈ 81%, Llama 62.3%).
• Textual metrics: Sentiment and keyword distribution aligned more closely for Claude and ChatGPT than Llama; synthetic reviews were typically much shorter (e.g., 16.6 words vs. 59.3 for real).
• Utility: Classifier accuracy for sentiment prediction on TSTR varied minimally from real (Claude 67.68%, ChatGPT 67.35%, real 67.92%), while Llama lagged (62.26%).
• Privacy: Quantitative MIA results indicated measurable privacy risk; higher MIA success rates signal less privacy preservation (see original tables for exact values) (Yuan et al., 2024).

The framework’s extensibility to public health data was also demonstrated. In a Hepatitis C case study, SynthEval enabled model selection (e.g., privacy-boosted Bayesian network with ε-risk < 0.09) under utility-privacy trade-off constraints (Lautrup et al., 2024).

6. Comparison to Related Evaluation Frameworks and Limitations

SynthEval distinguishes itself from alternative toolkits by combining:

• Modular, plug-in metrics registry supporting tabular and textual data.
• Uniform handling of mixed data, leveraging Gower’s distance and empirical permutation schemes.
• Explicit privacy risk quantification beyond simple row overlap, incorporating MIA, ε-risk, and ADR metrics.
• Preset and customizable benchmark configurations enabling reproducibility and consistency across studies (Lautrup et al., 2024, Yuan et al., 2024).

Limitations include potential blind spots in domain-specific logical consistency or semantic coherence not fully captured by the implemented metrics, and the accuracy of privacy risk metrics in high-dimensional, complex real-world samples. The effectiveness of the utility metrics is dependent on the choice and adequacy of the downstream tasks selected.

7. Practical Guidance for Researchers

For practitioners adopting SynthEval:

• Ensure categorical variable coding adheres to consistent label schemas across real and synthetic data.
• Select or define utility tasks that reflect intended deployment scenarios; TSTR is effective for diagnostic classification tasks, but supervised tasks may require extensions.
• Interpret privacy metrics in the context of deployment risk: a high MIA recall does not imply absolute privacy violation, but signals increased risk surface.
• Modular design supports addition of domain-adapted metrics, enabling fits to specialized regulatory or compliance regimes.

SynthEval thus provides an extensible, reproducible, and principled platform for the quantitative assessment of synthetic data in machine learning pipelines centered on LLMs and tabular generation (Yuan et al., 2024, Lautrup et al., 2024).