---
title: SynthEval Framework Overview
url: https://www.emergentmind.com/topics/syntheval-framework
type: topic
---

# SynthEval Framework Overview

SynthEval Framework

The SynthEval framework refers to a suite of open-source methodologies and software tools for comprehensive, quantitative evaluation of synthetic data generated by machine learning models, with specific focus on large language models (LLMs) and tabular data generative models. SynthEval variants have been designed to assess synthetic data across fidelity, utility, and privacy axes, with implementations supporting statistical and downstream ML benchmarking, compatibility with mixed-type tabular data, and principled risk assessment for privacy leakage. This article provides an integrated perspective synthesizing core methods and experimental results from leading SynthEval frameworks, particularly as elaborated in "A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models" [2404.14445], "SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data" [2404.15821], and related works.

## 1. Architectural Principles and Workflow

SynthEval is organized as a modular, end-to-end pipeline for structured synthetic data evaluation. Its architecture features:

- **Data Ingestion & Preprocessing:** Real and synthetic datasets (comprising both categorical/numerical tabular attributes and free-form text, e.g., product reviews) are loaded and harmonized. Categorical fields are typically label-encoded, with automatic type detection and normalization. Both train/test splits for utility analysis and participant labeling for privacy metrics are generated here [2404.14445], [2404.15821].

- **Multi-faceted Evaluation Engine:** Three main evaluation modules are invoked sequentially or in combination:
  1. **Fidelity assessment:** Measures how well synthetic data matches the real data in statistical and structural attributes.
  2. **Utility assessment:** Quantifies downstream task performance, routinely using a Train-Synthetic-Test-Real (TSTR) protocol for classifiers.
  3. **Privacy risk assessment:** Empirically estimates the risk of privacy leakage under scenarios such as membership inference attacks.

- **Reporting & Visualization:** Outputs are consolidated as dashboards or tables, with optional visualizations (e.g., distribution overlays, ROC curves) to facilitate interpretability and comparative analysis across models [2404.15821].

- **Extensibility:** The core framework exposes a modular metrics API and configuration system (JSON- or YAML-driven) that enables custom evaluation bundles and rapid integration of new metrics or privacy tests.

## 2. Evaluation Metrics and Mathematical Formulations

### 2.1 Fidelity Metrics

Fidelity measures the congruence between synthetic and reference data distributions and structures:

- **Structure Preserving Score (SPS):**
  $$
  \mathrm{SPS} = \frac{|r \cap s|}{|r \cup s|}
  $$
  where $r$ and $s$ are sets of column names in real and synthetic tables [2404.14445].

- **Integrity Score (IS):**
  $$
  \mathrm{IS} = \frac{|\{x \in s: x \in C_r\}|}{|s|}
  $$
  with $C_r$ the set of real category values or continuous range bounds.

- **Column Shape Similarity:**
  - Kolmogorov–Smirnov for continuous columns:
    $$
    D_{KS} = \sup_x \left| F_r(x) - F_s(x) \right|
    $$
  - Total Variation Distance (TVD) for categoricals:
    $$
    \mathrm{TVD}(P, Q) = \frac{1}{2} \sum_x |P(x) - Q(x)|
    $$
  - The average of $(1-D_{KS})$ or $(1-\mathrm{TVD})$ is used as a summary shape score [2404.14445], [2404.15821].

- **Textual Review Fidelity:** Comparison of sentiment distributions, keyword overlap, and average token length between synthetic and real free-text reviews [2404.14445].

### 2.2 Utility Metrics

- **Classification Utility (TSTR):** Downstream task accuracy and mean absolute error (MAE):
  $$
  \mathrm{Accuracy} = \frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} + \mathrm{TN} + \mathrm{FP} + \mathrm{FN}} \qquad \mathrm{MAE} = \frac{1}{N} \sum_i |\hat{y}_i - y_i|
  $$
  Here, a logistic regression classifier is trained on synthetic data and tested on real data [2404.14445].

- **Comprehensive Utility Metrics:** Extended variants of SynthEval support dimension-wise mean difference, overlap of confidence intervals, differences in correlation or mutual information matrices, and cross-classifier consistency metrics, including differences in AUROC and micro-F1 between real and synthetic [2404.15821].

### 2.3 Privacy Metrics

- **Membership Inference Attack (MIA) Rate:** A random forest classifier is trained to predict whether samples belong to the original real training set or arose from a synthetic set. The MIA success rate (recall) is:
  $$
  \mathrm{MIA}_{\mathrm{recall}} = \frac{\#\{\text{true train samples recovered}\}}{\#\{\text{train samples tested}\}}
  $$
  Higher recall signals increased privacy leakage risk [2404.14445], [2404.15821].

- **ε-Identifiability, Attribute Disclosure:** Additional privacy metrics in the extended framework include:
  - **$\varepsilon$-risk:** Proportion of real samples closest (by Gower’s distance) to a synthetic point than to any other real point, entropy-weighted to emphasize rare records.
  - **Attribute Disclosure Risk (ADR):** Macro-F1 of a random forest classifier predicting real record attributes using nearest synthetic neighbors.

- **Hit Rate, NN Ratios:** Fraction of real records exactly matched or median synthetic-to-real nearest neighbor distance ratio [2404.15821].

## 3. Unified Evaluation across Mixed-Type and Textual Data

SynthEval's methodological innovation is the seamless treatment of numerical and categorical variables without imposing strict preprocessing regimes (e.g., one-hot encoding). Core practices include:

- **Gower’s Distance:** For any nearest-neighbor or distance-based metric, Gower’s composite distance is used, handling numerical and categorical types uniformly:
  $$
  d_G(x, y) = \frac{1}{p} \sum_{j=1}^p d_j(x_j, y_j)
  $$
  with $d_j$ appropriately normalized or binarized per variable type [2404.15821].

- **"Mixed" Correlation Matrices:** Synthesis of statistical dependencies using Pearson (numeric-numeric), Cramer’s V (categorical-categorical), and the correlation ratio $\eta$ (numeric-categorical).

- **Permutation-Based p-values:** Empirical statistical significance is computed via permutations, avoiding reliance on model-based assumptions.

- **Text Handling:** Textual columns (e.g., product reviews) analyzed through sentiment, n-gram statistics, and entity frequency, augmenting table-centric metrics for broader data modalities [2404.14445].

## 4. Implementation and Extensibility

SynthEval is implemented in Python 3, leveraging common data science libraries:
- pandas, numpy for data manipulation,
- scikit-learn, scipy for statistical testing and machine learning (KS tests, random forests, logistic regression),
- matplotlib/seaborn for visualization,
- NLTK or TextBlob for sentiment and keyword extraction where text is evaluated [2404.14445].

A metrics registry auto-discovers per-metric classes, and extensibility is provided by a plug-in mechanism: new metric modules can be integrated by conforming to a prescribed evaluate interface. Configuration supports both preset and custom JSON-based profiles, with code-level exposure of distance metrics, cross-validation folds, and permutation parameters [2404.15821].

The reporting subsystem produces tables and plots for interactive or batch analysis, and benchmarking mode aggregates normalized metric values according to ranking strategies (linear, normal, quantile) for robust model comparison. The framework can be invoked via both importable Python API and command-line interface [2404.15821].

## 5. Empirical Results and Use Cases

In the reference implementation, SynthEval was applied to synthetic product review data generated by ChatGPT-3.5, Claude 3 Opus, and Llama 2 13B, using Amazon reviews as ground truth. Distinct outcomes include:
- **Fidelity:** All models preserved schema structure (SPS=100%), but varied in data integrity (Claude 98.4%, ChatGPT 93.9%, Llama 87.6%) and column shape similarity (Claude/ChatGPT ≈ 81%, Llama 62.3%).
- **Textual metrics:** Sentiment and keyword distribution aligned more closely for Claude and ChatGPT than Llama; synthetic reviews were typically much shorter (e.g., 16.6 words vs. 59.3 for real).
- **Utility:** Classifier accuracy for sentiment prediction on TSTR varied minimally from real (Claude 67.68%, ChatGPT 67.35%, real 67.92%), while Llama lagged (62.26%).
- **Privacy:** Quantitative MIA results indicated measurable privacy risk; higher MIA success rates signal less privacy preservation (see original tables for exact values) [2404.14445].

The framework’s extensibility to public health data was also demonstrated. In a Hepatitis C case study, SynthEval enabled model selection (e.g., privacy-boosted Bayesian network with ε-risk < 0.09) under utility-privacy trade-off constraints [2404.15821].

## 6. Comparison to Related Evaluation Frameworks and Limitations

SynthEval distinguishes itself from alternative toolkits by combining:
- Modular, plug-in metrics registry supporting tabular and textual data.
- Uniform handling of mixed data, leveraging Gower’s distance and empirical permutation schemes.
- Explicit privacy risk quantification beyond simple row overlap, incorporating MIA, ε-risk, and ADR metrics.
- Preset and customizable benchmark configurations enabling reproducibility and consistency across studies [2404.15821], [2404.14445].

Limitations include potential blind spots in domain-specific logical consistency or semantic coherence not fully captured by the implemented metrics, and the accuracy of privacy risk metrics in high-dimensional, complex real-world samples. The effectiveness of the utility metrics is dependent on the choice and adequacy of the downstream tasks selected.

## 7. Practical Guidance for Researchers

For practitioners adopting SynthEval:
- Ensure categorical variable coding adheres to consistent label schemas across real and synthetic data.
- Select or define utility tasks that reflect intended deployment scenarios; TSTR is effective for diagnostic classification tasks, but supervised tasks may require extensions.
- Interpret privacy metrics in the context of deployment risk: a high MIA recall does not imply absolute privacy violation, but signals increased risk surface.
- Modular design supports addition of domain-adapted metrics, enabling fits to specialized regulatory or compliance regimes.

SynthEval thus provides an extensible, reproducible, and principled platform for the quantitative assessment of synthetic data in machine learning pipelines centered on LLMs and tabular generation [2404.14445], [2404.15821].

Source: https://www.emergentmind.com/topics/syntheval-framework