SynthTextEval: Synthetic Text Evaluation Toolkit

• SynthTextEval is an open-source Python toolkit for generating, auditing, and evaluating synthetic text, emphasizing privacy, fairness, and utility in sensitive domains such as healthcare and law.
• It integrates a control code–driven generation module with differential privacy, quantitative evaluation metrics, and an expert-review GUI for comprehensive text assessment.
• Empirical studies in legal and clinical settings show its effectiveness in balancing performance metrics like accuracy, F1-score, and privacy leakage against practical ethical tradeoffs.

SynthTextEval is an open-source Python toolkit for standardized generation, auditing, and evaluation of synthetic text in high-stakes application domains, with an explicit focus on healthcare and law. Its motivation stems from the increasing viability of LLM–generated text for downstream tasks and privacy-preserving data sharing, contingent on the ability to robustly and holistically evaluate such data for utility, fairness, privacy leakage, distributional fidelity, and domain expert acceptability. SynthTextEval consolidates the evaluation of these dimensions in a single modular platform and includes built-in modules for differential privacy (DP)–constrained text generation, multi-faceted analysis, and expert review (Ramesh et al., 9 Jul 2025).

1. Toolkit Architecture and Components

SynthTextEval comprises three principal modules:

1. Generation module: Enables control-code–driven synthetic text construction from a fine-tuned autoregressive LLM, supporting optional differential privacy via DP-SGD.
2. Evaluation modules: Quantitatively assess synthetic data across utility, fairness, privacy, text quality/distributional similarity, and corpus descriptive statistics.
3. Expert-review GUI: Provides a retrieval-based interface for domain expert qualitative comparison of synthetic and real text.

The architecture is data-agnostic, permitting users to either upload existing synthetic corpora or generate new synthetic corpora within the toolkit for evaluation. Each evaluation dimension can be invoked individually, or the full audit pipeline can be run end-to-end (Ramesh et al., 9 Jul 2025).

2. Synthetic Text Generation Methodology

The base generation approach directly follows established protocols:

• Utilizes a pretrained autoregressive LM fine-tuned on the real (in-domain) text corpus.
• Employs control codes (metadata prefixes specifying attributes such as ICD-9 code, gender, country, or year) as input labels, steering the output distribution during both training and inference:

$P(x) = \prod_{i=1}^{n} P(x_i \mid x_{<i}, c)$

where $c$ denotes the control code.

• After fine-tuning, synthetic text is generated via top-$k$ and/or nucleus sampling to enhance output diversity.
• Differential privacy is optionally enforced by fine-tuning with DP-SGD, yielding formal $(\varepsilon, \delta)$-DP guarantees:

$$\Pr[F(D) \in y] \le e^{\varepsilon}\Pr[F(D') \in y] + \delta$$

for neighboring datasets $D, D'$ differing by one record.

The generation workflow is: prepare corpus → attach control codes → fine-tune (with/without DP) → generate synthetic text (with top-$k$/nucleus sampling) → optional quality filtering → pass to evaluation modules. Representative control codes in the case studies include diagnosis codes and demographic fields (healthcare) or country/year (law) (Ramesh et al., 9 Jul 2025).

3. Evaluation Dimensions and Metrics

SynthTextEval operationalizes the following evaluation dimensions:

3.1 Utility

Assesses whether synthetic data can substitute for real data in downstream modeling. Supported tasks are classification (e.g., ICD-9 code prediction, legal case attribution) and coreference resolution, chosen for their different structural and semantic requirements.

• Synthetic corpora are automatically silver-annotated using external pretrained models.
• Models are trained on synthetic (silver-labeled) data and evaluated on real gold-standard test sets.
• Reported metrics include accuracy and F1 for classification, F1 for coreference. For instance:

$$\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

$$F_1 = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

where TP, FP, etc. are standard confusion matrix entries.

3.2 Fairness

Evaluates subgroup parity in model predictions using metrics:

• Equalized Odds (EOD):

$$EOD = \max_{d \in D} TPR_d - \min_{d \in D} TPR_d + \max_{d \in D} FPR_d - \min_{d \in D} FPR_d$$

where $c$0 is the set of demographic subgroups.

• Equality Difference (ED): Sum of deviations between subgroup and overall rates for quantities such as FPR, TPR, TNR, and FNR:

$c$1

Demographic parity is not implemented; only equalized odds and equality-difference–style metrics are reported.

3.3 Privacy Leakage

Privacy leakage is assessed empirically in generated text itself:

• Canary-based audits: Measure the recoverability (rank, perplexity) of secret canary strings injected into the training set:
  • Lower rank or perplexity denotes higher risk.
  • DP increases rank/perplexity, lowering leakage.
• Entity leakage percentage:

$c$2

Quantifies the frequency of private entity repetition.

• Context leakage:

$c$3

where $c$4 is a window around the entity.

Membership inference attacks and advantage are not the main focus, and are not implemented in this setting.

3.4 Distributional Similarity and Quality

Distributional differences between real and synthetic corpora are quantified using:

• Perplexity:

$c$5

The paper states higher perplexity indicates higher semantic coherence.

• Fréchet Inception Distance (FID):

$c$6

where $c$7 and $c$8 are embedding means and covariances.

• MAUVE: Area under a divergence frontier based on KL divergence between embedding distributions of real and synthetic text.

Jensen–Shannon divergence is not included in the toolkit; KL appears only as part of MAUVE’s formulation.

3.5 Expert Qualitative Review

A GUI is provided for domain experts (e.g., clinicians, legal professionals) to inspect and annotate synthetic text. Features include:

• Nearest-neighbor retrieval (embedding-based) of real text for comparison.
• Entity-matching mode to examine leakage.
• Side-by-side visualization and free-form comment entry.
• No rubric-based or formalized annotation; qualitative review is captured via saved comments and case-by-case inspection.

4. Implementation and Software Features

SynthTextEval is an installable Python package (available via PyPI) and offers both command-line and GUI interfaces. The package structure exposes:

• Automatic generation and saving of synthetic corpora, with customizable control codes, privacy budgets ($c$9, $k$0), sampling parameters (top-$k$1, top-$k$2).
• Evaluation via a suite of metrics, yielding numeric outputs and visualizations.
• Expert review launched via command-line for interactive annotation.

Inputs accepted include corpora (.jsonl files), control code metadata, subgroup attributes, and labels or annotators. Outputs consist of generated text, numerical summary files (.json, .csv), visual reports, and qualitative commentary.

Dependencies are not exhaustively listed but include standard NLP libraries, pretrained LMs (including BERT), external models for silver annotation, and embedding models (e.g., GPT-2 for MAUVE computation).

5. Empirical Case Studies: Law and Healthcare

A. Law: Text Anonymization Benchmark (TAB)

• Synthetic European court cases generated with and without DP (control over country, year).
• TAB synthetic cases are approximately half as long and have half as many unique words as real cases.
• FID shows both synthetic datasets are close to real data; MAUVE and perplexity are worse for synthetic data.
• Non-DP synthetic data yields higher F1 (0.96) and accuracy (0.99) on country classification than DP synthetic (F1: 0.54, accuracy: 0.95) and real data (F1: 0.67, accuracy: 0.32).
• High PII reoccurrence in generations: 50.3% (non-DP), 48.5% (DP).

B. Healthcare: MIMIC-III and i2b2/VA

• Synthetic notes generated for the 10 most frequent ICD-9 codes with/without DP (control over diagnosis/gender/ethnicity).
• Utility for ICD-9 code prediction: non-DP $k$3 between 0.61 and 0.06; DP $k$4 between 0.40 and 0.27—both below real-data training.
• For mention/coreference detection: synthetic (DP and non-DP) $k$5, both much lower than real gold ($k$6).
• Fairness (race): real is fairest (FNED $k$7), DP synthetic is least fair (FNED $k$8).
• Privacy leakage: substantially lower (4.8% non-DP, 3.8% DP) than for legal data; DP increases canary rank and perplexity, improving privacy for all canary types.

6. Best Practices and Interpretive Recommendations

SynthTextEval’s design and case studies embed the following recommendations:

• Synthetic text evaluation must be multidimensional; utility metrics alone are insufficient. Downstream performance does not imply fairness, privacy, or overall suitability.
• Metrics must be matched to application context, but evaluation should be broad enough to detect major risks.
• Formal DP guarantees should be interpreted in conjunction with empirical leakage measurements. Privacy claims may not transfer across settings or datasets.
• Fairness assessment across sensitive subgroups is necessary; synthetic text may amplify inequities.
• Expert judgment should be incorporated, as some harms and failures are domain- and context-specific and can evade automatic metrics.
• No metric suite is comprehensive; passing all provided audits does not guarantee ethical adequacy or practical safety.

A central lesson is that the quality of synthetic text in high-stakes domains reflects a complex privacy–utility–fairness tradeoff surface, not a single objective. Non-DP synthetic corpora often provide utility and sometimes fairness advantages, but at the cost of privacy risk; DP reduces leakage but may degrade utility and exacerbate fairness concerns, particularly with limited data (Ramesh et al., 9 Jul 2025).

7. Significance, Scope, and Limitations

SynthTextEval represents the first consolidated open framework for rigorous multi-dimensional auditing of synthetic text in sensitive domains. Its principal contributions are the integration and standardization of a diverse metric suite, demonstrated effectiveness on legal and clinical corpora, and explicit support for both automated and expert-in-the-loop analysis. Limitations include the reliance on specific metric classes (e.g., no membership inference audit, no demographic parity), possible constraints from silver annotation strategy, and the toolkit’s dependence on the quality and granularity of available real and synthetic data. These aspects suggest future research could extend SynthTextEval’s evaluation repertoire or adapt it to additional domains and threat models (Ramesh et al., 9 Jul 2025).