---
title: 'TSTR: Train on Synthetic, Test on Real'
url: https://www.emergentmind.com/topics/train-on-synthetic-test-on-real-tstr
type: topic
---

# TSTR: Train on Synthetic, Test on Real

Train on Synthetic, Test on Real (TSTR) is a rigorous evaluation protocol for assessing the capacity of synthetic data to replace or augment real data in supervised learning. In the TSTR setting, a model is wholly trained on synthetic data and then evaluated on a real, held-out test set. This protocol enables precise quantification of the domain gap and practical utility of synthetic data for downstream tasks. TSTR has been systematically analyzed across a variety of modalities (images, time series, tabular data, multimodal settings), tasks (object detection, segmentation, classification, temporal prediction), and with different synthetic data generation frameworks including GANs, VAEs, diffusion models, and simulation pipelines.

## 1. Core Definition and Mathematical Formalism

Let $S_{\mathrm{syn}}$ be a synthetic dataset (e.g. images $x'_j$, labels $y'_j$), and $T_{\mathrm{real}}$ a real test set of $(x_i, y_i)$. Under TSTR, a predictor $f_\theta$ (parameterized by $\theta$) is trained on $S_{\mathrm{syn}}$:
\[
\theta^* = \arg\min_\theta \frac{1}{|S_{\mathrm{syn}}|} \sum_{(x', y') \in S_{\mathrm{syn}}} \mathcal{L}(f_\theta(x'), y')
\]
and then evaluated on $T_{\mathrm{real}}$:
\[
\mathrm{TSTR} = \frac{1}{|T_{\mathrm{real}}|} \sum_{(x, y) \in T_{\mathrm{real}}} 1[f_\theta(x) = y]
\]
For regression tasks, the metric may be RMSE, $R^2$, or other task-appropriate losses. TSTR performance is typically compared with a real-to-real (TRTR) baseline and the ratio or gap is analyzed to quantify the synthetic data’s utility [2510.19728, 2312.00694, 2511.17590, 2210.07617, 2508.02294, 2405.19822, 2302.04305, 1908.00967, 1706.02633].

## 2. Use Cases and Empirical Findings Across Modalities

TSTR has been employed in diverse research contexts:
- **Object Detection and Segmentation:** Synthetic datasets such as GTAV or procedurally generated CAD scenes allow training detectors (YOLOv3, Mask R-CNN, Faster R-CNN, Swin Transformer) and evaluating generalization to real benchmarks like BDD100K, Cityscapes, or WISDOM-Real [2312.00694, 2405.19822, 1809.05825, 2304.12463].
- **Satellite/Landcover Segmentation:** Mask-conditional GANs with SPADE blocks produce synthetic satellite imagery to train U-Nets for per-pixel landcover, matching or surpassing real-trained models when synthetic and real are mixed [2302.04305].
- **Time Series (Medical, Forecasting):** RGAN/RCGAN, TimeDiff, and VAE+Diffusion models generate synthetic ICU/aviation time series. TSTR shows that state-of-the-art generators, particularly transformer-based and diffusion-based, can recover 94–97% of real-data utility in flight and ICU prediction tasks, with retention gaps $\Delta_\mathrm{TSTR}\approx0.01$ AUROC in MIMIC-III/eICU and up to 97% $R^2$ retention in aviation forecasting [1706.02633, 2510.19728, 2508.02294].
- **Multimodal Relation Extraction:** TSTR can be used when only one modality is present at train time. Methods such as MI²RAGE leverage chained cross-modal generation and mutual-information-based filtering to construct synthetic multimodal data, achieving or surpassing state-of-the-art performance compared to real multimodal training [2312.03025].
- **Tabular Data:** In tabular domains, TSTR (often classification accuracy or AUROC) is the default utility metric for synthetic data in sensitive settings (healthcare, enterprise) [2511.17590, 2508.02294].

## 3. Empirical Performance and Critical Phenomena

Observed TSTR performance exposes the domain adaptation gap and the efficacy of different strategies:

| Domain / Task                | Real→Real Baseline | TSTR (Synthetic→Real) | Retention/Gain               | Reference           |
|------------------------------|--------------------|----------------------|------------------------------|---------------------|
| YOLOv3 on BDD (object det.)  | 0.43 mAP           | 0.12 mAP             | $\Delta_{TSTR}\approx$0.31   | [2312.00694]        |
| U-Net (sat. landcover, mIoU) | 0.52               | 0.41–0.42            | $~80\%$ of real; mixing: 0.58 | [2302.04305]        |
| ICU Mortality (AUROC)        | 0.88               | 0.87–0.88            | $\Delta_{TSTR}\leq0.01$      | [2510.19728]        |
| Aviation Forecast (R$^2$)    | 0.44               | 0.43                 | 98% retention                | [2508.02294]        |
| ImageNet1K, ResNet-50 (Top-1)| 79.6%              | 70.9–76.0%           | 89–95% (at $>6\times$ synth) | [2310.10402]        |

Synthetic data matched or surpassed real-only training with increased synthetic diversity, improved generators, and careful domain translation or augmentation. However, pronounced failures occur when high-level scene statistics, class-conditional cues, or domain-specific textures are not aligned; e.g., domain gap in detector heads [2312.00694], mode collapse [2210.07617], or overfitting to synthetic artifacts [1908.00967]. Mixing synthetic with a small amount of real data consistently boosts utility [2302.04305, 2408.14559].

## 4. Diagnostic Analyses and Layerwise Similarity

TSTR provides a practical test of end-to-end generalization, but various papers have supplemented this with more diagnostic metrics:
- **CKA (Centered Kernel Alignment):** Layerwise CKA similarity between real-trained and synthetic-trained models reveals that low-level features are robust (CKA $>0.9$ in early layers), but mid- and high-level features (detection head, scene semantics) diverge sharply (CKA as low as $0.05-0.2$ in the detection head) and correlate with mAP drop [2312.00694].
- **Feature Distance Metrics:** Measures such as train2test Mahalanobis distance and AP$_{\text{t2t}}$ expose how well synthetic-augmented pools cover the real test feature space and relate directly to AP; lower distances predict higher detection accuracy [2408.14559].
- **SHAP Attribution Distance:** In tabular settings, the SHAP Distance reveals that TSTR accuracy can remain high even when the model’s semantic reasoning patterns diverge from real-trained models, thus TSTR should be viewed as a minimal utility criterion rather than a guarantee of semantic fidelity [2511.17590].
- **Ablation and Sampling Strategies:** Adversarial (student–teacher) sampling, mutual-information filtering, and automated curriculum methods improve TSTR by focusing generation or training on hard or informative synthetic samples, mitigating overfitting and domain collapse [2312.03025, 1908.00967, 2412.05466].

## 5. Methodological Innovations and Recommendations

Multiple strategies have improved TSTR outcomes:
- **Synthetic Generation Strategies:**
  - Conditional mask-driven GANs (SPADE) and domain-randomized renderings for spatial/semantic alignment [2302.04305, 2303.09165, 1809.05825].
  - Transformer-based and diffusion-based tabular, image, and time-series generators to maximize coverage of real-world joint distributions [2508.02294, 2510.19728, 2310.10402].
- **Training Recipe:**
  - Pretrain backbones on real data and freeze early feature statistics when training on synthetic images [2405.19822].
  - Use strong data augmentation (MixUp, Mosaic, large-scale jitter, color transforms) to avoid overfitting to synthetic artifacts and to bridge photorealism deficiencies [2405.19822, 2304.12463].
  - Fine-tune mid-to-late network layers and detection heads; early convolutional features are less domain-specific and may be frozen [2312.00694].
- **Evaluation and Deployment:**
  - Always benchmark TSTR alongside TRTR and, where meaningful, train-on-real, test-on-synthetic (TRTS) for dataset auditing [2510.19728, 2210.07617].
  - Apply attribution-based metrics or layerwise CKA to prevent falsely high TSTR readings that mask semantic failure [2511.17590, 2312.00694].
  - For small data settings, mixing modest amounts of real, even from cross-domain, with synthetic can vastly increase effective training set size [2408.14559, 2302.04305].
- **Subgroup and Privacy Analysis:**
  - Synthetic data can support granular (subgroup-level) model evaluation, often matching or outperforming small real test subsets in statistical reliability, enabling privacy-preserving benchmarking [2510.19728].

## 6. TSTR Limitations, Analytical Insights, and Future Work

TSTR is a powerful, practical measure but not a comprehensive guarantee:
- **Blind Spots:** TSTR will not reveal if synthetic data reproduces incorrect label reasoning, has semantic gaps, or over-smooths rare/corner cases; attribution-based checks or task-specific diagnostics are required [2511.17590, 2210.07617].
- **Task Specificity:** TSTR utility is tightly linked to the supervised task and may not generalize to new objectives or transfer learning regimes. Precision/recall/F1, calibration, and robustness metrics are essential to supplement mAP or accuracy [2312.00694, 2511.17590].
- **Assumptions:** Many TSTR pipelines assume that $P_{syn}(y|x)\approx P_{real}(y|x)$, i.e., that synthetic and real domains share label semantics. If this fails, the TSTR metric conflates distribution and annotation artifacts.
- **Generator Quality is Pivotal:** The best performing frameworks for TSTR use deep distribution-matching objectives (MMD, guidance, attribute-driven prompting, mutual information selection) and, when provided with sufficient diversity and scale, outperform GANs or Copula models [2310.10402, 2510.19728, 2412.05466, 2508.02294]. Scaling up synthetic set cardinality partially substitutes for narrower synthetic fidelity, with observed linear-to-saturating gains [2310.10402].

Future directions include curriculum-based or online TSTR, active domain adaptation during deployment, further extension to more complex or multimodal tasks, automated generator evaluation and selection, and formal privacy overlays for sensitive domains [2510.19728, 2312.03025, 2412.05466].

## 7. Summary Table: TSTR Evaluation Landscape

| Study / Domain                | Generator/Methodology            | TSTR Metric/Result        | Recommendations / Limitations           |
|-------------------------------|----------------------------------|--------------------------|-----------------------------------------|
| Object Detection [2312.00694] | YOLOv3 on GTAV/BDD               | mAP, 0.12 (vs 0.43 real) | Fine-tune middle layers, diversify cues |
| Satellite Segmentation [2302.04305] | Mask-cond. SPADE-GAN      | mIoU, 0.41–0.42           | Mix 50-50 real/synth for best mIoU      |
| Tabular, Aviation [2508.02294]| REaLTabFormer, TabSyn            | $R^2$ retention: 94–97%   | Use transformer autoregressive models   |
| ICU Time Series [2510.19728]  | Enhanced TimeAutoDiff             | $\Delta_\mathrm{TSTR} \le 0.01$ AUROC  | Add MMD/consistency penalties           |
| ImageNet Classification [2310.10402] | Diffusion+MMD+CLIP      | Top-1: 70.9–76%           | Scale synthetic data, combine w/ guidance|
| Multimodal Extraction [2312.03025]| MI²RAGE, CCG+MI Filter   | F1: 92.8 (surpasses real) | Diversity and high MI essential         |
| 2D Pose [1908.00967]          | Synthetic Humans+Augmentation     | mAP: 13.4 (synthetic)     | Teacher-student, focus on hard samples  |

TSTR is now a foundational evaluation for learning with synthetic data, offering actionable diagnosis of the synthetic–real gap and providing clear guidance for synthetic data generation and task-specific transfer. Best practices mandate pairing TSTR with diversity/attribution-based diagnostics and optimizing both distributional match and label fidelity for target deployments.

Source: https://www.emergentmind.com/topics/train-on-synthetic-test-on-real-tstr