When and how synthetic data improve generalization and transfer

Characterize the regimes in which synthetic data augmentation improves out‑of‑distribution generalization and transfer learning performance, including identifying beneficial types of distributional shift, quantifying the impact of generative‑model estimation error, and developing diagnostics for harmful extrapolation.

Background

Synthetic data are increasingly used to improve out‑of‑distribution generalization by covering underrepresented regions or simulating shifts, but theoretical understanding of when this helps remains limited.

The paper explicitly calls for characterizing the conditions under which synthetic augmentation aids generalization and transfer, emphasizing the roles of shift types, generative‑model estimation error, and the need for diagnostics to detect harmful extrapolation.

References

A central open problem is therefore to characterize when and how synthetic data improve generalization ability and transferability. This includes, but is not limited to, identifying the types of distributional shifts for which synthetic augmentation is beneficial, understanding the role of the estimation error of the generative model, and developing diagnostics to detect harmful extrapolation.

— Harnessing Synthetic Data from Generative AI for Statistical Inference  (2603.05396 - Abdel-Azim et al., 5 Mar 2026) in Section 4, Extrapolation, Generalization, and Transfer

The real-world test set we leveraged for experiments in this paper still represents a limited set of environments and imaging conditions, and therefore cannot determine whether improvements generalize across different sites, water conditions, illumination, terrain, and sensor configurations.

— Towards Scaling Marine Perception with Synthetic Data  (2609.20680 - Ma et al., 17 Sep 2026) in Discussion, final paragraph before the Conclusion

This question becomes more practical when considering data scalability: curating large-scale real videos with explicit reasoning goals is expensive, and synthetic data offers a more controllable and scalable alternative, but it remains unclear whether such scaling can systematically improve video reasoning in realistic scenarios.

— VGI-BENCH: Probing Visual Intelligence in Video Generation Models  (2608.19583 - He et al., 20 Aug 2026) in Section 5.2, “How Well does Scaling Tuning Transfer?”

Relatedly, mean-pooled representations from frozen transformer encoders like RoBERTa are known to exhibit anisotropy \citep{ethayarajh-2019-contextual}, which can distort cosine-based distance measures; it remains an open question whether our distance-sensitive findings would hold under embeddings specifically designed for cosine geometry, such as sentence-transformers \citep{reimers-gurevych-2019-sentence}.

— The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification  (2609.03652 - Sorahi et al., 3 Sep 2026) in Section Limitations, second limitation paragraph

One possible contributor is coverage-narrowing, the tendency of LLM-generated text to under-represent the tails of the source distribution [28,29]; however, with only 5 and 6 positive test cases for subacute and chronic, our data can neither confirm nor exclude this, and we raise it only as a hypothesis for adequately powered future work.

— Comparison of techniques for fine-tuning open-weight models for entity extraction from radiology reports  (2609.40236 - Mansuri et al., 30 Sep 2026) in Discussion, paragraph beginning “A fine-tuned open-weight model matched a leading hosted model”