Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data (2505.10551v1)

Abstract: With the development of photorealistic diffusion models, models trained in part or fully on synthetic data achieve progressively better results. However, diffusion models still routinely generate images that would not exist in reality, such as a dog floating above the ground or with unrealistic texture artifacts. We define the concept of feasibility as whether attributes in a synthetic image could realistically exist in the real-world domain; synthetic images containing attributes that violate this criterion are considered infeasible. Intuitively, infeasible images are typically considered out-of-distribution; thus, training on such images is expected to hinder a model's ability to generalize to real-world data, and they should therefore be excluded from the training set whenever possible. However, does feasibility really matter? In this paper, we investigate whether enforcing feasibility is necessary when generating synthetic training data for CLIP-based classifiers, focusing on three target attributes: background, color, and texture. We introduce VariReal, a pipeline that minimally edits a given source image to include feasible or infeasible attributes given by the textual prompt generated by a LLM. Our experiments show that feasibility minimally affects LoRA-fine-tuned CLIP performance, with mostly less than 0.3% difference in top-1 accuracy across three fine-grained datasets. Also, the attribute matters on whether the feasible/infeasible images adversarially influence the classification performance. Finally, mixing feasible and infeasible images in training datasets does not significantly impact performance compared to using purely feasible or infeasible datasets.

Evaluating the Impact of Feasibility in Synthetic Training Data

The paper titled "Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data" introduces an analytical framework to explore the necessity and impact of feasibility in synthetic training data. As synthetic data generation becomes increasingly prevalent due to the data-intensive nature of large-scale pre-trained models, understanding whether attributes in synthetic images need to be feasible—meaning they could realistically exist in the world—becomes crucial, especially for training models like CLIP classifiers.

The paper delineates the concept of feasibility in synthetic data, defining feasible images as those whose attributes could occur in real-world scenarios. VariReal, a proposed pipeline, minimally alters real images to create synthetic variants with either feasible or infeasible attributes. This methodology focuses on three primary attributes: background, color, and texture. The process utilizes a combination of diffusion models, mask-based editing, and LLMs to ensure precise adherence to feasibility criteria or intentional detours into infeasibility.

Key Methodological Insights:

1. VariReal Pipeline: The introduced pipeline effectively generates both feasible and infeasible synthetic images by editing based on generated prompts. These prompts are obtained using GPT-4, which provides attribute descriptors validated through a user paper to ensure the differentiation between feasible and infeasible scenarios.
2. Training Experiments: The paper employs LoRA to fine-tune CLIP models under different synthetic training scenarios, including wholly synthetic datasets and mixed datasets comprising both real and synthetic images. The experiments test the efficacy across three comprehensively controlled settings: fully feasible, fully infeasible, and mixed feasibility.
3. Synthetic vs. Real Data: The validation against real-world datasets (Oxford Pets, FGVC Aircraft, Stanford Cars) indicates that mixing feasible and infeasible data does not impair model generalization compared to using purely real or purely synthetic datasets. This finding is consistent even when synthetic data constitute five times the size of real data.

Key Findings:

• Feasibility Impact: The results reveal minimal accuracy fluctuations (typically less than 0.3%) in CLIP performance when trained with feasible versus infeasible data. This suggests that strict adherence to feasibility in synthetic data may be unnecessary for effective model training.
• Attribute-Specific Impact: Background modifications yield consistent performance improvements, suggesting that variations in environmental context aid learning. However, modifications to core object attributes like color and texture sometimes negatively impact performance, indicating potential disruptions to class-relevant signals.
• Synthetic Data Utilization: The analysis confirms that infeasible data, often considered out-of-distribution (OOD), can still provide beneficial stochasticity without adversely affecting generalization.

Theoretical and Practical Implications:

This research underscores the nuanced role of feasibility in synthetic data. While feasible data slightly outperforms infeasible data, the differences are negligible, suggesting that focusing on broader data augmentation strategies might be more beneficial than strictly enforcing feasibility. The findings also challenge the assumption that synthetic data must closely mimic reality, encouraging the exploration of broader synthetic datasets that might introduce beneficial variations.

The paper implies that in scenarios where real-world data is scarce, combining feasible and infeasible synthetic data allows for a robust augmentation approach, enhancing model robustness and accuracy without necessitating an unattainable perfection in mimicry.

Future Prospects in AI and Synthetic Data:

As AI endeavors become more data-intensive, the rapid generation of synthetic data offers promising avenues for model training. This paper paves the way for more relaxed synthetic data creation strategies, potentially reducing the meticulous screening for feasibility and instead focusing on how meaningful variations can enhance learning. Future research could expand on this by exploring other attributes beyond background, color, and texture, such as motion dynamics in video data or context-based textual descriptions in multi-modal datasets.

By illuminating the limited impact of feasibility in synthetic training data, this research invites further exploration into computationally efficient and diverse data generation techniques that leverage the broad variability offered by synthetic methods.