Closing the synthetic-to-real domain gap for unpaired photographs

Develop methods to close the domain gap between the synthetic material renderings used for self-supervised pretraining and unpaired real photographs, enabling the physically-grounded visual representations learned from synthetic data to transfer reliably to real-world images.

Background

The paper introduces a self-supervised ViT-based backbone trained with material-aware physical augmentations, using multiple renderings of the same material under varied geometry and illumination to learn physically-grounded features. This pretraining is conducted entirely on synthetic data generated from procedural materials rendered with a path tracer.

While the model demonstrates strong invariance to geometry and lighting and improved material-aware performance, the authors highlight that transferring these representations to real photographs without paired supervision is hindered by a synthetic-to-real domain gap. Addressing this gap is essential for broad application of the learned features to real-world imagery and tasks.

References

Moreover, our pretraining relies entirely on synthetic data; closing the domain gap to unpaired real photographs remains an open challenge.

— Φeat: Physically-Grounded Feature Representation  (2511.11270 - Vecchio et al., 14 Nov 2025) in Section 6 (Limitations)

Whether it detects real defects as reliably, and whether the generated patterns match what occurs in service, can only be assessed once real in situ defect images become available.

— Visual Anomaly Synthesis for Model Selection in Data Scarcity  (2609.37360 - Pröll et al., 29 Sep 2026) in Section Conclusion

Additionally, since a real scene cannot be captured simultaneously with and without the same flare, the removal model is trained on flares composited onto otherwise clean images rather than on paired real captures; closing this synthetic-to-real gap would likely require a capture rig that optically isolates the flare, which we leave to future work.

— Lens Flare Removal and Reconstruction  (2609.39527 - Yenamandra et al., 30 Sep 2026) in Section 6, Future Work