Can Artificial Intelligence Reconstruct Ancient Mosaics? (2210.06145v1)
Abstract: A large number of ancient mosaics have not reached us because they have been destroyed by erosion, earthquakes, looting or even used as materials in newer construction. To make things worse, among the small fraction of mosaics that we have been able to recover, many are damaged or incomplete. Therefore, restoration and reconstruction of mosaics play a fundamental role to preserve cultural heritage and to understand the role of mosaics in ancient cultures. This reconstruction has traditionally been done manually and more recently using computer graphics programs but always by humans. In the last years, AI has made impressive progress in the generation of images from text descriptions and reference images. State of the art AI tools such as DALL-E2 can generate high quality images from text prompts and can take a reference image to guide the process. In august 2022, DALL-E2 launched a new feature called outpainting that takes as input an incomplete image and a text prompt and then generates a complete image filling the missing parts. In this paper, we explore whether this innovative technology can be used to reconstruct mosaics with missing parts. Hence a set of ancient mosaics have been used and reconstructed using DALL-E2; results are promising showing that AI is able to interpret the key features of the mosaics and is able to produce reconstructions that capture the essence of the scene. However, in some cases AI fails to reproduce some details, geometric forms or introduces elements that are not consistent with the rest of the mosaic. This suggests that as AI image generation technology matures in the next few years, it could be a valuable tool for mosaic reconstruction going forward.
- L. Alterio, G. Russo, and F. Silvestri, “Seismic vulnerability reduction for historical buildings with non-invasive subsoil treatments: The case study of the mosaics palace at herculaneum,” International Journal of Architectural Heritage, vol. 11, no. 3, pp. 382–398, 2017. [Online]. Available: https://doi.org/10.1080/15583058.2016.1238969
- D. Riccio, S. Caggiano, M. De Marsico, R. Distasi, and M. Nappi, “Mosaic+: tools to assist virtual restoration,” 08 2015.
- F. Stanco, S. Battiato, and G. Gallo, “Digital imaging for cultural heritage preservation,” Analysis, Restoration, and Reconstruction of Ancient Artworks, 2011.
- L. Fazio, M. Lo Brutto, and G. Dardanelli, “Survey and virtual reconstruction of ancient roman floors in an archaeological context,” ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLII-2/W11, pp. 511–518, 05 2019.
- M. Monti and G. Maino, “Non-metric digital reconstruction of roman mosaics excavated in the city of ravenna (italy),” Virtual Archaeology Review, vol. 9, no. 19, p. 66–75, Jul. 2018. [Online]. Available: http://ojs.upv.es/index.php/var/article/view/7227
- L. Bordoni, F. Mele, A. Sorgente, P. Mulholland, A. Wolff, E. Kilfeather, and M. Maguire, “Artificial intelligence for cultural heritage.”
- D. Bienvenido-Huertas, J. E. Nieto-Julián, J. J. Moyano, J. M. Macías-Bernal, and J. Castro, “Implementing artificial intelligence in h-bim using the j48 algorithm to manage historic buildings,” International Journal of Architectural Heritage, vol. 14, no. 8, pp. 1148–1160, 2020. [Online]. Available: https://doi.org/10.1080/15583058.2019.1589602
- A. Bartoli, G. Fenu, E. Medvet, F. A. Pellegrino, and N. Timeus, “Segmentation of mosaic images based on deformable models using genetic algorithms,” in Smart Objects and Technologies for Social Good, O. Gaggi, P. Manzoni, C. Palazzi, A. Bujari, and J. M. Marquez-Barja, Eds. Cham: Springer International Publishing, 2017, pp. 233–242.
- A. Felicetti, M. Paolanti, P. Zingaretti, R. Pierdicca, and E. S. Malinverni, “Mo. se.: Mosaic image segmentation based on deep cascading learning,” Virtual Archaeology Review, vol. 12, no. 24, pp. 25–38, 2021.
- C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” 2022. [Online]. Available: https://arxiv.org/abs/2205.11487
- J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu, “Scaling autoregressive models for content-rich text-to-image generation,” 2022. [Online]. Available: https://arxiv.org/abs/2206.10789
- M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, and J. Tang, “Cogview: Mastering text-to-image generation via transformers,” 2021. [Online]. Available: https://arxiv.org/abs/2105.13290
- A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” 2021. [Online]. Available: https://arxiv.org/abs/2102.12092
- A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022. [Online]. Available: https://arxiv.org/abs/2204.06125
- A handbook of Roman art : a survey of the visual arts of the Roman world. Oxford: Phaidon, 1983.
- M. Henig and G. Soffe, “The thruxton roman villa and its mosaic pavement,” Journal of the British Archaeological Association, vol. 146, no. 1, pp. 1–28, 1993. [Online]. Available: https://doi.org/10.1179/jba.1993.146.1.1
- A. Amanatiadis, V. G. Kaburlasos, and E. B. Kosmatopoulos, “Understanding deep convolutional networks through gestalt theory,” in 2018 IEEE International Conference on Imaging Systems and Techniques (IST), 2018, pp. 1–6.