Papers
Topics
Authors
Recent
Search
2000 character limit reached

ScrollTimes: Tracing the Provenance of Paintings as a Window into History

Published 15 Jun 2023 in cs.HC and cs.CV | (2306.08834v2)

Abstract: The study of cultural artifact provenance, tracing ownership and preservation, holds significant importance in archaeology and art history. Modern technology has advanced this field, yet challenges persist, including recognizing evidence from diverse sources, integrating sociocultural context, and enhancing interactive automation for comprehensive provenance analysis. In collaboration with art historians, we examined the handscroll, a traditional Chinese painting form that provides a rich source of historical data and a unique opportunity to explore history through cultural artifacts. We present a three-tiered methodology encompassing artifact, contextual, and provenance levels, designed to create a "Biography" for handscroll. Our approach incorporates the application of image processing techniques and LLMs to extract, validate, and augment elements within handscroll using various cultural heritage databases. To facilitate efficient analysis of non-contiguous extracted elements, we have developed a distinctive layout. Additionally, we introduce ScrollTimes, a visual analysis system tailored to support the three-tiered analysis of handscroll, allowing art historians to interactively create biographies tailored to their interests. Validated through case studies and expert interviews, our approach offers a window into history, fostering a holistic understanding of handscroll provenance and historical significance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Beijing Erudition Digital Technology Research Center. http://dh.ersjk.com.
  2. Chinese Historical Figures Seal Data (CHFSD). https://www.zjlib.cn/zyszzy/index.htm#searchs_1_div.
  3. Hieronymus Bosch. https://sammlung.staedelmuseum.de/en.
  4. Mapping Paintings. http://www.mappingpaintings.org/.
  5. Person Authority Database (PerAD). https://authority.dila.edu.tw/person/.
  6. Place Authority Database (PlaAD). https://authority.dila.edu.tw/place/.
  7. Time Authority Database (TAD). https://authority.dila.edu.tw/time/.
  8. Frequency-tuned salient region detection. In Proc. CVPR, pp. 1597–1604, 2009. doi: 10 . 1109/CVPR . 2009 . 5206596
  9. S. Avidan and A. Shamir. Seam carving for content-aware image resizing. In Proc. SIGGRAPH, pp. 10–es. ACM, NY, USA, 2007. doi: 10 . 1145/1275808 . 1276390
  10. E. Benard and B. Moon. Goddesses who rule. Oxford University Press, 2000.
  11. An integrated approach to 3d web visualization of cultural heritage heterogeneous datasets. Remote Sensing, 11(21), 2019. doi: 10 . 3390/rs11212508
  12. P. Burke. Eyewitnessing: The uses of images as historical evidence. Cornell University Press, 2001.
  13. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv:1706.05587, Dec. 2017. doi: 10 . 48550/arXiv . 1706 . 05587
  14. F. Cheng. Empty and Full: The Language of Chinese Painting. Shambhala Publications, 1994.
  15. Essential terms of Chinese painting. City University of HK Press, 2018.
  16. Understanding korean food culture from korean paintings. Journal of Ethnic Foods, 3(1):42–50, 2016. doi: 10 . 1016/j . jef . 2016 . 01 . 002
  17. M. Damova and D. Dannells. Reason-able view of linked data for cultural heritage. In Proc. S3T, pp. 17–24. Springer, Berlin, Heidelberg, 2011. doi: 10 . 1007/978-3-642-23163-6_3
  18. Connoisseur: Provenance analysis in paintings. In Proc. SSCI, pp. 1–8, 2021. doi: 10 . 1109/SSCI50451 . 2021 . 9659547
  19. E. Davis and B. Heravi. Linked data and cultural heritage: A systematic review of participation, collaboration, and motivation. J. Comput. Cult. Herit., 14(2), may 2021. doi: 10 . 1145/3429458
  20. K. Davis. Old metadata in a new world: Standardizing the getty provenance index for linked data. Art Libraries Journal, 44(4):162–166, 2019. doi: 10 . 1017/alj . 2019 . 24
  21. Optimized image resizing using seam carving and scaling. ACM Trans. Graph., 28(5):1–10, dec 2009. doi: 10 . 1145/1618452 . 1618471
  22. Feature fusion based cross-modal retrieval for traditional chinese painting. In Proc. ICCST, pp. 383–387, 2020. doi: 10 . 1109/ICCST50977 . 2020 . 00080
  23. Enriching a small artwork collection through semantic linking. In The Semantic Web. Latest Advances and New Domains, pp. 724–740. Springer, Cham, 2016. doi: 10 . 1007/978-3-319-34129-3_44
  24. A knowledge management architecture for digital cultural heritage. J. Comput. Cult. Herit., 10(3), jul 2017. doi: 10 . 1145/3012289
  25. Evaluation and analysis of white space in wu guanzhong’s chinese paintings. Leonardo, 52(2):111–116, 2019. doi: 10 . 1162/leon_a_01409
  26. G. Feigenbaum and I. J. Reist. Provenance: An alternate history of art. Getty Publications, 2012.
  27. ipoet: interactive painting poetry creation with visual multimodal analysis. Journal of Visualization, 25(3):671–685, 2022. doi: 10 . 1007/s12650-021-00780-0
  28. Promptmagician: Interactive prompt engineering for text-to-image creation. IEEE Transactions on Visualization and Computer Graphics, 2023. To appear. doi: 10 . 48550/arXiv . 2307 . 09036
  29. Xnli: Explaining and diagnosing nli-based visual data analysis. IEEE Transactions on Visualization and Computer Graphics, pp. 1–14, 2023. doi: 10 . 1109/TVCG . 2023 . 3240003
  30. W. C. Fong. Why chinese painting is history. The Art Bulletin, 85(2):258–280, 2003. doi: 10 . 1080/00043079 . 2003 . 10787072
  31. Chest: A linked open data-based application to annotate and carry out learning tasks about cultural heritage. In Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption, pp. 441–447. Springer, Cham, 2022. doi: 10 . 1007/978-3-031-16290-9_34
  32. YOLOX: Exceeding YOLO Series in 2021, Aug. 2021. doi: 10 . 48550/arXiv . 2107 . 08430
  33. J.-B. Grill et al. Bootstrap your own latent - a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
  34. Survey on visual analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics, 28(12):5091–5112, 2022. doi: 10 . 1109/TVCG . 2021 . 3100413
  35. Liberroad: Probing into the journey of chinese classics through visual analytics. IEEE Transactions on Visualization and Computer Graphics, pp. 1–11, 2023. doi: 10 . 1109/TVCG . 2023 . 3326944
  36. F. Haskell. History and its Images: Art and the Interpretation of the Past. Yale University Press, 1993. doi: 10 . 5860/choice . 31-1330
  37. Road extraction by using atrous spatial pyramid pooling integrated encoder-decoder network and structural similarity loss. Remote Sensing, 11(9):1015, 2019.
  38. Videopro: A visual analytics approach for interactive video programming. IEEE Transactions on Visualization and Computer Graphics, 2023. To appear. doi: 10 . 48550/arXiv . 2308 . 00401
  39. Deep residual learning for image recognition. In Proc. CVPR, June 2016. doi: 10 . 1109/cvpr . 2016 . 90
  40. Viewing chinese art on an interactive tabletop. IEEE Computer Graphics and Applications, 33(3):16–21, 2013. doi: 10 . 1109/MCG . 2013 . 50
  41. C.-C. Hung. A study on a content-based image retrieval technique for chinese paintings. The Electronic Library, 2018. doi: 10 . 1108/EL-10-2016-0219
  42. P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proc. STOC, pp. 604–613, 1998. doi: 10 . 1145/276698 . 276876
  43. Mtffnet: a multi-task feature fusion framework for chinese painting classification. Cognitive Computation, 13:1287–1296, 2021. doi: 10 . 1007/s12559-021-09896-9
  44. Dct–cnn-based classification method for the gongbi and xieyi techniques of chinese ink-wash paintings. Neurocomputing, 330:280–286, 2019. doi: 10 . 1016/j . neucom . 2018 . 11 . 003
  45. Reconstructing traditional chinese paintings with immersive virtual reality. In Proc. CHI EA, pp. 1–8. ACM, NY, USA, 2020. doi: 10 . 1145/3334480 . 3382934
  46. Anchorage: Visual analysis of satisfaction in customer service videos via anchor events. IEEE Transactions on Visualization and Computer Graphics, pp. 1–13, 2023. doi: 10 . 1109/TVCG . 2023 . 3245609
  47. Painting-91: a large scale database for computational painting categorization. Machine vision and applications, 25:1385–1397, 2014. doi: 10 . 1007/s00138-014-0621-6
  48. Sliding window approach for online energy disaggregation using artificial neural networks. In Proceedings of the 10th Hellenic Conference on Artificial Intelligence, pp. 1–6, 2018.
  49. Report on narrative visualization techniques for opdb data. Technical report, Technical report, InTaVia project, 2021.
  50. Understanding narrative linearity for telling expressive time-oriented stories. In Proc. CHI. ACM, NY, USA, 2021. doi: 10 . 1145/3411764 . 3445344
  51. A. Lengyel. Three thousand years of chinese painting. China Review International, 7(1):260–269, 2000. doi: 10 . 1353/cri . 2000 . 0035
  52. Pm-vis: A visual analytics system for tracing and analyzing the evolution of pottery motifs. IEEE Transactions on Visualization and Computer Graphics, 2024. To appear.
  53. Computing for chinese cultural heritage. Visual Informatics, 6(1):1–13, 2022.
  54. Taxthemis: Interactive mining and exploration of suspicious tax evasion groups. IEEE Transactions on Visualization and Computer Graphics, 27(2):849–859, 2021. doi: 10 . 1109/TVCG . 2020 . 3030370
  55. Automatic traditional chinese painting classification: A benchmarking analysis. Computational Intelligence, 36(3):1183–1199, 2020. doi: 10 . 1111/coin . 12328
  56. C. Liu and H. Jiang. Classification of traditional chinese paintings based on supervised learning methods. In Proc. ICSPCC, pp. 641–644, 2014. doi: 10 . 1109/ICSPCC . 2014 . 6986272
  57. The multiple faces of cultural heritage: Towards an integrated visualization platform for tangible and intangible cultural assets. In 2022 IEEE 7th Workshop on Visualization for the Digital Humanities (VIS4DH), pp. 13–18, 2022. doi: 10 . 1109/VIS4DH57440 . 2022 . 00008
  58. S. Mohammad and S. Kiritchenko. Wikiart emotions: An annotated dataset of emotions evoked by art. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018), 2018.
  59. N. P. Museum. National Palace Museum Open Data. https://theme.npm.edu.tw/opendata/.
  60. Celebrity contagion and the value of objects. Journal of consumer research, 38(2):215–228, 2011.
  61. A web gis-based integration of 3d digital models with linked open data for cultural heritage exploration. ISPRS International Journal of Geo-Information, 10(10), 2021. doi: 10 . 3390/ijgi10100684
  62. N. Otsu. A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics, 9(1):62–66, 1979. doi: 10 . 1109/tsmc . 1979 . 4310076
  63. P. Rodríguez-Gonzálvez et al. 4d reconstruction and visualization of cultural heritage: Analyzing our legacy through time. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42:609, 2017. doi: 10 . 5194/isprs-archives-xlii-2-w3-609-2017
  64. Diachronic reconstruction and visualization of lost cultural heritage sites. ISPRS International Journal of Geo-Information, 8(2), 2019. doi: 10 . 3390/ijgi8020061
  65. K. Roth et al. Japan encyclopedia. Harvard University Press, 2002.
  66. I. D. Rowland. The intellectual background of the School of Athens: Tracking divine wisdom in the Rome of Julius II. Cambridge University Press, 1997.
  67. Ruiz-Calleja et al. Casual learn: A linked data-based mobile application for learning about local cultural heritage. Semantic Web, 14(2):181–195, 2023. doi: 10 . 3233/SW-212907
  68. Identifying, exploring, and interpreting time series shapes in multivariate time intervals. Visual Informatics, 7(1):77–91, 2023. doi: 10 . 1016/j . visinf . 2023 . 01 . 001
  69. O. Sirén. The Chinese on the Art of Painting. Dover Publications, 2013.
  70. F. Tang et al. Animated construction of chinese brush paintings. IEEE Transactions on Visualization and Computer Graphics, 24(12):3019–3031, 2018. doi: 10 . 1109/TVCG . 2017 . 2774292
  71. Pcolorizor: Re-coloring ancient chinese paintings with ideorealm-congruent poems. In Proc. UIST. ACM, NY, USA, 2023. doi: 10 . 1145/3586183 . 3606814
  72. Towards an understanding of distributed asymmetric collaborative visualization on problem-solving. In Proc. VR, pp. 387–397. IEEE, Los Alamitos, 2023. doi: 10 . 1109/VR55154 . 2023 . 00054
  73. P. D. Turney. Learning algorithms for keyphrase extraction. Information retrieval, 2:303–336, 2000. doi: 10 . 1023/A:1009976227802
  74. Attentional wavelet network for traditional chinese painting transfer. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 3077–3083, 2021. doi: 10 . 1109/ICPR48806 . 2021 . 9413063
  75. Improving named entity recognition by external context retrieving and cooperative learning. In Proc. ACL-IJCNLP, pp. 1800–1812. ACL, Online, Aug. 2021. doi: 10 . 18653/v1/2021 . acl-long . 142
  76. Visualization of cultural heritage collection data: State of the art and future challenges. IEEE Transactions on Visualization and Computer Graphics, 25(6):2311–2330, 2019. doi: 10 . 1109/TVCG . 2018 . 2830759
  77. Explaining with examples lessons learned from crowdsourced introductory description of information visualizations. IEEE Transactions on Visualization and Computer Graphics, 29(3):1638–1650, 2023. doi: 10 . 1109/TVCG . 2021 . 3128157
  78. Puzzlefixer: A visual reassembly system for immersive fragments restoration. IEEE Transactions on Visualization and Computer Graphics, 29(1):429–439, 2023. doi: 10 . 1109/tvcg . 2022 . 3209388
  79. Detail-preserving cyclegan-adain framework for image-to-ink painting translation. IEEE Access, 8:132002–132011, 2020. doi: 10 . 1109/ACCESS . 2020 . 3009470
  80. Visual knowledge guided intelligent generation of chinese seal carving. Frontiers of Information Technology & Electronic Engineering, 23(10):1479–1493, 2022.
  81. Cohortva: A visual analytic system for interactive exploration of cohorts based on historical data. IEEE Transactions on Visualization and Computer Graphics, 29(1):756–766, 2023. doi: 10 . 1109/TVCG . 2022 . 3209483
  82. Visual reasoning for uncertainty in spatio-temporal events of historical figures. IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2022. doi: 10 . 1109/TVCG . 2022 . 3146508
  83. Computational approaches for traditional chinese painting: From the "six principles of painting" perspective. Journal of Computer Science and Technology, 2023. To appear. doi: 10 . 48550/arXiv . 2307 . 14227
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up