Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Algorithmic Ways of Seeing: Using Object Detection to Facilitate Art Exploration (2403.19174v1)

Published 28 Mar 2024 in cs.HC and cs.CV

Abstract: This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (96)
  1. 2018. SMK Open: Setting art free. https://www.smk.dk/en/article/smk-open/
  2. Object detection. Computer Vision: A Reference Guide (2020), 1–9. https://doi.org/10.1007/978-3-030-03243-2_660-1
  3. Sofian Audry. 2021. Art in the age of machine learning. MIT Press. https://doi.org/10.7551/mitpress/12832.001.0001
  4. Kevin Bacon. 2019. AI as provocation rather than solution. https://gifting.digital/brighton-museum/
  5. Matching words and pictures. The Journal of Machine Learning Research 3 (2003), 1107–1135.
  6. Sensitive Pictures: Emotional Interpretation in the Museum. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3491102.3502080 event-place: New Orleans, LA, USA.
  7. Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14. https://doi.org/10.1145/3411764.3445481
  8. Lennart Björneborn. 2017. Three key affordances for serendipity: Toward a framework connecting environmental and personal factors in serendipitous encounters. Journal of Documentation 73, 5 (13 Oct. 2017), 1053–1081. https://doi.org/10.1108/JD-07-2016-0097
  9. David M Blei and Michael I Jordan. 2003. Modeling annotated data. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval. 127–134.
  10. Margaret A. Boden and Ernest A. Edmonds. 2019. A Taxonomy of Computer Art. Chapter 2, 23–59. https://doi.org/10.7551/mitpress/8817.003.0005
  11. Ian Bogost. 2019. The AI-Art Gold Rush Is Here. https://www.theatlantic.com/technology/archive/2019/03/ai-created-art-invades-chelsea-gallery-scene/584134/ Section: Technology.
  12. Padraig Boulton and Peter Hall. 2019. Artistic Domain Generalisation Methods are Limited by their Deep Representations. arXiv:1907.12622 [cs] (July 2019). http://arxiv.org/abs/1907.12622 arXiv: 1907.12622.
  13. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
  14. Eva Cetinic and James She. 2022. Understanding and Creating Art with AI: Review and Outlook. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 18, 2, Article 66 (feb 2022), 22 pages. https://doi.org/10.1145/3475799
  15. Virtual reality usability and accessibility for cultural heritage practices: Challenges mapping and recommendations. Electronics 10, 12 (2021), 1430.
  16. Brendan Ciecko. 2017. Examining the impact of artificial intelligence in museums. Museums and the Web (2017). See, https://mw17.mwconf.org/paper/exploring-artificial-intelligence-in-museums.
  17. Brendan Ciecko. 2020. AI Sees What? The Good, the Bad, and the Ugly of Machine Vision for Museum Collections. In Museums and the Web 2020. Museums and the Web, Online. https://mw20.museweb.net/paper/ai-sees-what-the-good-the-bad-and-the-ugly-of-machine-vision-for-museum-collections/
  18. Navigating Context, Pathways and Relationships in Museum Collections using Formal Concept Analysis. International Journal for Digital Art History 4 (Dec. 2019), 5.13–5.27. https://doi.org/10.11588/dah.2019.4.72070
  19. Leendert D. Couprie. 1983. Iconclass: an iconographic classification system. Art Libraries Journal 8, 2 (1983), 32–49. https://doi.org/10.1017/S0307472200003436
  20. Kate Crawford and Trevor Paglen. 2019. Excavating AI: The Politics of Training Sets for Machine Learning. https://excavating.ai/
  21. Rossana Damiano. 2019. Investigating the Effectiveness of Narrative Relations for the Exploration of Cultural Heritage Archives: A Case Study on the Labyrinth system. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization. 417–423. https://doi.org/10.1145/3314183.3323870
  22. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys (Csur) 40, 2 (2008), 1–60.
  23. Drawing Apprentice: An Enactive Co-Creative Agent for Artistic Collaboration. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (Glasgow, United Kingdom) (C&C ’15). Association for Computing Machinery, New York, NY, USA, 185–186. https://doi.org/10.1145/2757226.2764555
  24. Samuel Dodge and Lina Karam. 2017. A study and comparison of human and deep learning recognition performance under visual distortions. In 2017 26th international conference on computer communication and networks (ICCCN). IEEE, 1–7. https://doi.org/10.1109/ICCCN.2017.8038465
  25. UX design innovation: Challenges for working with machine learning as a design material. Conference on Human Factors in Computing Systems - Proceedings 2017-May (2017), 278–288. https://doi.org/10.1145/3025453.3025739
  26. The Information Flaneur: A Fresh Look at Information Seeking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 1215–1224. https://doi.org/10.1145/1978942.1979124
  27. Ziv Epstein, Aaron Hertzmann, the Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, Robert Mahari, Alex “Sandy” Pentland, Olga Russakovsky, Hope Schroeder, and Amy Smith. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110–1111. https://doi.org/10.1126/science.adh4451 arXiv:https://www.science.org/doi/pdf/10.1126/science.adh4451
  28. John H. Falk and Lynn Diane Dierking. 2012. The Museum Experience Revisited. Routledge, London, England. https://doi.org/10.4324/9781315417851
  29. Critically Assessing AI/ML for Cultural Heritage: Potentials and Challenges. In Handbook of Critical Studies of Artificial Intelligence, Simon Lindgren (Ed.). Edward Elgar, Cheltenham.
  30. Ambiguity as a Resource for Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 233–240. https://doi.org/10.1145/642611.642653
  31. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR 2019). New Orleans, Louisiana, United States. https://doi.org/10.48550/arXiv.1811.12231
  32. T. Giannini and J. P. Bowen. 2019. Museums and Digital Culture. Springer, 3–48. https://doi.org/10.1007/978-3-319-97457-6
  33. Human-centred machine learning. In Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. 3558–3565. https://doi.org/10.1145/2851581.2856492
  34. Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision. 1440–1448. https://doi.org/10.1109/ICCV.2015.169
  35. Cross-depiction problem: Recognition and synthesis of photographs and artwork. Computational Visual Media 1, 2 (June 2015), 91–103. https://doi.org/10.1007/s41095-015-0017-1
  36. Rex Hartson and Pardha S. Pyla. 2019. Chapter 4 & Chapter 29. Morgan Kaufmann.
  37. HarvardArtMuseums. [n. d.]. About the AI Explorer. https://ai.harvardartmuseums.org/about Acessed March 8th 2023.
  38. Douglas Heaven. 2019. Why deep-learning AIs are so easy to fool. Nature 574 (Oct. 2019), 163–166. https://doi.org/10.1038/d41586-019-03013-5
  39. EMDialog: Bringing information visualization into the museum. IEEE transactions on visualization and computer graphics 14, 6 (2008), 1181–1188. https://doi.org/10.1109/TVCG.2008.127
  40. Eva Hornecker and Luigina Ciolfi. 2019. Human-Computer Interactions in Museums. Synthesis Lectures on Human-Centered Informatics 12, 2 (2019), i–171. https://doi.org/10.2200/S00901ED1V01Y201902HCI042 _eprint: https://doi.org/10.2200/S00901ED1V01Y201902HCI042.
  41. Ayanna Howard. 2020. Are we trusting AI too much? Examining human-robot interactions in the real world. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. 1–1. https://doi.org/10.1145/3319502.3374842
  42. The Close-up Cloud: Visualizing Details of Image Collections in Dynamic Overviews. International Journal for Digital Art History 5 (Dec. 2020). https://doi.org/10.11588/dah.2020.5.72039
  43. Improving Object Detection in Art Images Using Only Style Transfer. In The International Joint Conference on Neural Networks (IJCNN). IEEE, Virtual Event. https://doi.org/10.1109/IJCNN52387.2021.9534264 _eprint: 2102.06529.
  44. Bill Kules and Ben Shneiderman. 2008. Users can change their web search tactics: Design guidelines for categorized overviews. Information Processing & Management 44, 2 (2008), 463–484. https://doi.org/10.1016/j.ipm.2007.07.014
  45. Designing the user experience of machine learning systems. In AAAI Spring Symposium Proceedings (Technical Report SS-17-04). 27–29.
  46. Lucian Leahu. 2016. Ontological Surprises: A Relational Perspective on Machine Learning. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS ’16). Association for Computing Machinery, New York, NY, USA, 182–186. https://doi.org/10.1145/2901790.2901840
  47. Deep learning. nature 521, 7553 (2015), 436–444.
  48. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 12888–12900. https://proceedings.mlr.press/v162/li22n.html
  49. Grounded Language-Image Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10965–10975.
  50. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755. https://doi.org/10.1007/978-3-319-10602-1_48
  51. Exploring requirements for online art collections. Proceedings of the American Society for Information Science and Technology 50, 1 (2013), 1–4. https://doi.org/10.1002/meet.14505001109
  52. Designing for Interpersonal Museum Experiences. In Museums and the Challenge of Change: Old Institutions in a New World, Graham Black (Ed.). Routledge, London & New York, 223–238.
  53. Kristin MacDonough. 2018. Smartify. Multimedia & Technology Reviews (April 2018). https://doi.org/10.17613/95t4-2t63
  54. Interaction-driven design: A new approach for interactive product development. In Proceedings of the Designing Interactive Systems Conference. 448–457.
  55. Hypersocial Museum: addressing the social interaction challenge with museum scenarios and attention-based approaches. QPSR of the numediart research program 2 (01 2009), 91–96.
  56. Lev Manovich. 2020. Cultural Analytics. The MIT Press, Cambridge, MA. https://mitpress.mit.edu/books/cultural-analytics Publisher: The MIT Press.
  57. Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46. https://doi.org/10.1145/1121949.1121979
  58. Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 72 (nov 2019), 23 pages. https://doi.org/10.1145/3359174
  59. Christofer Meinecke. 2022. Labeling of Cultural Heritage Collections on the Intersection of Visual Analytics and Digital Humanities. In 2022 IEEE 7th Workshop on Visualization for the Digital Humanities (VIS4DH). 19–24. https://doi.org/10.1109/VIS4DH57440.2022.00009
  60. Pavol Navrat. 2012. Cognitive traveling in digital space: from keyword search through exploratory information seeking. Central European Journal of Computer Science 2 (2012), 170–182. https://doi.org/10.2478/s13537-012-0024-6
  61. Jonas Oppenlaender. 2022. The Creativity of Text-to-Image Generation. In Proceedings of the 25th International Academic Mindtrek Conference (Tampere, Finland) (Academic Mindtrek ’22). Association for Computing Machinery, New York, NY, USA, 192–202. https://doi.org/10.1145/3569219.3569352
  62. A Survey of Definitions and Models of Exploratory Search. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics (Limassol, Cyprus) (ESIDA ’17). Association for Computing Machinery, New York, NY, USA, 3–8. https://doi.org/10.1145/3038462.3038465
  63. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  64. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv:2204.06125 [cs.CV]
  65. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 8821–8831.
  66. Overtrust of robots in emergency evacuation scenarios. In 2016 11th ACM/IEEE international conference on human-robot interaction (HRI). IEEE, 101–108. https://doi.org/10.1109/HRI.2016.7451740
  67. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.
  68. Visual interface design for digital cultural heritage: A guide to rich-prospect browsing. Routledge.
  69. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252. https://doi.org/10.1007/s11263-015-0816-y
  70. LabelMe: a database and web-based tool for image annotation. International journal of computer vision 77 (2008), 157–173.
  71. Tony Russell-Rose and Tyler Tate. 2012. Designing the search experience: The information architecture of discovery. Newnes.
  72. Phoebe Sengers and Bill Gaver. 2006. Staying Open to Interpretation: Engaging Multiple Meanings in Design and Evaluation. In Proceedings of the 6th Conference on Designing Interactive Systems (University Park, PA, USA) (DIS ’06). Association for Computing Machinery, New York, NY, USA, 99–108. https://doi.org/10.1145/1142405.1142422
  73. Jonas Heide Smith. 2019. SMK’s collection search levels up. https://medium.com/smk-open/smks-collection-search-levels-up-cf8e967e9346
  74. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. PMLR, 2256–2265.
  75. Searching the Literature: An Analysis of an Exploratory Search Task. In ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22). ACM, Regensburg, Germany, 12 pages. https://doi.org/10.1145/3498366.3505818
  76. Semi-supervised Human Pose Estimation in Art-historical Images. In Proceedings of the 30th ACM International Conference on Multimedia. 1107–1116. https://doi.org/10.1145/3503161.3548371
  77. John Stack. 2020. Computer Vision and the Science Museum Group Collection. Science Museum Group Digital Lab. https://lab.sciencemuseum.org.uk/computer-vision-and-the-science-museum-group-collection-a6c20efb0ac9
  78. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation 23, 5 (2019), 828–841. https://doi.org/10.1109/TEVC.2019.2890858
  79. Loic Tallon. 2018. Creating Access beyond metmuseum. org: The Met Collection on Wikipedia. The Met Museum Blog, February 7 (2018). https://www.metmuseum.org/blogs/now-at-the-met/2018/open-access-at-the-met-year-one
  80. A Systematic Approach for Developing a Robust Artwork Recognition Framework Using Smartphone Cameras. Algorithms 15, 9 (2022). https://doi.org/10.3390/a15090305
  81. The bohemian bookshelf: supporting serendipitous book discoveries through information visualization. In Proceedings of the SIGCHI conference on human factors in computing systems. 1461–1470. https://doi.org/10.1145/2207676.2208607
  82. Small codes and large image databases for recognition. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.
  83. Elena Villaespesa. 2019. Museum collections and online users: development of a segmentation model for the metropolitan museum of art. Visitor Studies 22, 2 (2019), 233–252. https://doi.org/10.1080/10645578.2019.1668679
  84. Elena Villaespesa and Oonagh Murphy. 2021. This is not an apple! Benefits and challenges of applying computer vision to museum collections. Museum Management and Curatorship 36, 4 (2021), 362–383. https://doi.org/10.1080/09647775.2021.1873827
  85. Annika Waern and Anders Sundnes Løvlie (Eds.). 2022. Hybrid Museum Experiences. Amsterdam University Press, Amsterdam. https://www.aup.nl/en/book/9789048552849/hybrid-museum-experiences
  86. Detecting People in Artwork with CNNs. In Computer Vision – ECCV 2016 Workshops (Lecture Notes in Computer Science), Gang Hua and Hervé Jégou (Eds.). Springer International Publishing, 825–841. https://doi.org/10.1007/978-3-319-46604-0_57
  87. Ryen W. White and Resa A. Roth. 2009. Exploratory Search: Beyond the Query—Response Paradigm. Springer International Publishing. https://doi.org/10.1007/978-3-031-02260-9
  88. Mitchell Whitelaw. 2015. Generous interfaces for digital cultural collections. Digital Humanities Quarterly 9, 1 (2015). https://www.digitalhumanities.org/dhq/vol/9/1/000205/000205.html
  89. From keyword search to exploration: Designing future search interfaces for the web. Foundations and Trends® in Web Science 2, 1 (2010), 1–97. https://doi.org/10.1561/1800000003
  90. Visualization of Cultural Heritage Collection Data: State of the Art and Future Challenges. IEEE Transactions on Visualization and Computer Graphics 25, 6 (2019), 2311–2330. https://doi.org/10.1109/TVCG.2018.2830759
  91. Pathways through information landscapes: Alternative design criteria for digital art collections. International Conference on Information Systems (ICIS 2013): Reshaping Society Through Information Systems Design 3.
  92. Pathways through information landscapes: Alternative design criteria for digital art collections. In International Conference on Information Systems. Milan, Italy. https://www.researchgate.net/publication/259010843_Pathways_through_information_landscapes_Alternative_design_criteria_for_digital_art_collections
  93. Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376301
  94. Faceted Metadata for Image Search and Browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 401–408. https://doi.org/10.1145/642611.642681
  95. Research Through Design As a Method for Interaction Design Research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07). ACM, New York, NY, USA, 493–502. https://doi.org/10.1145/1240624.1240704
  96. Joanna Zylinska. 2020. AI Art: Machine Visions and Warped Dreams. Open Humanites Press. http://www.openhumanitiespress.org/books/titles/ai-art/
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets