Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes (2305.11927v2)

Published 19 May 2023 in cs.HC, cs.CV, and cs.LG

Abstract: Creating Computer Vision (CV) models remains a complex practice, despite their ubiquity. Access to data, the requirement for ML expertise, and model opacity are just a few points of complexity that limit the ability of end-users to build, inspect, and improve these models. Interactive ML perspectives have helped address some of these issues by considering a teacher in the loop where planning, teaching, and evaluating tasks take place. We present and evaluate two interactive visualizations in the context of Sprite, a system for creating CV classification and detection models for images originating from videos. We study how these visualizations help Sprite's users identify (evaluate) and select (plan) images where a model is struggling and can lead to improved performance, compared to a baseline condition where users used a query language. We found that users who had used the visualizations found more images across a wider set of potential types of model errors.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. A. Adadi and M. Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138–52160, 2018.
  2. Power to the people: The role of humans in interactive machine learning. Ai Magazine, 35(4):105–120, 2014.
  3. Yolov4: Optimal speed and accuracy of object detection, 2020.
  4. J. Brooke. ”SUS-A quick and dirty usability scale.” Usability evaluation in industry. CRC Press, June 1996. ISBN: 9780748404605.
  5. Apolo: making sense of large network data by combining rich user interaction and machine learning. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 167–176, 2011.
  6. Diagnostic visualization for non-expert machine learning practitioners: A design study. In 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 87–95. IEEE, 2016.
  7. J. Choo and S. Liu. Visual analytics for explainable deep learning. IEEE computer graphics and applications, 38(4):84–92, 2018.
  8. Ganslider: How users control generative models for images using multiple sliders with and without feedforward information. In CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2022.
  9. A review of user interface design for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2):1–37, 2018.
  10. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent user interfaces, pp. 39–45, 2003.
  11. R. Fiebrink and P. R. Cook. The wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht), vol. 3, 2010.
  12. Vatld: a visual analytics system to assess, understand and improve traffic light detection. IEEE transactions on visualization and computer graphics, 27(2):261–271, 2020.
  13. S. G. Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, vol. 50, pp. 904–908. Sage publications Sage CA: Los Angeles, CA, 2006.
  14. Deep residual learning for image recognition, 2015.
  15. Understanding and visualizing data iteration in machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems, pp. 1–13, 2020.
  16. Interactive machine learning: experimental evidence for the human in the algorithmic loop. Applied Intelligence, 49(7):2401–2414, 2019.
  17. T. Kraska. Northstar: An interactive data science system. Proceedings of the VLDB Endowment, 11(12):2150–2164, 2018.
  18. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics, 1(1):48–56, 2017.
  19. Microsoft. Getting started with sql queries, 2021.
  20. Microsoft. What is azure custom vision?, 2021.
  21. Understanding and supporting knowledge decomposition for machine teaching. In ACM conference on Designing Interactive Systems (DIS), July 2020.
  22. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717–1724, 2014.
  23. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction, 35(5-6):413–451, 2020. doi: 10 . 1080/07370024 . 2020 . 1734931
  24. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics, 23(1):61–70, 2016.
  25. ” why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144, 2016.
  26. Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742, 2017.
  27. The oz of wizard: simulating the human for interaction research. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, pp. 101–108, 2009.
  28. Anchorviz: Facilitating semantic data exploration and concept discovery for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS), 10(1):1–38, 2019.
  29. A teaching language for building object detection models. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, pp. 1223–1234, 2020.
  30. Leveraging explanations in interactive machine learning: An overview. arXiv preprint arXiv:2207.14526, 2022.
  31. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 2022.
  32. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1792, 2014.
  33. YouTube. Hospital construction, 2021.
  34. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27–39, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets