Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Contextual Inquiry of People with Vision Impairments in Cooking (2402.15108v1)

Published 23 Feb 2024 in cs.HC

Abstract: Individuals with vision impairments employ a variety of strategies for object identification, such as pans or soy sauce, in the culinary process. In addition, they often rely on contextual details about objects, such as location, orientation, and current status, to autonomously execute cooking activities. To understand how people with vision impairments collect and use the contextual information of objects while cooking, we conducted a contextual inquiry study with 12 participants in their own kitchens. This research aims to analyze object interaction dynamics in culinary practices to enhance assistive vision technologies for visually impaired cooks. We outline eight different types of contextual information and the strategies that blind cooks currently use to access the information while preparing meals. Further, we discuss preferences for communicating contextual information about kitchen objects as well as considerations for the deployment of AI-powered assistive technologies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Karissa Bell. [n.d.]. Microsoft’s ’talking camera’ app for the blind isn’t as magical as it sounds — Mashable. https://mashable.com/article/microsoft-seeing-ai-app-for-blind. (Accessed on 09/10/2023).
  2. BeMyEyes. 2023. Introducing: Be My AI. https://www.bemyeyes.com/blog/introducing-be-my-ai. (Accessed on 11/22/2023).
  3. Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42.
  4. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 333–342.
  5. Food experiences and eating patterns of visually impaired and blind people. Canadian Journal of Dietetic practice and research 70, 1 (2009), 13–18.
  6. A smart kitchen for ambient assisted living. Sensors 14, 1 (2014), 1629–1653.
  7. Jullien Bouchet and Laurence Nigay. 2004. ICARE: a component-based approach for the design and development of multimodal interfaces. In CHI’04 extended abstracts on Human factors in computing systems. 1325–1328.
  8. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
  9. Recipescape: An interactive tool for analyzing cooking instructions at scale. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.
  10. Learning to detect human-object interactions. In 2018 ieee winter conference on applications of computer vision (wacv). IEEE, 381–389.
  11. A smart kitchen for nutrition-aware cooking. IEEE Pervasive Computing 9, 4 (2010), 58–65.
  12. Introducing ATMs in India: a contextual inquiry. Interacting with Computers 16, 1 (2004), 29–44.
  13. “I’m Stuck!”: A Contextual Inquiry of People with Visual Impairments in Authentication. In Eleventh Symposium On Usable Privacy and Security ({normal-{\{{SOUPS}normal-}\}} 2015). 151–168.
  14. Bengisu Dost. [n.d.]. How can we improve Microsoft’s “Seeing AI” application? — by Bengisu Dost — Medium. https://bengisudost.medium.com/how-can-we-improve-microsofts-seeing-ai-application-4cb4b074e0d6. (Accessed on 09/10/2023).
  15. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023).
  16. Thomas Erickson and David W McDonald. 2007. HCI Remixed: Reflections on Works That Have Influenced the HCI Community. Mit Press.
  17. Maurizio Ficocelli and Goldie Nejat. 2012. The design of an interactive assistive kitchen system. Assistive Technology 24, 4 (2012), 246–258.
  18. Using computer vision to access appliance displays. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility. 281–282.
  19. Physically Grounded Vision-Language Models for Robotic Manipulation. arXiv preprint arXiv:2309.02561 (2023).
  20. GoPro. 2023. GoPro HERO11 Black Action Camera (Waterproof + Stabilization). https://gopro.com/en/us/shop/cameras/hero11-black/CHDHX-111-master.html. (Accessed on 09/04/2023).
  21. Rúben Gouveia and Evangelos Karapanos. 2013. Footprint tracker: supporting diary studies with lifelogging. In Proceedings of the SIGCHI conference on human factors in computing systems. 2921–2930.
  22. Vizlens: A robust and interactive screen reader for interfaces in the real world. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 651–664.
  23. Facade: Auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5826–5838.
  24. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3608–3617.
  25. SenseCam: A wearable camera that stimulates and rehabilitates autobiographical memory. Memory 19, 7 (2011), 685–696.
  26. Karen Holtzblatt and Hugh Beyer. 1995. Contextual Design: using customer work models to drive system design. In Conference companion on human factors in computing systems. 391–392.
  27. An analysis of the impact of visual impairment on activities of daily living and vision-related quality of life in a visually impaired adult population. British Journal of Visual Impairment 37, 1 (2019), 50–63.
  28. Holtzblatt Karen and Jones Sandra. 2017. Contextual inquiry: A participatory technique for system design. In Participatory design. CRC Press, 177–210.
  29. Exploiting sensecam for helping the blind in business negotiations. In Computers Helping People with Special Needs: 10th International Conference, ICCHP 2006, Linz, Austria, July 11-13, 2006. Proceedings 10. Springer, 1147–1154.
  30. Face recognition assistant for people with visual impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–24.
  31. A study on the adoption of IoT smart home service: using Value-based Adoption Model. Total Quality Management & Business Excellence 28, 9-10 (2017), 1149–1165.
  32. Video analysis: Methodology and methods. Peter Lang Frankfurt am Main.
  33. A König and K Thongpull. 2015. Lab-on-Spoon–a 3-D integrated hand-held multi-sensor system for low-cost food quality, safety, and processing monitoring in assisted-living systems. Journal of Sensors and Sensor Systems 4, 1 (2015), 63–75.
  34. Food shopping, sensory determinants of food choice and meal preparation by visually impaired people. Obstacles and expectations in daily food experiences. Appetite 113 (2017), 14–22.
  35. Calculating Cooking Recipe’s Difficulty based on Cooking Activities. In Proceedings of the 9th Workshop on Multimedia for Cooking and Eating Activities in conjunction with The 2017 International Joint Conference on Artificial Intelligence. 19–24.
  36. Fine-grained kitchen activity recognition using rgb-d. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 208–211.
  37. FMT: A Wearable Camera-Based Object Tracking Memory Aid for Older Adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–25.
  38. “I Choose Assistive Devices That Save My Face” A Study on Perceptions of Accessibility and Assistive Technology Use Conducted in China. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
  39. Non-visual cooking: exploring practices and challenges of meal preparation by people with visual impairments. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. 1–11.
  40. “It Feels Like Taking a Gamble”: Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impairments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–15.
  41. Understanding Visual Arts Experiences of Blind People. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
  42. Michael Xieyang Liu. 2023. Tool Support for Knowledge Foraging, Structuring, and Transfer During Online Sensemaking. Ph.D. Dissertation. Ph. D. Dissertation. Carnegie Mellon University. http://reports-archive. adm ….
  43. Unakite: Scaffolding developers’ decision-making using the web. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 67–80.
  44. To reuse or not to reuse? A framework and system for evaluating summarized knowledge. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–35.
  45. Wigglite: Low-cost Information Collection and Triage. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–16.
  46. Understanding blind people’s experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5988–5999.
  47. Microsoft. 2023. Seeing AI App from Microsoft. https://www.microsoft.com/en-us/ai/seeing-ai. (Accessed on 09/03/2023).
  48. Clearspeech: A display reader for the visually handicapped. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14, 4 (2006), 492–500.
  49. Ambient kitchen: designing situated services using a high fidelity prototyping environment. In Proceedings of the 2nd international conference on pervasive technologies related to assistive environments. 1–7.
  50. The handbook of multimodal-multisensor interfaces, volume 1: Foundations, user modeling, and common modality combinations. Morgan & Claypool.
  51. Interaction Design: Beyond Human-Computer Interaction (4 ed.). Wiley, Hoboken, NJ.
  52. Mary Elizabeth Raven and Alicia Flanders. 1996. Using contextual inquiry to learn about your audiences. ACM SIGDOC Asterisk Journal of Computer Documentation 20, 1 (1996), 1–13.
  53. Guidelines for multimodal user interface design. Commun. ACM 47, 1 (2004), 57–59.
  54. Designing interactions for 3D printed models with blind people. In Proceedings of the 19th international acm sigaccess conference on computers and accessibility. 200–209.
  55. ” Person, Shoes, Tree. Is the Person Naked?” What People with Vision Impairments Want in Image Descriptions. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–13.
  56. Sebastian Stein and Stephen J McKenna. 2013. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 729–738.
  57. Fluxmarker: Enhancing tactile graphics with dynamic tactile markers. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 190–199.
  58. TapTapSee. 2023. TapTapSee - Blind and Visually Impaired Assistive Technology - powered by CloudSight.ai Image Recognition API. https://taptapseeapp.com/. (Accessed on 09/10/2023).
  59. Real-time detection and reading of LED/LCD displays for visually impaired persons. In 2011 IEEE Workshop on Applications of Computer Vision (WACV). IEEE, 491–496.
  60. Improving computer vision-based indoor wayfinding for blind persons with context information. In Computers Helping People with Special Needs: 12th International Conference, ICCHP 2010, Vienna, Austria, July14-16, 2010, Proceedings, Part II 12. Springer, 255–262.
  61. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress 11 (1994), 29–41.
  62. Trieu Minh Vu and Riva Khanna. 2018. Application of artificial intelligence in smart kitchen. International Journal of Innovative Technology and Interdisciplinary Sciences 1, 1 (2018), 1–8.
  63. Practices and Barriers of Cooking Training for Blind and Low Vision People. (2023).
  64. Design of a kitchen-monitoring and decision-making system to support AAL applications. Sensors 21, 13 (2021), 4449.
  65. CueSee: exploring visual cues for people with low vision to facilitate a visual search task. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 73–84.
  66. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16816–16825.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets