Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps (2307.00756v1)

Published 3 Jul 2023 in cs.HC and cs.AI

Abstract: With the emergence of deep learning techniques, smartphone apps are now embedded on-device AI features for enabling advanced tasks like speech translation, to attract users and increase market competitiveness. A good interaction design is important to make an AI feature usable and understandable. However, AI features have their unique challenges like sensitiveness to the input, dynamic behaviours and output uncertainty. Existing guidelines and tools either do not cover AI features or consider mobile apps which are confirmed by our informal interview with professional designers. To address these issues, we conducted the first empirical study to explore user-AI-interaction in mobile apps. We aim to understand the status of on-device AI usage by investigating 176 AI apps from 62,822 apps. We identified 255 AI features and summarised 759 implementations into three primary interaction pattern types. We further implemented our findings into a multi-faceted search-enabled gallery. The results of the user study demonstrate the usefulness of our findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Khalid Alharbi and Tom Yeh. 2015. Collect, decompile, extract, stats, and diff: Mining design pattern changes in Android apps. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 515–524.
  2. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13.
  3. Anonymous Author(s). 2022a. Github. Interaction design gallery. https://github.com/algaeSpace/interaction_gallery
  4. Anonymous Author(s). 2022b. Interaction design gallery. https://algaespace.github.io/interaction_gallery/Gallery_static/index_page/search.html
  5. Constantinos K Coursaris and Dan J Kim. 2011. A meta-analytical review of empirical mobile usability studies. Journal of usability studies 6, 3 (2011), 117–171.
  6. Understanding Real-world Threats to Deep Learning Models in Android Apps. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 785–799.
  7. Statista Research Department. 2022. Share of global smartphone shipments by operating system from 2014 to 2023. https://www.statista.com/statistics/272307/market-share-forecast-for-smartphone-operating-systems/#:~:text=Smartphones%20running%20the%20Android%20operating,percent%20share%20of%20the%20market.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  9. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things 2, 3 (2021), 1–49.
  10. UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the 2017 chi conference on human factors in computing systems. 278–288.
  11. Why people hate your app: Making sense of user feedback in a mobile app store. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1276–1284.
  12. Guidelines for handheld mobile device interface design. In Proceedings of DSI 2004 Annual Meeting. Citeseer, 3751–3756.
  13. Google. 2020. Machine learning patterns. https://m2.material.io/design/machine-learning/understanding-ml-patterns.html#
  14. Google. 2022a. Google - People + AI Research (PAIR). https://pair.withgoogle.com/
  15. Google. 2022b. Machine learning for mobile developers. https://developers.google.com/ml-kit
  16. Google. 2022c. Material Design. https://material.io
  17. Google. 2022d. TensorFlow Hub. https://tfhub.dev/
  18. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 (2020).
  19. Lars Erik Holmquist. 2017. Intelligence on tap: artificial intelligence as a new design material. interactions 24, 4 (2017), 28–33.
  20. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
  21. Yujin Huang and Chunyang Chen. 2022. Smart app attack: hacking deep learning models in android apps. IEEE Transactions on Information Forensics and Security 17 (2022), 1827–1840.
  22. Robustness of on-device models: Adversarial attack to deep learning models on android apps. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 101–110.
  23. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016).
  24. Apple Inc. 2022a. Core ML - Machine Learning. https://developer.apple.com/machine-learning/core-ml
  25. Apple Inc. 2022b. Designing for iOS. https://developer.apple.com/design/human-interface-guidelines/platforms/designing-for-ios/
  26. Apple Inc. 2022c. Human Interface Guidelines - Machine Learning. https://developer.apple.com/design/human-interface-guidelines/technologies/machine-learning/introduction/
  27. Mining Visual Evolution in 21 Years of Web Design. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2676–2682.
  28. Colors–Messengers of Concepts: Visual Design Mining for Learning Color Semantics. ACM Transactions on Computer-Human Interaction(TOCHI) 24, 1 (2017).
  29. Page Laubheimer. 2015. Preventing User Errors: Avoiding Unconscious Slips. https://www.nngroup.com/articles/slips/
  30. Deep learning. nature 521, 7553 (2015), 436–444.
  31. Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.
  32. Question-Driven Design Process for Explainable AI User Experiences. arXiv preprint arXiv:2104.03483 (2021).
  33. David G Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, Vol. 2. Ieee, 1150–1157.
  34. A survey of app store analysis for software engineering. IEEE transactions on software engineering 43, 9 (2017), 817–847.
  35. Aliaksei Miniukovich and Antonella De Angeli. 2015. Computation of interface aesthetics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1163–1172.
  36. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015).
  37. Maria Rosala. 2020. User Control and Freedom (Usability Heuristic). https://www.nngroup.com/articles/user-control-and-freedom/
  38. Juergen Sauer and Andreas Sonderegger. 2009. The influence of prototype fidelity and aesthetics of design in usability tests: Effects on user behaviour, subjective evaluation and emotion. Applied ergonomics 40, 4 (2009), 670–677.
  39. Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504.
  40. Statista. 2022. Biggest app stores in the world 2022 — Statista. https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/
  41. Mind your weight (s): A large-scale study on insufficient machine learning model protection in mobile apps. In 30th USENIX Security Symposium (USENIX Security 21). 1955–1972.
  42. Human-AI interaction: intermittent, continuous, and proactive. Interactions 28, 6 (2021), 67–71.
  43. A first look at deep learning apps on smartphones. In The World Wide Web Conference. 2125–2136.
  44. Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI. International Journal of Human–Computer Interaction (2022), 1–25.
  45. Qian Yang. 2017. The role of design in creating machine-learning-enhanced user experience. In 2017 AAAI spring symposium series.
  46. Qian Yang. 2020. Profiling Artificial Intelligence as a Material for User Experience Design. Ph. D. Dissertation. Carnegie Mellon University.
  47. Mapping machine learning advances from hci research to reveal starting places for design innovation. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–11.
  48. Investigating how experienced UX designers effectively work with machine learning. In Proceedings of the 2018 Designing Interactive Systems Conference. 585–596.
  49. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–13.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jason Ching Yuen Siu (1 paper)
  2. Jieshan Chen (23 papers)
  3. Yujin Huang (18 papers)
  4. Zhenchang Xing (99 papers)
  5. Chunyang Chen (86 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now