Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AI Assistance for UX: A Literature Review Through Human-Centered AI (2402.06089v2)

Published 8 Feb 2024 in cs.HC

Abstract: Recent advancements in HCI and AI research attempt to support user experience (UX) practitioners with AI-enabled tools. Despite the potential of emerging models and new interaction mechanisms, mainstream adoption of such tools remains limited. We took the lens of Human-Centered AI and presented a systematic literature review of 359 papers, aiming to synthesize the current landscape, identify trends, and uncover UX practitioners' unmet needs in AI support. Guided by the Double Diamond design framework, our analysis uncovered that UX practitioners' unique focuses on empathy building and experiences across UI screens are often overlooked. Simplistic AI automation can obstruct the valuable empathy-building process. Furthermore, focusing solely on individual UI screens without considering interactions and user flows reduces the system's practical value for UX designers. Based on these findings, we call for a deeper understanding of UX mindsets and more designer-centric datasets and evaluation metrics, for HCI and AI communities to collaboratively work toward effective AI support for UX.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (266)
  1. User Experience Design Using Machine Learning: A Systematic Review. IEEE Access 10 (2022), 51501–51514. https://doi.org/10.1109/ACCESS.2022.3173289 Conference Name: IEEE Access.
  2. Eren Akça and Ömer Özgür Tanriöver. 2021. A comprehensive appraisal of perceptual visual complexity analysis methods in GUI design. Displays 69 (Sept. 2021), 102031. https://doi.org/10.1016/j.displa.2021.102031
  3. Khalid Alemerien and Kenneth Magel. 2014. GUIEvaluator: A Metric-tool for Evaluating the Complexity of Graphical User Interfaces.. In SEKE. 13–18.
  4. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233
  5. Design Concept: Get Comfortable Sleep Using Ambient Experience with Smart Pillow. In Asian CHI Symposium 2021 (Asian CHI Symposium 2021). Association for Computing Machinery, New York, NY, USA, 174–176. https://doi.org/10.1145/3429360.3468205
  6. Towards Automatic Persona Generation Using Social Media. In 2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW). IEEE, Vienna, Austria, 206–211. https://doi.org/10.1109/W-FiCloud.2016.51
  7. Imaginary People Representing Real Numbers: Generating Personas from Online Social Media Data. ACM Transactions on the Web 12, 4 (Nov. 2018), 1–26. https://doi.org/10.1145/3265986
  8. InspirationWall: Supporting Idea Generation Through Automatic Information Exploration. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition. ACM, Glasgow United Kingdom, 103–106. https://doi.org/10.1145/2757226.2757252
  9. Gary Ang and Ee-Peng Lim. 2022. Learning Semantically Rich Network-based Multi-modal Mobile User Interface Embeddings. ACM Transactions on Interactive Intelligent Systems 12, 4 (Dec. 2022), 1–29. https://doi.org/10.1145/3533856
  10. Variational Transformer Networks for Layout Generation. https://doi.org/10.48550/arXiv.2104.02416 arXiv:2104.02416 [cs].
  11. UIBert: Learning Generic Multimodal Representations for UI Understanding. http://arxiv.org/abs/2107.13731 arXiv:2107.13731 [cs].
  12. Anna Baj-Rogowska and Marcin Sikorski. 2023. Exploring the usability and user experience of social media apps through a text mining approach. Engineering Management in Production and Services 15, 1 (March 2023), 86–105. https://doi.org/10.2478/emj-2023-0007
  13. Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. https://doi.org/10.48550/arXiv.1705.07962 arXiv:1705.07962 [cs].
  14. Eva Bittner and Omid Shoury. 2019. Designing Automated Facilitation for Design Thinking: A Chatbot for Supporting Teams in the Empathy Map Method. https://doi.org/10.24251/HICSS.2019.029
  15. Parzival Borlinghaus and Stephan Huber. 2021. Comparing Apples and Oranges: Human and Computer Clustered Affinity Diagrams Under the Microscope. In 26th International Conference on Intelligent User Interfaces. ACM, College Station TX USA, 413–422. https://doi.org/10.1145/3397481.3450674
  16. Evaluating a Large Language Model on Searching for GUI Layouts. Proceedings of the ACM on Human-Computer Interaction 7, EICS (June 2023), 178:1–178:37. https://doi.org/10.1145/3593230
  17. Learning GUI Completions with User-defined Constraints. ACM Transactions on Interactive Intelligent Systems 12, 1 (March 2022), 6:1–6:40. https://doi.org/10.1145/3490034
  18. Denis Bulygin. 2022. How do Conversational Agents Transform Qualitative Interviews? Exploration and Support of Researchers’ Needs in Interviews at Scale. In 27th International Conference on Intelligent User Interfaces. ACM, Helsinki Finland, 124–128. https://doi.org/10.1145/3490100.3516478
  19. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. https://doi.org/10.1145/3411764.3445762
  20. Paper2Wire: a case study of user-centred development of machine learning tools for UX designers. In Proceedings of Mensch und Computer 2020. ACM, Magdeburg Germany, 33–41. https://doi.org/10.1145/3404983.3405506
  21. Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 57–69. https://doi.org/10.1145/3126594.3126653 arXiv:1708.02660 [cs].
  22. UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (IUI ’17). Association for Computing Machinery, New York, NY, USA, 245–255. https://doi.org/10.1145/3025171.3025190
  23. Wireframe-based UI Design Search through Image Autoencoder. ACM Transactions on Software Engineering and Methodology 29, 3 (July 2020), 1–31. https://doi.org/10.1145/3391613
  24. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. https://doi.org/10.1145/3377811.3380327 arXiv:2003.00380 [cs].
  25. Towards Complete Icon Labeling in Mobile Applications. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–14. https://doi.org/10.1145/3491102.3502073
  26. Using Machine Learning to Support Qualitative Coding in Social Science: Shifting the Focus to Ambiguity. ACM Transactions on Interactive Intelligent Systems 8, 2 (June 2018), 1–20. https://doi.org/10.1145/3185515
  27. Learning From Unique Perspectives: User-Aware Saliency Modeling. 2701–2710. https://openaccess.thecvf.com/content/CVPR2023/html/Chen_Learning_From_Unique_Perspectives_User-Aware_Saliency_Modeling_CVPR_2023_paper.html
  28. PLay: Parametrically Conditioned Layout Generation using Latent Diffusion. https://doi.org/10.48550/arXiv.2301.11529 arXiv:2301.11529 [cs].
  29. Mingming Cheng. 2016. Sharing economy: A review and agenda for future research. International Journal of Hospitality Management 57 (Aug. 2016), 60–70. https://doi.org/10.1016/j.ijhm.2016.06.003
  30. Visual saliency model based on crowdsourcing eye tracking data and its application in visual design. Personal and Ubiquitous Computing 27, 3 (June 2023), 613–630. https://doi.org/10.1007/s00779-020-01463-7
  31. ML for UX? - An Inventory and Predictions on the Use of Machine Learning Techniques for UX Research. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (NordiCHI ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3419249.3420163
  32. Elizabeth F Churchill. 2019. Scaling UX with design systems. Interactions 26, 5 (2019), 22–23.
  33. Deep Reinforcement Learning based Android Application GUI Testing. In Brazilian Symposium on Software Engineering. ACM, Joinville Brazil, 186–194. https://doi.org/10.1145/3474624.3474634
  34. Translational Resources: Reducing the Gap Between Academic Research and HCI Practice. In Proceedings of the 2017 Conference on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association for Computing Machinery, New York, NY, USA, 957–968. https://doi.org/10.1145/3064663.3064667
  35. A translational science model for HCI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
  36. Lucas Franco Colusso. 2020. Understanding and Tooling Translational Research in Human-Computer Interaction. Ph. D. Dissertation. University of Washington.
  37. British Design Council. [n. d.]. The Double Diamond - Design Council. https://www.designcouncil.org.uk/our-resources/the-double-diamond/
  38. Automated assessment of visual aesthetics of Android user interfaces with deep learning. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (IHC ’22). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3554364.3559113
  39. Learning user interface element interactions. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Beijing China, 296–306. https://doi.org/10.1145/3293882.3330569
  40. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 845–854. https://doi.org/10.1145/3126594.3126651
  41. ZIPT: Zero-Integration Performance Testing of Mobile App Designs. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 727–736. https://doi.org/10.1145/3126594.3126647
  42. Nicola Dell and Neha Kumar. 2016. The Ins and Outs of HCI for Development. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2220–2232. https://doi.org/10.1145/2858036.2858081
  43. Detecting Emotions Through Machine Learning for Automatic UX Evaluation. In Human-Computer Interaction – INTERACT 2021 (Lecture Notes in Computer Science), Carmelo Ardito, Rosa Lanzilotti, Alessio Malizia, Helen Petrie, Antonio Piccinno, Giuseppe Desolda, and Kori Inkpen (Eds.). Springer International Publishing, Cham, 270–279. https://doi.org/10.1007/978-3-030-85613-7_19
  44. Aparna Dhinakaran. [n. d.]. Survey: Massive Retooling Around Large Language Models Underway. https://www.forbes.com/sites/aparnadhinakaran/2023/04/26/survey-massive-retooling-around-large-language-models-underway/ Section: AI.
  45. The Idea Machine: LLM-based Expansion, Rewriting, Combination, and Suggestion of Ideas. In Creativity and Cognition. ACM, Venice Italy, 623–627. https://doi.org/10.1145/3527927.3535197
  46. The Sharing Economy in Computing: A Systematic Literature Review. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (Dec. 2017), 1–26. https://doi.org/10.1145/3134673
  47. Webthetics: Quantifying webpage aesthetics with deep learning. International Journal of Human-Computer Studies 124 (April 2019), 56–66. https://doi.org/10.1016/j.ijhcs.2018.11.006
  48. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction 17, 4 (Dec. 2011), 18:1–18:24. https://doi.org/10.1145/1879831.1879836
  49. Aeonium: Visual analytics to support collaborative qualitative coding. In 2017 IEEE Pacific Visualization Symposium (PacificVis). IEEE, Seoul, South Korea, 220–229. https://doi.org/10.1109/PACIFICVIS.2017.8031598
  50. Optimizing User Interface Layouts via Gradient Descent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376589
  51. Analysing app reviews for software engineering: a systematic literature review. Empirical Software Engineering 27, 2 (March 2022), 43. https://doi.org/10.1007/s10664-021-10065-7
  52. Juha Eskonen. 2019. Deep Reinforcement Learning in Automated User Interface Testing. (May 2019). https://aaltodoc.aalto.fi:443/handle/123456789/37895 Accepted: 2019-05-12T15:06:20Z.
  53. Automating GUI Testing with Image-Based Deep Reinforcement Learning. In 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). 160–167. https://doi.org/10.1109/ACSOS49614.2020.00038
  54. Automatic Detection of Usability Problem Encounters in Think-aloud Sessions. ACM Transactions on Interactive Intelligent Systems 10, 2 (June 2020), 1–24. https://doi.org/10.1145/3385732
  55. Human-ai collaboration for ux evaluation: Effects of explanation and synchronization. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–32.
  56. Susan Farrell. 2017. UX Research Cheat Sheet. https://www.nngroup.com/articles/ux-research-cheat-sheet/
  57. Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery. http://arxiv.org/abs/2204.06700 arXiv:2204.06700 [cs].
  58. LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. https://doi.org/10.48550/arXiv.2305.15393 arXiv:2305.15393 [cs].
  59. GUIS2Code: A Computer Vision Tool to Generate Code Automatically from Graphical User Interface Sketches. In Artificial Neural Networks and Machine Learning – ICANN 2021 (Lecture Notes in Computer Science), Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (Eds.). Springer International Publishing, Cham, 53–65. https://doi.org/10.1007/978-3-030-86365-4_5
  60. SmartColor: Automatic Web Color Scheme Generation Based on Deep Learning. In 2021 12th International Conference on Information and Communication Systems (ICICS). 285–290. https://doi.org/10.1109/ICICS52457.2021.9464536 ISSN: 2573-3346.
  61. Jessica L. Feuston and Jed R. Brubaker. 2021. Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (Oct. 2021), 1–25. https://doi.org/10.1145/3479856
  62. Automated Usability Tests for Mobile Devices through Live Emotions Logging. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 636–643. https://doi.org/10.1145/2786567.2792902
  63. Predicting Visual Importance Across Graphic Design Types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA, 249–260. https://doi.org/10.1145/3379337.3415825
  64. Brad Frost. 2016. Atomic Design. http://atomicdesign.bradfrost.com/
  65. Instruction-Finetuned Foundation Models for Multimodal Web Navigation. (2023).
  66. CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis. https://doi.org/10.48550/arXiv.2304.07366 arXiv:2304.07366 [cs].
  67. Simret Araya Gebreegziabher. 2023. PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis. (2023).
  68. UX Heatmaps: Mapping User Experience on Visual Interfaces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose California USA, 4850–4860. https://doi.org/10.1145/2858036.2858271
  69. Analogy Mining for Specific Design Needs. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–11. https://doi.org/10.1145/3173574.3173695
  70. QuAD: Deep-Learning Assisted Qualitative Data Analysis with Affinity Diagrams. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. ACM, New Orleans LA USA, 1–7. https://doi.org/10.1145/3491101.3519863
  71. Google. [n. d.]. Motion – Material Design 3. https://m3.material.io/styles/motion/overview
  72. Colin M. Gray. 2016. ”It’s More of a Mindset Than a Method”: UX Practitioners’ Conception of Design Methods. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4044–4055. https://doi.org/10.1145/2858036.2858410
  73. AI in User Interface Design and Evaluation. IT Professional 25, 2 (March 2023), 20–22. https://doi.org/10.1109/MITP.2023.3267139 Conference Name: IT Professional.
  74. LayoutTransformer: Layout Generation and Completion with Self-attention. https://doi.org/10.48550/arXiv.2006.14615 arXiv:2006.14615 [cs].
  75. Daniel Gustafsson. 2019. Analysing the Double diamond design process through research & implementation. (2019). https://aaltodoc.aalto.fi:443/handle/123456789/39285 Accepted: 2019-07-14T17:02:11Z.
  76. Emitza Guzman and Walid Maalej. 2014. How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews. In 2014 IEEE 22nd International Requirements Engineering Conference (RE). IEEE, Karlskrona, Sweden, 153–162. https://doi.org/10.1109/RE.2014.6912257
  77. Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–15. https://doi.org/10.1145/3411764.3445569
  78. AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1958–1970. https://doi.org/10.1109/ICSE48619.2023.00166 ISSN: 1558-1225.
  79. ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces. http://arxiv.org/abs/2012.12350 arXiv:2012.12350 [cs].
  80. Steffen Hedegaard and Jakob Grue Simonsen. 2013. Extracting usability and user experience information from online user reviews. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 2089–2098. https://doi.org/10.1145/2470654.2481286
  81. Steffen Hedegaard and Jakob Grue Simonsen. 2014. Mining until it hurts: automatic extraction of usability issues from online reviews compared to traditional usability evaluation. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI ’14). Association for Computing Machinery, New York, NY, USA, 157–166. https://doi.org/10.1145/2639189.2639211
  82. CoColor: Interactive Exploration of Color Designs. In Proceedings of the 28th International Conference on Intelligent User Interfaces. ACM, Sydney NSW Australia, 106–127. https://doi.org/10.1145/3581641.3584089
  83. Getting inspired! understanding how and why examples are used in creative design practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). Association for Computing Machinery, New York, NY, USA, 87–96. https://doi.org/10.1145/1518701.1518717
  84. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. https://doi.org/10.1145/302979.303030
  85. Graph Neural Networks for Nomination and Representation Learning of Web Elements. http://arxiv.org/abs/2111.02168 arXiv:2111.02168 [cs].
  86. A Method for Quickly Establishing Personas. In Artificial Intelligence in HCI, Helmut Degen and Lauren Reinerman-Jones (Eds.). Vol. 12217. Springer International Publishing, Cham, 16–32. https://doi.org/10.1007/978-3-030-50334-5_2 Series Title: Lecture Notes in Computer Science.
  87. Swire: Sketch-based User Interface Retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300334
  88. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. https://doi.org/10.48550/arXiv.2110.07775 arXiv:2110.07775 [cs].
  89. Seth Hunter and Pattie Maes. [n. d.]. WordPlay: A Table-Top Interface for Collaborative Brainstorming and Decision Making. ([n. d.]).
  90. Angel Hsing-Chi Hwang and Andrea Stevenson Won. 2021. IdeaBot: Investigating Social Facilitation in Human-Machine Team Creativity. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–16. https://doi.org/10.1145/3411764.3445270
  91. Taichi Iki and Akiko Aizawa. 2022. Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTs. https://doi.org/10.48550/arXiv.2203.07828 arXiv:2203.07828 [cs].
  92. Evaluation of mobile interfaces as an optimization problem. Procedia computer science 112 (2017), 235–248.
  93. LayoutDM: Discrete Diffusion Model for Controllable Layout Generation. https://doi.org/10.48550/arXiv.2303.08137 arXiv:2303.08137 [cs].
  94. AI-Enabled Design Tools: Current Trends and Future Possibilities. In [ ] With Design: Reinventing Design Modes, Gerhard Bruyns and Huaxin Wei (Eds.). Springer Nature, Singapore, 2836–2847. https://doi.org/10.1007/978-981-19-4472-7_183
  95. Jincheul Jang and Mun Yong Yi. 2017. Modeling User Satisfaction from the Extraction of User Experience Elements in Online Product Reviews. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, Denver Colorado USA, 1718–1725. https://doi.org/10.1145/3027063.3053097
  96. Yeonju Jang and Eunil Park. 2022. Satisfied or not: user experience of mobile augmented reality in using natural language processing techniques on review comments. Virtual Reality 26, 3 (Sept. 2022), 839–848. https://doi.org/10.1007/s10055-021-00599-y
  97. Creating Manageable Persona Sets from Large User Populations. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–6. https://doi.org/10.1145/3290607.3313006
  98. Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (April 2021), 1–23. https://doi.org/10.1145/3449168 arXiv:2102.03702 [cs].
  99. LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction. https://doi.org/10.48550/arXiv.2208.08037 arXiv:2208.08037 [cs].
  100. Layout Generation for Various Scenarios in Mobile Shopping Applications. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3544548.3581446
  101. Automatic Generation of Personas Using YouTube Social Media Data. Proceedings of the 50th Hawaii International Conference on System Sciences (HICSS) (2017).
  102. Learning to predict where humans look. In 2009 IEEE 12th International Conference on Computer Vision. 2106–2113. https://doi.org/10.1109/ICCV.2009.5459462 ISSN: 2380-7504.
  103. Two-handed Design: Development of Food Personality Framework Using Mixed Method Needfinding. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. ACM, New Orleans LA USA, 1–9. https://doi.org/10.1145/3491101.3503554
  104. Thisaranie Kaluarachchi and Manjusri Wickramasinghe. 2023. A systematic literature review on automatic website generation. Journal of Computer Languages 75 (June 2023), 101202. https://doi.org/10.1016/j.cola.2023.101202
  105. Creative sketching partner: an analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI ’20). Association for Computing Machinery, New York, NY, USA, 221–230. https://doi.org/10.1145/3377325.3377522
  106. Miki Kaukanen. 2020. Evaluating the impacts of machine learning to the future of A/B testing. (2020). https://lutpub.lut.fi/handle/10024/161780 Accepted: 2020-10-30T09:10:47Z.
  107. Learning Sensitive Combinations of A/B Test Metrics. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (WSDM ’17). Association for Computing Machinery, New York, NY, USA, 651–659. https://doi.org/10.1145/3018661.3018708
  108. Constrained Graphic Layout Generation via Latent Optimization. In Proceedings of the 29th ACM International Conference on Multimedia. 88–96. https://doi.org/10.1145/3474085.3475497 arXiv:2108.00871 [cs].
  109. Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models. (2023).
  110. Stylette: Styling the Web with Natural Language. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3491102.3501931
  111. Yui Kita and Jun Rekimoto. 2018. V8 Storming: How Far Should Two Ideas Be?. In Proceedings of the 9th Augmented Human International Conference. ACM, Seoul Republic of Korea, 1–8. https://doi.org/10.1145/3174910.3174937
  112. Exploring the future of design tooling: The role of artificial intelligence in tools for user experience professionals. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3544549.3573874
  113. May AI? Design Ideation with Cooperative Contextual Bandits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300863
  114. Aesthetics++: Refining Graphic Designs by Exploring Design Principles and Human Preference. IEEE Transactions on Visualization and Computer Graphics 29, 6 (June 2023), 3093–3104. https://doi.org/10.1109/TVCG.2022.3151617 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
  115. Using Random Ordering in User Experience Testing to Predict Final User Satisfaction. Informatics 9, 4 (Dec. 2022), 85. https://doi.org/10.3390/informatics9040085 Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.
  116. Kitti Koonsanit and Nobuyuki Nishiuchi. 2021. Predicting Final User Satisfaction Using Momentary UX Data and Machine Learning Techniques. Journal of Theoretical and Applied Electronic Commerce Research 16, 7 (Dec. 2021), 3136–3156. https://doi.org/10.3390/jtaer16070171 Number: 7 Publisher: Multidisciplinary Digital Publishing Institute.
  117. Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4627–4639. https://doi.org/10.1145/3025453.3025883
  118. DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations. IEEE Transactions on Image Processing 26, 9 (Sept. 2017), 4446–4456. https://doi.org/10.1109/TIP.2017.2710620 Conference Name: IEEE Transactions on Image Processing.
  119. Dynamic User Interface Composition. SN Computer Science 4, 3 (March 2023), 259. https://doi.org/10.1007/s42979-023-01672-w
  120. Webzeitgeist: design mining the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 3083–3092. https://doi.org/10.1145/2470654.2466420
  121. James A. Landay. 1996. SILK: sketching interfaces like krazy. In Conference companion on Human factors in computing systems common ground - CHI ’96. ACM Press, Vancouver, British Columbia, Canada, 398–399. https://doi.org/10.1145/257089.257396
  122. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, Oldenburg Germany, 1–4. https://doi.org/10.1145/3406324.3410710
  123. Describing UI Screenshots in Natural Language. ACM Transactions on Intelligent Systems and Technology (Sept. 2022). https://doi.org/10.1145/3564702 Just Accepted.
  124. Modeling how different user groups perceive webpage aesthetics. Universal Access in the Information Society (Aug. 2022). https://doi.org/10.1007/s10209-022-00910-x
  125. Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–13. https://doi.org/10.1145/3491102.3502042
  126. Gang Li and Yang Li. 2023. Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus. https://doi.org/10.48550/arXiv.2209.14927 arXiv:2209.14927 [cs].
  127. Webpage saliency prediction with multi-features fusion. In 2016 IEEE International Conference on Image Processing (ICIP). 674–678. https://doi.org/10.1109/ICIP.2016.7532442 ISSN: 2381-8549.
  128. Understanding Nonlinear Collaboration between Human and AI Agents: A Co-design Framework for Creative Design. arXiv preprint arXiv:2401.07312 (2024).
  129. LayoutGAN: Synthesizing Graphic Layouts With Vector-Wireframe Adversarial Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 7 (July 2021), 2388–2399. https://doi.org/10.1109/TPAMI.2019.2963663 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  130. A Zero-Shot Language Agent for Computer Control with Structured Reflection. arXiv preprint arXiv:2310.08740 (2023).
  131. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver Colorado USA, 6038–6049. https://doi.org/10.1145/3025453.3025483
  132. Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Online, 215–223. https://doi.org/10.18653/v1/2020.acl-demos.25
  133. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–15. https://doi.org/10.1145/3411764.3445049
  134. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New Orleans LA USA, 577–589. https://doi.org/10.1145/3332165.3347899
  135. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. http://arxiv.org/abs/2010.04295 arXiv:2010.04295 [cs].
  136. VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling. http://arxiv.org/abs/2112.05692 arXiv:2112.05692 [cs].
  137. Towards Measuring and Inferring User Interest from Gaze. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW ’17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 525–533. https://doi.org/10.1145/3041021.3054182
  138. Zhoufan Li. 2021. Qualitative Coding in the Computational Era: A Hybrid Approach to Improve Reliability and Reduce Effort for Coding Ethnographic Interviews - Google Search. https://www.google.com/search?q=Qualitative+Coding+in+the+Computational+Era%3A+A+Hybrid+Approach+to+Improve+Reliability+and+Reduce+Effort+for+Coding+Ethnographic+Interviews&sourceid=chrome&ie=UTF-8
  139. Extraction of affective responses from customer reviews: an opinion mining and machine learning approach. International Journal of Computer Integrated Manufacturing 33, 7 (July 2020), 670–685. https://doi.org/10.1080/0951192X.2019.1571240
  140. A framework of artificial intelligence augmented design support. Human–Computer Interaction 35, 5-6 (Nov. 2020), 511–544. https://doi.org/10.1080/07370024.2020.1733576 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/07370024.2020.1733576.
  141. Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience. http://arxiv.org/abs/2302.10395 arXiv:2302.10395 [cs].
  142. Adriano Luiz de Souza Lima and Christiane Gresse von Wangenheim. 2022. Assessing the Visual Esthetics of User Interfaces: A Ten-Year Systematic Mapping. International Journal of Human–Computer Interaction 38, 2 (Jan. 2022), 144–164. https://doi.org/10.1080/10447318.2021.1926118 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447318.2021.1926118.
  143. ButtonTips: Design Web Buttons with Suggestions. In 2019 IEEE International Conference on Multimedia and Expo (ICME). 466–471. https://doi.org/10.1109/ICME.2019.00087 ISSN: 1945-788X.
  144. Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration. http://arxiv.org/abs/1802.08802 arXiv:1802.08802 [cs].
  145. David Llàcer Giner. 2020. A Deep Learning Based Approach to Automated App Testing. Master’s thesis. Universitat Politècnica de Catalunya. https://upcommons.upc.edu/handle/2117/335561 Accepted: 2021-01-19T14:22:53Z.
  146. Gustavo Lopez and Luis A. Guerrero. 2017. Awareness Supporting Technologies used in Collaborative Systems: A Systematic Literature Review. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 808–820. https://doi.org/10.1145/2998181.2998281
  147. Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3491101.3519809
  148. Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do, but what AI should do: Towards a framework of task delegability. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/hash/d67d8ab4f4c10bf22aa353e27879133c-Abstract.html
  149. Daniel Peña López. [n. d.]. Triggering ideas with Generative AI. ([n. d.]).
  150. AI as mediator between composers, sound designers, and creative media producers. http://arxiv.org/abs/2303.01457 arXiv:2303.01457 [cs].
  151. On the automatic classification of app reviews. Requirements Engineering 21, 3 (Sept. 2016), 311–331. https://doi.org/10.1007/s00766-016-0251-9
  152. Reimagining Application User Interface (UI) Design using Deep Learning Methods: Challenges and Opportunities. http://arxiv.org/abs/2303.13055 arXiv:2303.13055 [cs].
  153. Magic Layouts: Structural Prior for Component Detection in User Interface Designs. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Nashville, TN, USA, 15804–15813. https://doi.org/10.1109/CVPR46437.2021.01555
  154. Megh Marathe and Kentaro Toyama. 2018. Semi-Automated Coding for Qualitative Research: A User-Centered Inquiry and Initial Prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–12. https://doi.org/10.1145/3173574.3173922
  155. How examples may (and may not) constrain creativity. Memory & Cognition 24, 5 (Sept. 1996), 669–680. https://doi.org/10.3758/BF03201091
  156. Nikolas Martelaro and Wendy Ju. 2017. The Needfinding Machine. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’17). Association for Computing Machinery, New York, NY, USA, 355–356. https://doi.org/10.1145/3029798.3034811
  157. Lucas Memmert and Navid Tavanapour. 2023. Towards Human-AI-Collaboration in Brainstorming: Empirical Insights into the Perception of Working with a Generative AI. (2023).
  158. Investigating Usability and User Experience from the User Postings in Social Systems. In Social Computing and Social Media, Gabriele Meiselwitz (Ed.). Vol. 9182. Springer International Publishing, Cham, 216–228. https://doi.org/10.1007/978-3-319-20367-6_22 Series Title: Lecture Notes in Computer Science.
  159. Marília Soares Mendes and Elizabeth Sucupira Furtado. 2017. UUX-Posts: a tool for extracting and classifying postings related to the use of a system. In Proceedings of the 8th Latin American Conference on Human-Computer Interaction. ACM, Antigua Guatemala Guatemala, 1–8. https://doi.org/10.1145/3151470.3151471
  160. HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale. In Proceedings of the ACM Web Conference 2023. ACM, Austin TX USA, 3837–3848. https://doi.org/10.1145/3543507.3583496
  161. Aliaksei Miniukovich and Antonella De Angeli. 2015. Computation of Interface Aesthetics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). Association for Computing Machinery, New York, NY, USA, 1163–1172. https://doi.org/10.1145/2702123.2702575
  162. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339, jul21 1 (July 2009), b2535–b2535. https://doi.org/10.1136/bmj.b2535
  163. Soumik Mohian and Christoph Csallner. 2022. PSDoodle: fast app screen search via partial screen doodle. In Proceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft ’22). Association for Computing Machinery, New York, NY, USA, 89–99. https://doi.org/10.1145/3524613.3527816
  164. Soumik Mohian and Christoph Csallner. 2023. Searching Mobile App Screens via Text + Doodle. https://doi.org/10.48550/arXiv.2305.06165 arXiv:2305.06165 [cs].
  165. fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–19. https://doi.org/10.1145/3544548.3581242
  166. Kate Moran. 2018. Quantitative User-Research Methodologies: An Overview. https://www.nngroup.com/articles/quantitative-user-research-methods/
  167. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (Feb. 2020), 196–221. https://doi.org/10.1109/TSE.2018.2844788 Conference Name: IEEE Transactions on Software Engineering.
  168. GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network. In CHI Conference on Human Factors in Computing Systems. 1–15. https://doi.org/10.1145/3491102.3517511 arXiv:2203.03827 [cs].
  169. Siva Natarajan and Christoph Csallner. 2018. P2A: a tool for converting pixels to animated mobile application user interfaces. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft ’18). Association for Computing Machinery, New York, NY, USA, 224–235. https://doi.org/10.1145/3197231.3197249
  170. Jakob Nielsen. 2017. A 100-Year View of User Experience (by Jakob Nielsen). https://www.nngroup.com/articles/100-years-ux/
  171. Alison Nightingale. 2009. A guide to systematic literature reviews. Surgery (Oxford) 27, 9 (Sept. 2009), 381–384. https://doi.org/10.1016/j.mpsur.2009.07.005
  172. Don Norman and Jakob Nielsen. 1998. The Definition of User Experience. Nielsen Norman Group (Aug 1998). https://www.nngroup.com/articles/definition-user-experience/
  173. Donald A Norman. 2010. The research-Practice Gap: The need for translational developers. interactions 17, 4 (2010), 9–12.
  174. Eye Tracking, Usability, and User Experience: A Systematic Review. International Journal of Human–Computer Interaction 0, 0 (June 2023), 1–17. https://doi.org/10.1080/10447318.2023.2221600 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447318.2023.2221600.
  175. Color compatibility from large datasets. ACM Transactions on Graphics 30, 4 (July 2011), 1–12. https://doi.org/10.1145/2010324.1964958
  176. Exploratory font selection using crowdsourced attributes. ACM Transactions on Graphics 33, 4 (July 2014), 92:1–92:9. https://doi.org/10.1145/2601097.2601110
  177. Judith S Olson and Wendy A Kellogg. 2014. Ways of Knowing in HCI. Vol. 2. Springer.
  178. A machine learning-based usability evaluation method for eLearning systems. Decision Support Systems 56 (Dec. 2013), 63–73. https://doi.org/10.1016/j.dss.2013.05.003
  179. Directing user attention via visual flow on web designs. ACM Transactions on Graphics 35, 6 (Dec. 2016), 240:1–240:11. https://doi.org/10.1145/2980179.2982422
  180. Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–16. https://doi.org/10.1145/3411764.3445734
  181. MUBot: Learning to Test Large-Scale Commercial Android Apps like a Human. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). 543–552. https://doi.org/10.1109/ICSME55016.2022.00074 ISSN: 2576-3148.
  182. Kara Pernice. 2019. Affinity Diagramming: Collaborate, Sort and Prioritize UX Ideas (Video). https://www.nngroup.com/videos/affinity-diagramming/
  183. Using Natural Language Processing and Sentiment Analysis to Augment Traditional User-Centered Design: Development and Usability Study. JMIR mHealth and uHealth 8, 8 (Aug. 2020), e16862. https://doi.org/10.2196/16862 Company: JMIR mHealth and uHealth Distributor: JMIR mHealth and uHealth Institution: JMIR mHealth and uHealth Label: JMIR mHealth and uHealth Publisher: JMIR Publications Inc., Toronto, Canada.
  184. Deep learning for automatic usability evaluations based on images: A case study of the usability heuristics of thermostats. Energy and Buildings 163 (March 2018), 111–120. https://doi.org/10.1016/j.enbuild.2017.12.043
  185. RUITE: Refining UI Layout Aesthetics Using Transformer Encoder. In 26th International Conference on Intelligent User Interfaces - Companion (IUI ’21 Companion). Association for Computing Machinery, New York, NY, USA, 81–83. https://doi.org/10.1145/3397482.3450716
  186. Android in the Wild: A Large-Scale Dataset for Android Device Control. https://doi.org/10.48550/arXiv.2307.10088 arXiv:2307.10088 [cs].
  187. Predicting users’ first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. In Proceedings of the SIGCHI conference on human factors in computing systems. 2049–2058.
  188. Andreas Riegler and Clemens Holzmann. 2015. UI-CAT: Calculating user interface complexity metrics for mobile applications. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia. 390–394.
  189. Tim Rietz and Alexander Maedche. 2020. Towards the Design of an Interactive Machine Learning System for Qualitative Coding. In International Conference on Information Systems, ICIS 2020 - Making Digital Inclusive: Blending the Local and the Global, India, December 13-16, 2020. Ed.: Joey George. 1830. https://publikationen.bibliothek.kit.edu/1000124563
  190. Tim Rietz and Alexander Maedche. 2021. Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. https://doi.org/10.1145/3411764.3445591
  191. High-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New Orleans, LA, USA, 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042
  192. Maria Rosala. 2020. The Discovery Phase in UX Projects. https://www.nngroup.com/articles/discovery-phase/
  193. Maria Rosala. 2022. How to Analyze Qualitative Data from UX Research: Thematic Analysis. https://www.nngroup.com/articles/thematic-analysis/
  194. Do predictions of visual perception aid design? ACM Transactions on Applied Perception 8, 2 (Feb. 2011), 12:1–12:20. https://doi.org/10.1145/1870076.1870080
  195. Jenny Ruiz and Monique Snoeck. 2022. Feedback Generation for Automatic User Interface Design Evaluation. In Software Technologies (Communications in Computer and Information Science), Hans-Georg Fill, Marten van Sinderen, and Leszek A. Maciaszek (Eds.). Springer International Publishing, Cham, 67–93. https://doi.org/10.1007/978-3-031-11513-4_4
  196. Stuart J Russell and Peter Norvig. 2010. Artificial intelligence a modern approach. London.
  197. Niloufar Salehi. 2023. I tried out SyntheticUsers, so you don’t have to. https://niloufars.substack.com/p/i-tried-out-syntheticusers-so-you
  198. A Literature Review of Quantitative Persona Creation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–14. https://doi.org/10.1145/3313831.3376502
  199. A Survey of 15 Years of Data-Driven Persona Development. International Journal of Human–Computer Interaction 37, 18 (Nov. 2021), 1685–1708. https://doi.org/10.1080/10447318.2021.1908670
  200. Alberto Samele and Nicolas Burny. 2023. Bootstrapped Evaluation with OctoDollop: A Mobile Application for Evaluating Mobile GUI Aesthetics in Context. In Companion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ’23 Companion). Association for Computing Machinery, New York, NY, USA, 73–75. https://doi.org/10.1145/3596454.3597186
  201. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–21. https://doi.org/10.1145/3491102.3517497
  202. LoFi Sketch: A Large Scale Dataset of Smartphone Low Fidelity Sketches. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. ACM, New Orleans LA USA, 1–5. https://doi.org/10.1145/3491101.3519624
  203. MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities. In Proceedings of the 32nd Australian Conference on Human-Computer Interaction (OzCHI ’20). Association for Computing Machinery, New York, NY, USA, 403–412. https://doi.org/10.1145/3441000.3441030
  204. SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors. In 26th International Conference on Intelligent User Interfaces. ACM, College Station TX USA, 67–69. https://doi.org/10.1145/3397482.3450725
  205. UISketch: A Large-Scale Dataset of UI Element Sketches. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. https://doi.org/10.1145/3411764.3445784
  206. Predicting Eye Fixations on Webpage With an Ensemble of Early Features and High-Level Representations from Deep Network. IEEE Transactions on Multimedia 17, 11 (Nov. 2015), 2084–2093. https://doi.org/10.1109/TMM.2015.2483370 Conference Name: IEEE Transactions on Multimedia.
  207. Shaping the Emerging Norms of Using Large Language Models in Social Computing Research. http://arxiv.org/abs/2307.04280 arXiv:2307.04280 [cs].
  208. Integrating AI in Human-Human Collaborative Ideation. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–5. https://doi.org/10.1145/3544549.3573802
  209. Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press. Google-Books-ID: mSRXEAAAQBAJ.
  210. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annual Review of Psychology 70, 1 (2019), 747–770. https://doi.org/10.1146/annurev-psych-010418-102803 _eprint: https://doi.org/10.1146/annurev-psych-010418-102803.
  211. Dominik Siemon. 2023. Let the computer evaluate your idea: evaluation apprehension in human-computer collaboration. Behaviour & Information Technology 42, 5 (April 2023), 459–477. https://doi.org/10.1080/0144929X.2021.2023638
  212. Machine Learning Methods for Inferring Interaction Design Patterns from Textual Requirements. Proceedings 31, 1 (2019), 26. https://doi.org/10.3390/proceedings2019031026 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
  213. GUILGET: GUI Layout GEneration with Transformer. https://doi.org/10.48550/arXiv.2304.09012 arXiv:2304.09012 [cs].
  214. Literature Reviews in HCI: A Review of Reviews. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–24. https://doi.org/10.1145/3544548.3581332
  215. Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agenda. Information Technology & People (2023).
  216. Artificial intelligence and life in 2030: the one hundred year study on artificial intelligence. arXiv preprint arXiv:2211.06318 (2022).
  217. OwlEyes-online: a fully automated platform for detecting and localizing UI display issues. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 1500–1504. https://doi.org/10.1145/3468264.3473109
  218. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Association for Computing Machinery, 1–12. https://doi.org/10.1145/3173574.3174078 Place: New York, NY, USA.
  219. Amanda Swearngin and Yang Li. 2019. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300305
  220. Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3313831.3376593 arXiv:2001.05424 [cs].
  221. Generating Personas for Products on Social Media: A Mixed Method to Analyze Online Users. International Journal of Human–Computer Interaction 38, 13 (Aug. 2022), 1255–1266. https://doi.org/10.1080/10447318.2021.1990520 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447318.2021.1990520.
  222. A Conversational Agent as Facilitator: Guiding Groups Through Collaboration Processes. In Learning and Collaboration Technologies. Human and Technology Ecosystems, Panayiotis Zaphiris and Andri Ioannou (Eds.). Vol. 12206. Springer International Publishing, Cham, 108–129. https://doi.org/10.1007/978-3-030-50506-6_9 Series Title: Lecture Notes in Computer Science.
  223. Conversations with GUIs. In Designing Interactive Systems Conference 2021. ACM, Virtual Event USA, 1447–1457. https://doi.org/10.1145/3461778.3462124
  224. Analyzing users’ narratives to understand experience with interactive products. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 2079–2088. https://doi.org/10.1145/2470654.2481285
  225. Mathias Peter Verheijden and Mathias Funk. 2023. Collaborative Diffusion: Boosting Designerly Co-Creation with Generative AI. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–8. https://doi.org/10.1145/3544549.3585680
  226. Aaron Richard Vontell. 2019. Bility : automated accessibility testing for mobile applications. Thesis. Massachusetts Institute of Technology. https://dspace.mit.edu/handle/1721.1/121685 Accepted: 2019-07-15T20:33:58Z Journal Abbreviation: Automated accessibility testing for mobile applications.
  227. Yuki Wakatsuki and Yusuke Yamamoto. 2021. Clustering to Support Users Finding Unexpected Perspectives in Brainstorming. In 2021 10th International Congress on Advanced Applied Informatics (IIAI-AAI). 494–497. https://doi.org/10.1109/IIAI-AAI53430.2021.00086 ISSN: 2770-8470.
  228. Enabling Conversational Interaction with Mobile UI using Large Language Models. http://arxiv.org/abs/2209.08655 arXiv:2209.08655 [cs].
  229. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. http://arxiv.org/abs/2108.03353 arXiv:2108.03353 [cs].
  230. Learning to Select Elements for Graphic Design. In Proceedings of the 2020 International Conference on Multimedia Retrieval (ICMR ’20). Association for Computing Machinery, New York, NY, USA, 91–99. https://doi.org/10.1145/3372278.3390678
  231. Where is Your App Frustrating Users?. In Proceedings of the 44th International Conference on Software Engineering. 2427–2439. https://doi.org/10.1145/3510003.3510189 arXiv:2204.09310 [cs].
  232. Boosting GUI Prototyping with Diffusion Models. http://arxiv.org/abs/2306.06233 arXiv:2306.06233 [cs].
  233. Empowering LLM to use Smartphone for Intelligent Task Automation. https://doi.org/10.48550/arXiv.2308.15272 arXiv:2308.15272 [cs].
  234. Jacob O. Wobbrock and Julie A. Kientz. 2016. Research contributions in human-computer interaction. Interactions 23, 3 (April 2016), 38–44. https://doi.org/10.1145/2907069
  235. Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, London England United Kingdom, 1–10. https://doi.org/10.1145/2601248.2601268
  236. Screen Correspondence: Mapping Interchangeable Elements between UIs. http://arxiv.org/abs/2301.08372 arXiv:2301.08372 [cs].
  237. WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. https://doi.org/10.48550/arXiv.2301.13280 arXiv:2301.13280 [cs].
  238. Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13. https://doi.org/10.1145/3313831.3376324
  239. Understanding and Modeling User-Perceived Brand Personality from Mobile Application UIs. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300443
  240. Yu Xiao and Maria Watson. 2019. Guidance on Conducting a Systematic Literature Review. Journal of Planning Education and Research 39, 1 (March 2019), 93–112. https://doi.org/10.1177/0739456X17723971 Publisher: SAGE Publications Inc.
  241. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In 28th International Conference on Intelligent User Interfaces. ACM, Sydney NSW Australia, 75–78. https://doi.org/10.1145/3581754.3584136
  242. If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–14. https://doi.org/10.1145/3313831.3376131
  243. Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions. ACM Transactions on Computer-Human Interaction 27, 3 (June 2020), 1–37. https://doi.org/10.1145/3381804
  244. Computational model for predicting user aesthetic preference for GUI using DCNNs. CCF Transactions on Pervasive Computing and Interaction 3, 2 (June 2021), 147–169. https://doi.org/10.1007/s42486-021-00064-4
  245. TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking. https://doi.org/10.48550/arXiv.1504.06755 arXiv:1504.06755 [cs].
  246. Spatio-Temporal Modeling and Prediction of Visual Attention in Graphical User Interfaces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3299–3310. https://doi.org/10.1145/2858036.2858479
  247. Wei Xu. 2019. Toward human-centered AI: a perspective from human-computer interaction. Interactions 26, 4 (June 2019), 42–46. https://doi.org/10.1145/3328485
  248. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562 (2023).
  249. Exploiting user experience from online customer reviews for product design. International Journal of Information Management 46 (June 2019), 173–186. https://doi.org/10.1016/j.ijinfomgt.2018.12.006
  250. Don’t Do That! Hunting Down Visual Design Smells in Complex UIs Against Design Guidelines. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 761–772. https://doi.org/10.1109/ICSE43902.2021.00075 ISSN: 1558-1225.
  251. UIS-Hunter: Detecting UI Design Smells in Android Apps. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 89–92. https://doi.org/10.1109/ICSE-Companion52605.2021.00043 ISSN: 2574-1926.
  252. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13. https://doi.org/10.1145/3313831.3376301
  253. Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300468
  254. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. https://doi.org/10.48550/arXiv.2207.01206 arXiv:2207.01206 [cs].
  255. How Experienced Designers of Enterprise Applications Engage AI as a Design Material. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–13. https://doi.org/10.1145/3491102.3517491
  256. Robert J. Youmans and Thomaz Arciszewski. 2014. Design fixation: Classifications and modern methods of prevention. AI EDAM 28, 2 (May 2014), 129–137. https://doi.org/10.1017/S0890060414000043 Publisher: Cambridge University Press.
  257. Multimodal Icon Annotation For Mobile Applications. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction. ACM, Toulouse & Virtual France, 1–11. https://doi.org/10.1145/3447526.3472064
  258. LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models. https://doi.org/10.48550/arXiv.2303.11589 arXiv:2303.11589 [cs].
  259. Data-driven Personas: Constructing Archetypal Users with Clickstreams and User Telemetry. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose California USA, 5350–5359. https://doi.org/10.1145/2858036.2858523
  260. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–15. https://doi.org/10.1145/3411764.3445186
  261. Melting Your Models: An Integrated AI-based Creativity Support Tool for Inspiration Evolution. In 2022 15th International Symposium on Computational Intelligence and Design (ISCID). 97–101. https://doi.org/10.1109/ISCID56505.2022.00029 ISSN: 2473-3547.
  262. Seenomaly: vision-based linting of GUI animation effects against design-don’t guidelines. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 1286–1297. https://doi.org/10.1145/3377811.3380411
  263. Modeling Fonts in Context: Font Prediction on Web Designs. Computer Graphics Forum 37, 7 (Oct. 2018), 385–395. https://doi.org/10.1111/cgf.13576
  264. ICONATE: Automatic Compound Icon Generation and Ideation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13. https://doi.org/10.1145/3313831.3376618
  265. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. https://doi.org/10.48550/arXiv.2101.09978 arXiv:2101.09978 [cs].
  266. Qihao Zhu and Jianxi Luo. 2023. Toward Artificial Empathy for Human-Centered Design: A Framework. http://arxiv.org/abs/2303.10583 arXiv:2303.10583 [cs].
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yuwen Lu (8 papers)
  2. Yuewen Yang (4 papers)
  3. Qinyi Zhao (3 papers)
  4. Chengzhi Zhang (37 papers)
  5. Toby Jia-Jun Li (57 papers)
Citations (9)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets