Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tappy: Predicting Tap Accuracy of User-Interface Elements by Reverse-Engineering Webpage Structures (2403.03097v3)

Published 5 Mar 2024 in cs.HC

Abstract: Selecting a UI element is a fundamental operation on webpages, and the ease of tapping a target object has a significant impact on usability. It is thus important to analyze existing UIs in order to design better ones. However, tools proposed in previous studies cannot identify whether an element is tappable on modern webpages. In this study, we developed Tappy that can identify tappable UI elements on webpages and estimate the tap-success rate based on the element size. Our interviews of professional designers and engineers showed that Tappy helped discussions of UI design on the basis of its quantitative metric. Furthermore, we have launched this tool to be freely available to external users, so readers can access Tappy by visiting the website (https://tappy.yahoo.co.jp).

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Kayce Basques and Sofia Emelianova. 2015. Console Utilities API reference. https://developer.chrome.com/docs/devtools/console/utilities
  2. FFitts Law: Modeling Finger Touch with Fitts’ Law. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). ACM, New York, NY, USA, 1363–1372. https://doi.org/10.1145/2470654.2466180
  3. Xiaojun Bi and Shumin Zhai. 2013. Bayesian touch: a statistical criterion of target selection with finger touch. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New York, NY, USA, 51–60. https://doi.org/10.1145/2501988.2502058
  4. Xiaojun Bi and Shumin Zhai. 2016. Predicting Finger-Touch Accuracy Based on the Dual Gaussian Distribution Model. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). ACM, New York, NY, USA, 313–319. https://doi.org/10.1145/2984511.2984546
  5. ChromeDevTools. 2023. Chrome DevTools Protocol. https://chromedevtools.github.io/devtools-protocol/
  6. Josh Clark. 2016. Designing for Touch (1 ed.). A Book Apart, New York, USA.
  7. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. https://doi.org/10.1145/3126594.3126651
  8. Morgan Dixon and James Fogarty. 2010. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1525–1534. https://doi.org/10.1145/1753326.1753554
  9. A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 3167–3176. https://doi.org/10.1145/2207676.2208734
  10. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 969–978. https://doi.org/10.1145/1978942.1979086
  11. Prefab Layers and Prefab Annotations: Extensible Pixel-Based Interpretation of Graphical Interfaces. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 221–230. https://doi.org/10.1145/2642918.2647412
  12. A Computational Method for Evaluating UI Patterns. arXiv:1807.04191 [cs.HC]
  13. Age-Related Differences in Performance with Touchscreens Compared to Traditional Mouse Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 343–346. https://doi.org/10.1145/2470654.2470703
  14. Leah Findlater and Lotus Zhang. 2020. Input Accessibility: A Large Dataset and Summary Analysis of Age, Motor Ability and Input Performance. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 17, 6 pages. https://doi.org/10.1145/3373625.3417031
  15. Paul M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 6 (1954), 381–391. https://doi.org/10.1037/h0055392
  16. floatas. 2018a. Chrome Extension of Bubble Cursor. https://github.com/floatas/BubbleCursor Retrieved March 20, 2023.
  17. floatas. 2018b. Implementation of the Bubble-Cursor Chrome Extension. https://github.com/floatas/BubbleCursor/blob/master/lib/vimium.js Retrieved March 20, 2023.
  18. Google. 2023. Touch target size. https://support.google.com/accessibility/android/answer/7101858?hl=en
  19. Tovi Grossman and Ravin Balakrishnan. 2005. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor’s Activation Area. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI ’05). ACM, New York, NY, USA, 281–290. https://doi.org/10.1145/1054972.1055012
  20. Steven Hoober and Eric Berkman. 2011. Designing Mobile Interfaces: Patterns for Interaction Design (1 ed.). O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472, USA.
  21. Juan Pablo Hourcade and Theresa R. Berkel. 2008. Simple pen interaction performance of young and older adults using handheld computers. Interacting with Computers 20, 1 (2008), 166–183. https://doi.org/10.1016/j.intcom.2007.10.002
  22. Modeling the Endpoint Uncertainty in Crossing-Based Moving Target Selection. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376336
  23. Jeff Johnson. 2014. Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines (2 ed.). Morgan Kaufmann, San Francisco, USA. https://doi.org/10.1016/C2012-0-07128-1
  24. Learning Design Semantics for Mobile Apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 569–579. https://doi.org/10.1145/3242587.3242650
  25. Detecting Phishing Websites via Aggregation Analysis of Page Layouts. Procedia Computer Science 129 (2018), 224–230. https://doi.org/10.1016/j.procs.2018.03.053 2017 INTERNATIONAL CONFERENCE ON IDENTIFICATION,INFORMATION AND KNOWLEDGEIN THE INTERNET OF THINGS.
  26. MDN contributors. 2023a. Document: querySelectorAll() method - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll
  27. MDN contributors. 2023b. Element: getBoundingClientRect() method - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect
  28. MDN contributors. 2023c. EventTarget: addEventListener() method - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener
  29. MDN contributors. 2023d. padding - CSS: Cascading Style Sheets — MDN. https://developer.mozilla.org/en-US/docs/Web/CSS/padding
  30. MDN contributors. 2023e. Pointer events - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events
  31. MDN contributors. 2023f. Touch events - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/Touch_events
  32. MDN contributors. 2023g. UI Events - Web APIs — MDN. https://developer.mozilla.org/en-US/docs/Web/API/UI_Events
  33. Theresa Neil. 2014. Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps (1 ed.). O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472, USA.
  34. Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI. In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (Lincoln, Nebraska) (ASE ’15). IEEE Press, Washington, DC, USA, 248–259. https://doi.org/10.1109/ASE.2015.32
  35. Methods for Automatic Web Page Layout Testing and Analysis: A Review. IEEE Access 11 (2023), 13948–13964. https://doi.org/10.1109/ACCESS.2023.3242549
  36. Puppeteer. 2023. puppeteer/puppeteer: Node.js API for Chrome. https://github.com/puppeteer/puppeteer
  37. Quora. 2023. https://www.quora.com/
  38. Insights into Layout Patterns of Mobile User Interfaces by an Automatic Analysis of Android Apps. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (London, United Kingdom) (EICS ’13). Association for Computing Machinery, New York, NY, USA, 275–284. https://doi.org/10.1145/2494603.2480308
  39. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 36, 21 pages. https://doi.org/10.1145/3491102.3517497
  40. Carlos Eduardo Silva. 2012. Reverse Engineering of GWT Applications. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Copenhagen, Denmark) (EICS ’12). Association for Computing Machinery, New York, NY, USA, 325–328. https://doi.org/10.1145/2305484.2305542
  41. Amanda Swearngin and Yang Li. 2019. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300305
  42. Modeling Movement Times and Success Rates for Acquisition of One-Dimensional Targets with Uncertain Touchable Sizes. Proc. ACM Hum.-Comput. Interact. 5, ISS, Article 487 (nov 2021), 15 pages. https://doi.org/10.1145/3486953
  43. Clarifying the Effect of Edge Targets in Touch Pointing through Crowdsourced Experiments. Proc. ACM Hum.-Comput. Interact. 7, ISS, Article 433 (nov 2023), 19 pages. https://doi.org/10.1145/3626469
  44. Predicting Touch Accuracy for Rectangular Targets by Using One-Dimensional Task Results. Proc. ACM Hum.-Comput. Interact. 6, ISS, Article 579 (nov 2022), 13 pages. https://doi.org/10.1145/3567732
  45. Never-Ending Learning of User Interfaces. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 113, 13 pages. https://doi.org/10.1145/3586183.3606824
  46. Position-wise contextual advertising: Placing relevant ads at appropriate positions of a web page. Neurocomputing 120 (2013), 524–535. https://doi.org/10.1016/j.neucom.2013.04.018 Image Feature Detection and Description.
  47. Yahoo! JAPAN Search. 2023. https://search.yahoo.co.jp/
  48. Yahoo! JAPAN Tech Blog. 2023. https://techblog.yahoo.co.jp/
  49. Yahoo! JAPAN Travel. 2023. https://travel.yahoo.co.jp/dp/?sc_i=top2dp_top_lftmn
  50. Shota Yamanaka. 2018. Effect of Gaps with Penal Distractors Imposing Time Penalty in Touch-pointing Tasks. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (Barcelona, Spain) (MobileHCI ’18). ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3229434.3229435
  51. Towards More Practical Spacing for Smartphone Touch GUI Objects Accompanied by Distractors. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (ISS ’19). Association for Computing Machinery, New York, NY, USA, 157–169. https://doi.org/10.1145/3343055.3359698
  52. Shota Yamanaka and Hiroki Usuba. 2020. Rethinking the Dual Gaussian Distribution Model for Predicting Touch Accuracy in On-Screen-Start Pointing Tasks. Proc. ACM Hum.-Comput. Interact. 4, ISS, Article 205 (Nov. 2020), 20 pages. https://doi.org/10.1145/3427333
  53. Shota Yamanaka and Hiroki Usuba. 2023. Tuning Endpoint-variability Parameters by Observed Error Rates to Obtain Better Prediction Accuracy of Pointing Misses. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 579, 18 pages. https://doi.org/10.1145/3544548.3580746
  54. Shape-Adaptive Ternary-Gaussian Model: Modeling Pointing Uncertainty for Moving Targets of Arbitrary Shapes. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 777, 18 pages. https://doi.org/10.1145/3544548.3581217
  55. Modeling the Uncertainty in Pointing of Moving Targets with Arbitrary Shapes. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3334480.3382875
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Hiroki Usuba (3 papers)
  2. Junichi Sato (5 papers)
  3. Naomi Sasaya (2 papers)
  4. Shota Yamanaka (8 papers)
  5. Fumiya Yamashita (2 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets