Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs (2403.13690v1)

Published 20 Mar 2024 in cs.SE, cs.CV, and cs.HC

Abstract: Recent research has begun to examine the potential of automatically finding and fixing accessibility issues that manifest in software. However, while recent work makes important progress, it has generally been skewed toward identifying issues that affect users with certain disabilities, such as those with visual or hearing impairments. However, there are other groups of users with different types of disabilities that also need software tooling support to improve their experience. As such, this paper aims to automatically identify accessibility issues that affect users with motor-impairments. To move toward this goal, this paper introduces a novel approach, called MotorEase, capable of identifying accessibility issues in mobile app UIs that impact motor-impaired users. Motor-impaired users often have limited ability to interact with touch-based devices, and instead may make use of a switch or other assistive mechanism -- hence UIs must be designed to support both limited touch gestures and the use of assistive devices. MotorEase adapts computer vision and text processing techniques to enable a semantic understanding of app UI screens, enabling the detection of violations related to four popular, previously unexplored UI design guidelines that support motor-impaired users, including: (i) visual touch target size, (ii) expanding sections, (iii) persisting elements, and (iv) adjacent icon visual distance. We evaluate MotorEase on a newly derived benchmark, called MotorCheck, that contains 555 manually annotated examples of violations to the above accessibility guidelines, across 1599 screens collected from 70 applications via a mobile app testing tool. Our experiments illustrate that MotorEase is able to identify violations with an average accuracy of ~90%, and a false positive rate of less than 9%, outperforming baseline techniques.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Accessibility: Apple human interface guidelines. https://developer.apple.com/design/human-interface-guidelines/foundations/accessibility/.
  2. Accessibility scanner. https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.auditor.
  3. Android switch access service. https://support.google.com/accessibility/android/answer/6122836?hl=en.
  4. Google cloud vision api. https://cloud.google.com/vision/docs/ocr.
  5. Motorease github repository. https://github.com/SageSELab/MotorEase.
  6. Motorease website. https://sagelab.io/MotorEase.
  7. Motorease zenodo archive. https://zenodo.org/doi/10.5281/zenodo.10460700.
  8. https://www.section508.gov/blog/Universal-Design-What-is-it/.
  9. “design for android: android developers,” https://developer.android.com/design.
  10. Ada web accessibility lawsuit recap report https://blog.usablenet.com/2018-ada-web-accessibility-lawsuit-recap-report, 2018.
  11. Adalaws https://www.ada.gov/cguide.htm, 2019.
  12. Accessibility guide https://accessibility.18f.gov/checklist/accessibility, 2022.
  13. Accessibility https://developer.android.com/guide/topics/ui/accessibility, 2022.
  14. Motor impairment https://accessibility.huit.harvard.edu/disabilities/motor-impairment?page=1, 2022.
  15. Torchvision- torchvision main documentation. https://pytorch.org/vision/stable/index.html, 2022.
  16. Web content accessibility guidelines (wcag). https://www.w3.org/TR/WCAG21/, journal=W3C, 2022.
  17. https://f-droid.org/, F-droid.
  18. https://play.google.com/store, Google Play Store.
  19. https://labelstud.io, LabelStudio.
  20. Automatically generating tailored accessible user interfaces for ubiquitous services. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’11, page 187–194, New York, NY, USA, 2011. ACM.
  21. Are users the gold standard for accessibility evaluation? In Proceedings of the 11th Web for All Conference, W4A ’14, New York, NY, USA, 2014. ACM.
  22. Universal design principles combined with web accessibility guidelines: A case study. In Proceedings of the IX Symposium on Human Factors in Computing Systems, IHC ’10, page 169–178, Porto Alegre, BRA, 2010. Brazilian Computer Society.
  23. Accessibility issues in android apps: State of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, page 1323–1334, New York, NY, USA, 2020. ACM.
  24. Increased accessibility to nonverbal communication through facial and expression recognition technologies for blind/visually impaired subjects. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’11, page 259–260, New York, NY, USA, 2011. ACM.
  25. M. Bajammal and A. Mesbah. Semantic web accessibility testing via hierarchical visual analysis. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 1610–1621, 2021.
  26. M. Bajammal and A. Mesbah. Semantic web accessibility testing via hierarchical visual analysis. In Proceedings of the 43rd International Conference on Software Engineering, ICSE ’21, page 1610–1621. IEEE Press, 2021.
  27. Translating video recordings of complex mobile app ui gestures into replayable scenarios. IEEE Transactions on Software Engineering, 49(04):1782–1803, apr 2023.
  28. Translating video recordings of mobile app usages into replayable scenarios. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, page 309–321, New York, NY, USA, 2020. ACM.
  29. Model-based automated accessibility testing. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’15. ACM, 2015.
  30. Beyond web content accessibility guidelines: Expert accessibility reviews. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion, DSAI 2016, page 77–84, New York, NY, USA, 2016. ACM.
  31. From ui design image to gui skeleton: A neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, page 665–676, New York, NY, USA, 2018. ACM.
  32. Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, page 322–334, New York, NY, USA, 2020. ACM.
  33. Accessible or not? an empirical investigation of android app accessibility. IEEE Transactions on Software Engineering, 48(10):3954–3968, 2022.
  34. Bagel: An approach to automatically detect navigation-based web accessibility barriers for keyboard users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, 2023. ACM.
  35. Detecting and localizing keyboard accessibility failures in web applications. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, page 855–867, New York, NY, USA, 2021. ACM.
  36. On the relation between code elements and accessibility issues in android apps. In Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing, SAST 20, page 40–49, New York, NY, USA, 2020. ACM.
  37. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST ’17, page 845–854, New York, NY, USA, 2017. ACM.
  38. Automated accessibility testing of mobile apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pages 116–126, 2018.
  39. DesignABILITY. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, May 2019.
  40. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST ’07, page 231–240, New York, NY, USA, 2007. ACM.
  41. Practical gui testing of android applications via model abstraction and refinement. In Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, page 269–280. IEEE Press, 2019.
  42. V2s: A tool for translating video recordings of mobile app usages into replayable scenarios. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 65–68, 2021.
  43. Access overlays. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, Oct. 2011.
  44. B. A. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report, 07 2007.
  45. New metrics for understanding touch by people with and without limited fine motor function. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’21, New York, NY, USA, 2021. ACM.
  46. Accessibility of high-fidelity prototyping tools. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, May 2021.
  47. Tangiblegrid: Tangible web layout design for blind users. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST ’22, New York, NY, USA, 2022. ACM.
  48. Droidbot: a lightweight ui-guided test input generator for android. In ICSE-C. IEEE, 2017.
  49. Test automation in open-source android apps: A large-scale empirical study. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE ’20, page 1078–1089, New York, NY, USA, 2021. ACM.
  50. Mining android app usages for generating actionable gui-based execution scenarios. In Proceedings of the 12th Working Conference on Mining Software Repositories, MSR ’15, page 111–122. IEEE Press, 2015.
  51. How do developers test android applications? In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 613–622, 2017.
  52. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 399–410, 2017.
  53. Z. Liu. Discovering ui display issues with visual understanding. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE ’20, page 1373–1375, New York, NY, USA, 2021. ACM.
  54. Owl eyes: Spotting ui display issues via visual understanding. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE ’20, page 398–409, New York, NY, USA, 2021. ACM.
  55. 1 thumb, 4 buttons, 20 words per minute: Design and evaluation of h4-writer. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, page 471–480, New York, NY, USA, 2011. ACM.
  56. Aidui: Toward automated recognition of dark patterns in user interfaces. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1958–1970. IEEE, 2023.
  57. Sapienz: Multi-objective automated testing for android applications. In ISSTA. ACM, 2016.
  58. Accessibility of mobile applications: Evaluation by users with visual impairment and by automated tools. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems, IHC ’20, New York, NY, USA, 2020. ACM.
  59. T. B. McHugh and C. Barth. Assistive technology design as a computer science learning experience. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’20, New York, NY, USA, 2020. ACM.
  60. Data-driven accessibility repair revisited: On the effectiveness of generating labels for icons in android apps. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, page 107–118, New York, NY, USA, 2021. ACM.
  61. Blocks4all. In CHI’18. ACM, Apr. 2018.
  62. Designing for individuals: Usable touch-screen interaction through shared user models. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’12, page 151–158, New York, NY, USA, 2012. ACM.
  63. Motor-impaired touchscreen interactions in the wild. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers ; Accessibility, ASSETS ’14, page 123–130, New York, NY, USA, 2014. ACM.
  64. Automated reporting of gui design violations for mobile apps. In Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, page 165–175, New York, NY, USA, 2018. ACM.
  65. Crashscope: A practical tool for automated testing of android applications.
  66. Automatically discovering, reporting and reproducing android application crashes. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 33–44, 2016.
  67. Detecting and summarizing gui changes in evolving mobile apps. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE ’18, page 543–553, New York, NY, USA, 2018. ACM.
  68. MulongXie. Uied - ui element detection, detecting ui elements from ui screenshots or drawnings, 2021.
  69. How accessible is the process of web interface design? In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’13, New York, NY, USA, 2013. ACM.
  70. User interface design guidelines for smartphone applications for people with parkinson’s disease. Universal Access in the Information Society, 15(4):659–679, Oct. 2015.
  71. Follow that sound: Using sonification and corrective verbal feedback to teach touchscreen gestures. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’13, New York, NY, USA, 2013. ACM.
  72. Target size study for one-handed thumb use on small touchscreen devices. In Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI ’06, page 203–210, New York, NY, USA, 2006. ACM.
  73. Toward accessible mobile application design: Developing mobile application accessibility guidelines for people with visual impairment. In Proceedings of HCI Korea, HCIK ’15, page 31–38, Seoul, KOR, 2014. Hanbit Media, Inc.
  74. Rescribe. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, Oct. 2020.
  75. Personaltouch: Improving touchscreen usability by personalizing accessibility settings based on individual user’s touchscreen interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–11, New York, NY, USA, 2019. ACM.
  76. Glove: Global vectors for word representation. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543. ACL, 2014.
  77. N. P. K. Ramachandra and C. Csallner. Testing web-based applications with the ¡u¿v¡/u¿oice ¡u¿c¡/u¿ontrolled ¡u¿a¡/u¿ccessibility and ¡u¿t¡/u¿esting tool (vcat). In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE ’18, page 208–209, New York, NY, USA, 2018. ACM.
  78. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
  79. Examining image-based button labeling for accessibility in android apps through large-scale analysis. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, Oct. 2018.
  80. Latte: Use-case and assistive-service driven automated accessibility testing framework for android. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, 2021. ACM.
  81. Groundhog: An automated accessibility crawler for mobile apps. In 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022), 2022.
  82. Are user reviews useful for identifying accessibility issues that autistic users face? an exploratory study. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, IHC ’22, New York, NY, USA, 2022. ACM.
  83. Methodological standards in accessibility research on motor impairments: A survey. ACM Comput. Surv., may 2022. Just Accepted.
  84. Evaluating alternatives for better deaf accessibility to selected web-based multimedia. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’15, page 231–238, New York, NY, USA, 2015. ACM.
  85. Design and evaluation of mobile applications for people with visual impairments: A compilation of usable accessibility guidelines. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, IHC ’19, New York, NY, USA, 2019. ACM.
  86. Guided, stochastic model-based gui testing of android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, page 245–256, New York, NY, USA, 2017. ACM.
  87. A. Swearngin and Y. Li. Modeling mobile interface tappability using crowdsourcing and deep learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–11, New York, NY, USA, 2019. ACM.
  88. Can everyone use my app? an empirical study on accessibility in android apps. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 41–52, 2019.
  89. Vet: identifying and avoiding UI exploration tarpits. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 83–94, Athens Greece, Aug. 2021. ACM.
  90. S. Yan and P. G. Ramachandran. The current status of accessibility in mobile apps. ACM Trans. Access. Comput., 12(1), feb 2019.
  91. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, 2021. ACM.
  92. Optimization of switch keyboards. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’13, New York, NY, USA, 2013. ACM.
  93. Avgust: automating usage-based test generation from videos of app executions. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 421–433, New York, NY, USA, 2022. ACM.
  94. East: an efficient and accurate scene text detector. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 5551–5560, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Arun Krishnavajjala (1 paper)
  2. SM Hasan Mansur (3 papers)
  3. Justin Jose (3 papers)
  4. Kevin Moran (66 papers)
Citations (2)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets