Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing (2403.17807v1)

Published 26 Mar 2024 in cs.HC

Abstract: Previous research underscored the potential of danmaku--a text-based commenting feature on videos--in engaging hearing audiences. Yet, for many Deaf and hard-of-hearing (DHH) individuals, American Sign Language (ASL) takes precedence over English. To improve inclusivity, we introduce "Signmaku," a new commenting mechanism that uses ASL, serving as a sign language counterpart to danmaku. Through a need-finding study (N=12) and a within-subject experiment (N=20), we evaluated three design styles: real human faces, cartoon-like figures, and robotic representations. The results showed that cartoon-like signmaku not only entertained but also encouraged participants to create and share ASL comments, with fewer privacy concerns compared to the other designs. Conversely, the robotic representations faced challenges in accurately depicting hand movements and facial expressions, resulting in higher cognitive demands on users. Signmaku featuring real human faces elicited the lowest cognitive load and was the most comprehensible among all three types. Our findings offered novel design implications for leveraging generative AI to create signmaku comments, enriching co-learning experiences for DHH individuals.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. Evaluating the Quality of Learning Resources: A Learnersourcing Approach. IEEE Transactions on Learning Technologies 14, 1 (2021), 81–92. https://doi.org/10.1109/TLT.2021.3058644
  2. Nalan Aksakal. 2015. Theoretical view to the approach of the edutainment. Procedia-Social and Behavioral Sciences 186 (2015), 1232–1239.
  3. At a Different Pace: Evaluating Whether Users Prefer Timing Parameters in American Sign Language Animations to Differ from Human Signers’ Timing. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. 1–12.
  4. Muhammad Ali and Anza Nasir. 2023. Investigating the Social Media Practices among Aurally Challenged Students (ACS): The Challenges and Issues. Human Nature Journal of Social Sciences 4, 1 (2023), 518–532.
  5. If online learning works for you, what about deaf students? Emerging challenges of online learning for deaf and hearing-impaired students during COVID-19: a literature review. Universal access in the information society (2022), 1–20.
  6. Digital Speech Makeup: Voice Conversion Based Altered Auditory Feedback for Transforming Self-Representation. In Proceedings of the 2021 International Conference on Multimodal Interaction. 159–167.
  7. Riku Arakawa and Hiromu Yakura. 2021. Mindless Attractor: A False-Positive Resistant Intervention for Drawing Attention Using Auditory Perturbation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
  8. Charlotte Baker-Shenk. 1985. The facial behavior of deaf signers: Evidence of a complex language. American Annals of the Deaf 130, 4 (1985), 297–304.
  9. Factors associated with social interactions between deaf children and their hearing peers: A systematic literature review. Journal of deaf studies and deaf education 19, 3 (2014), 285–302.
  10. H Dirksen Bauman and Joseph Murray. 2009. Reframing: From hearing loss to deaf gain. Deaf Studies Digital Journal 1, 1 (2009), 1–10.
  11. Preferred appearance of captions generated by automatic speech recognition for deaf and hard-of-hearing viewers. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6.
  12. Deaf and hard-of-hearing users’ prioritization of genres of online video content requiring accurate captions. In Proceedings of the 17th International Web for All Conference. 1–12.
  13. Exploring collaborative caption editing to augment video-based learning. Educational technology research and development 70, 5 (2022), 1755–1779.
  14. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility. 16–31.
  15. Exploring collection of sign language datasets: Privacy, participation, and model performance. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1–14.
  16. CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–15.
  17. Exploring Think-Aloud Method with Deaf and Hard of Hearing College Students. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 1757–1772. https://doi.org/10.1145/3563657.3595980
  18. Exploring the complementary features of audio and text notes for video-based learning in mobile settings. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7.
  19. ”I was afraid, but now I enjoy being a streamer!” Understanding the Challenges and Prospects of Using Live Streaming for Online Education. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–32.
  20. Timeline-anchored comments in video-based learning: The impact of visual layout and content depth. International Journal of Human–Computer Interaction 38, 9 (2022), 868–883.
  21. Understanding gratifications of watching danmaku videos–videos with overlaid comments. In Cross-Cultural Design Methods, Practice and Impact: 7th International Conference, CCD 2015, Held as Part of HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part I 7. Springer, 153–163.
  22. Watching a movie alone yet together: understanding reasons for watching Danmaku videos. International Journal of Human–Computer Interaction 33, 9 (2017), 731–743.
  23. Technology support for the inclusion of deaf students in mainstream schools: a summary of research from 2007 to 2017. Universal Access in the Information Society 19 (2020), 195–200.
  24. Machine translation from signed to spoken languages: state of the art and challenges. Universal Access in the Information Society (apr 2023). https://doi.org/10.1007/s10209-023-00992-1
  25. Geoffrey Restall Coulter. 1979. American sign language typology. University of California, San Diego.
  26. Shane Dawson. 2006. A study of the relationship between student communication interaction and sense of community. The Internet and Higher Education 9, 3 (2006), 153–162.
  27. Multitask emotion recognition with incomplete labels. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). IEEE, 592–599.
  28. ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition. arXiv preprint arXiv:2304.05934 (2023).
  29. Understanding Differences in Human-Robot Teaming Dynamics between Deaf/Hard of Hearing and Hearing Individuals. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 552–556.
  30. Michele Friedner and Annelies Kusters. 2015. It’s a small world: International deaf spaces and encounters. Gallaudet University Press.
  31. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
  32. ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks. arXiv preprint arXiv:2207.04021 (2022).
  33. Beyond Entertainment: Unpacking Danmaku and Comments’ Role of Information Sharing and Sentiment Expression in Online Crisis Videos. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–27.
  34. Closed-captioned television presentation speed and vocabulary. American Annals of the deaf (1996), 284–292.
  35. Sushant Kafle and Matt Huenerfauth. 2016. Effect of speech recognition errors on text understandability for people who are deaf or hard of hearing. (2016).
  36. Karen L Kritzer and Chad E Smith. 2020. Educating deaf and hard-of-hearing students during COVID-19: What parents need to know. The Hearing Journal 73, 8 (2020), 32.
  37. Closed ASL interpreting for online videos. In Proceedings of the 14th International Web for All Conference. 1–4.
  38. Facial emotion recognition of students using convolutional neural network. In 2019 third international conference on intelligent computing in data sciences (ICDS). IEEE, 1–6.
  39. American sign language video anonymization to support online participation of deaf and hard of hearing users. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. 1–13.
  40. An exploration of captioning practices and challenges of individual content creators on YouTube for people with hearing impairments. arXiv preprint arXiv:2201.11226 (2022).
  41. Classification and quantification of Danmaku Interactions in online video lectures: An exploratory study. https://www.hindawi.com/journals/wcmc/2022/5656669/
  42. SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users. In Conference on Computers and Accessibility (ASSETS’22).
  43. Research on the Influencing Factors of Online Video Danmaku Watching and Participation. In 2023 4th International Conference on Education, Knowledge and Information Management (ICEKIM 2023). Atlantis Press, 711–721.
  44. An exploratory study: using Danmaku in online video-based lectures. Educational Media International 55, 3 (2018), 273–286.
  45. Social app accessibility for deaf signers. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–31.
  46. Access to postsecondary education through sign language interpreting. Journal of Deaf Studies and deaf education 10, 1 (2005), 38–50.
  47. Emma McDonnell. 2022. Understanding Social and Environmental Factors to Enable Collective Access Approaches to the Design of Captioning Technology. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–8.
  48. Automated 3D sign language caption generation for video. Universal Access in the Information Society 19 (2020), 725–738.
  49. How Many People Use ASL in the United States? Why Estimates Need Updating. Sign Language Studies 6 (03 2006). https://doi.org/10.1353/sls.2006.0019
  50. Huda Shaaban Awed Mohammad Ahmed Hammad. 2023. Investigating Social Media Usage Among the Deaf and Hard of Hearing Students. Journal of Southwest Jiaotong University 58, 1 (2023).
  51. United Nations. 2022. https://www.un.org/en/coronavirus/disability-inclusion
  52. Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: A systematic literature review. Education and Information Technologies 25 (2020), 5499–5547.
  53. Inside deaf culture. Harvard University Press.
  54. A Systematic Review of User Studies as a Basis for the Design of Systems for Automatic Sign Language Processing. ACM Transactions on Accessible Computing 15, 4 (2022), 1–33.
  55. Cross Transferring Activity Recognition to Word Level Sign Language Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2446–2453.
  56. Edutainment Tools for Deaf and Impairment Hearing Children: A Survey. Advanced Science Letters 24, 3 (2018), 1805–1808.
  57. Sign language recognition: A deep survey. Expert Systems with Applications 164 (2021), 113794.
  58. Unveiling teachers’ beliefs on visual cognition and learning styles of deaf and hard of hearing students: A Portuguese-Swedish study. Plos one 17, 2 (2022), e0263216.
  59. Alfred P Rovai. 2002. Development of an instrument to measure classroom community. The Internet and higher education 5, 3 (2002), 197–211.
  60. Video-based learning (VBL)—past, present and future: An overview of the research published from 2008 to 2019. Technology, Knowledge and Learning 26, 4 (2021), 1061–1077.
  61. Signing at scale: Learning to co-articulate signs for large-scale photo-realistic sign language production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5141–5151.
  62. Jeff Seaman and Hester Tinti-Kane. 2013. Social media for teaching and learning. Pearson Learning Systems London.
  63. Behavioral changes in speakers who are automatically captioned in meetings with deaf or hard-of-hearing peers. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 68–80.
  64. Umesh Sharma and Spencer J Salend. 2016. Teaching assistants in inclusive classrooms: A systematic analysis of the international research. Australian Journal of Teacher Education (Online) 41, 8 (2016), 118–134.
  65. Katta Spiel and Robin Angelini. 2022. Expressive Bodies Engaging with Embodied Disability Cultures for Collaborative Design Critiques. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–6.
  66. Michael Stinson and Shirin Antia. 1999. Considerations in educating deaf and hard-of-hearing students in inclusive settings. Journal of deaf studies and deaf education 4, 3 (1999), 163–175.
  67. Graph-based multimodal sequential embedding for sign language translation. IEEE Transactions on Multimedia 24 (2021), 4433–4445.
  68. David R Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation 27, 2 (2006), 237–246.
  69. The Italian Deaf community at the time of Coronavirus. Frontiers in Sociology 5 (2021), 125.
  70. Social media use for deaf and hard of hearing students in educational settings: a systematic review of literature. Deafness & Education International 19, 3-4 (2017), 144–161.
  71. Moving beyond ’one size fits all’: research considerations for working with vulnerable populations. Interactions 26 (10 2019), 34–39. https://doi.org/10.1145/3358904
  72. Traci Patricia Weast. 2008. Questions in American Sign Language: A quantitative analysis of raised and lowered eyebrows. The University of Texas at Arlington.
  73. Danmaku: A new paradigm of social interaction via online videos. ACM Transactions on Social Computing 2, 2 (2019), 1–24.
  74. Danmaku vs. forum comments: understanding user participation and knowledge sharing in online videos. In Proceedings of the 2018 ACM International Conference on Supporting Group Work. 209–218.
  75. Millions and billions of views: Understanding popular science and knowledge communication on video-sharing platforms. In Proceedings of the Ninth ACM Conference on Learning@ Scale. 163–174.
  76. Vtoonify: Controllable high-resolution portrait video style transfer. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–15.
  77. Understanding Danmaku’s potential in online video learning. In Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. 3034–3040.
  78. Video learning analytics: Investigating behavioral patterns and learner clusters in video-based online learning. The Internet and Higher Education 50 (2021), 100806.
  79. “It’s Just Part of Me:” Understanding Avatar Diversity and Self-presentation of People with Disabilities in Social Virtual Reality. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–16.
  80. Understanding Communication Strategies and Viewer Engagement with Science Knowledge Videos on Bilibili. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 668, 18 pages. https://doi.org/10.1145/3544548.3581476

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets