Papers
Topics
Authors
Recent
Search
2000 character limit reached

FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones

Published 22 Jul 2024 in cs.CV and cs.CL | (2407.15806v1)

Abstract: Progress in machine understanding of sign languages has been slow and hampered by limited data. In this paper, we present FSboard, an American Sign Language fingerspelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments. Fingerspelling recognition is an incomplete solution that is only one small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers as more broadly capable technology develops. At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks: plausible optimizations to help models run on device in real time.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. International day of sign languages, 2021. URL https://www.un.org/en/observances/sign-languages-day.
  2. URL https://www.who.int/health-topics/hearing-loss#tab=tab_1.
  3. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
  4. Computational phylogenetics reveal histories of sign languages. Science, 383(6682):519–523, 2024.
  5. Manfred Georg Mark Sherwood Phil Culliton Sam Sepah Sohier Dane Thad Starner Ashley Chow, Glenn Cameron. Google - american sign language fingerspelling recognition, 2023. URL https://kaggle.com/competitions/asl-fingerspelling.
  6. J. Keane D. Brentari G. Shakhnarovich B. Shi, A. Martinez Del Rio and K. Livescu. Fingerspelling recognition in the wild with iterative visual attention. ICCV, October 2019.
  7. J. Keane J. Michaux D. Brentari G. Shakhnarovich B. Shi, A. Martinez Del Rio and K. Livescu. American sign language fingerspelling recognition in the wild. SLT, December 2018.
  8. Popsignai: Using sign language recognition to improve american sign language learning in novice signers. IMWUT (in submission), 2024.
  9. Sign language recognition, generation, and translation: An interdisciplinary perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’19, page 16–31, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366762. doi: 10.1145/3308561.3353774. URL https://doi.org/10.1145/3308561.3353774.
  10. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing (TACCESS), 14(2):1–45, 2021a.
  11. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing, 14(2), July 2021b. URL https://www.microsoft.com/en-us/research/publication/the-fate-landscape-of-sign-language-ai-datasets-an-interdisciplinary-perspective/.
  12. United States Census Bureau. Tiger: Topologically integrated geographic encoding and referencing data (roads), 2019. URL https://www2.census.gov/geo/tiger/TIGER2019/ROADS/.
  13. Asl citizen: A community-sourced dataset for advancing isolated sign language recognition, 2023.
  14. How2sign: A large-scale multimodal dataset for continuous american sign language, 2021.
  15. Michael Erard. Why sign-language gloves don’t help deaf people. The Atlantic, Nov 2017.
  16. Sign languages in the world. Sociolinguistics and deaf communities, 1:5, 2015.
  17. A survey on sign language recognition using smartphones. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, pages 171–176, 2017.
  18. Mediapipe holistic - simultaneous face, hand and pose prediction, on device, Dec 2020. URL https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html.
  19. Sanjay Gulati. Language deprivation syndrome. In Language deprivation and deaf mental health, pages 24–53. Routledge, 2018.
  20. Auditory Deprivation Does Not Impair Executive Function, But Language Deprivation Might: Evidence From a Parent-Report Measure in Deaf Native Signing Children. The Journal of Deaf Studies and Deaf Education, 22(1):9–21, December 2016. ISSN 1081-4159. doi: 10.1093/deafed/enw054. URL https://doi.org/10.1093/deafed/enw054. _eprint: https://academic.oup.com/jdsde/article-pdf/22/1/9/8675438/enw054.pdf.
  21. Deaf children need language, not (just) speech. First Language, 39(4):367–395, 2019.
  22. Wyatte Hall. What You Don’t Know Can Hurt You: The Risk of Language Deprivation by Impairing Sign Language Development in Deaf Children. Maternal and Child Health Journal, 21, May 2017. doi: 10.1007/s10995-017-2287-y.
  23. Language deprivation syndrome: a possible neurodevelopmental disorder with sociocultural origins. Social Psychiatry and Psychiatric Epidemiology, 52, June 2017. doi: 10.1007/s00127-017-1351-7.
  24. Tap to sign: Towards using american sign language for text entry on smartphones. Proc. ACM Hum.-Comput. Interact., 7(MHCI), sep 2023. doi: 10.1145/3604274. URL https://doi.org/10.1145/3604274.
  25. Joseph Hill. Do deaf communities actually want sign language gloves? Nature Electronics, 3(9):512–513, 2020.
  26. Language acquisition for deaf children: Reducing the harms of zero tolerance to the use of alternative approaches. Harm Reduction Journal, 9(1):1–9, 2012.
  27. Support for parents of deaf children: Common questions and informed, evidence-based answers. International journal of pediatric otorhinolaryngology, 118:134–142, 2019.
  28. Variation, lexicalization and grammaticalization in signed languages. Langage et société, 131(1):19–35, 2010.
  29. Coarticulation in asl fingerspelling. In Proceedings of the North East Linguistic Society, volume 42, 2012.
  30. Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation, 2016.
  31. Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals, 1966.
  32. MediaPipe: A framework for building perception pipelines, 2019. URL https://arxiv.org/abs/1906.08172.
  33. Phrase sets for evaluating text entry techniques. In CHI ’03 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’03, page 754–755, New York, NY, USA, 2003. Association for Computing Machinery. ISBN 1581136374. doi: 10.1145/765891.765971. URL https://doi.org/10.1145/765891.765971.
  34. How the hand has shaped sign languages. Scientific Reports, 12(1):11980, 2022.
  35. How many people use asl in the united states? why estimates need updating. Sign Language Studies, 6, 03 2006. doi: 10.1353/sls.2006.0019.
  36. Ellis Monk. Monk skin tone scale, 2019. URL https://skintone.google.
  37. Evaluating the immediate applicability of pose estimation for sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3434–3440, June 2021.
  38. Should all deaf children learn sign language? Pediatrics, 136(1):170–176, 2015.
  39. Rob Neuhaus. Gibberish detector, 2014. URL https://github.com/rrenaud/Gibberish-Detector.
  40. How the alphabet came to be used in a sign language. Sign Language Studies, pages 10–33, 2003.
  41. How do people type on mobile devices? observations from a study with 37,000 volunteers. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 1–12, 2019.
  42. RSVP: Fingerspelled word recognition through rapid serial visual presentation. 2011.
  43. Evolutionary dynamics in the dispersal of sign languages. Royal Society Open Science, 7(1):191100, 2020.
  44. David Quinto-Pozos. Rates of fingerspelling in american sign language. In Poster presented at 10th Theoretical Issues in Sign Language Research conference, West Lafayette, Indiana, volume 30, 2010.
  45. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  46. Performance and user experience of touchscreen and gesture keyboards in a lab setting and in the wild. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 679–688, 2015.
  47. Towards privacy-aware sign language translation at scale, 2024.
  48. Record these hands android app. URL https://github.com/Accessible-Technology-in-Sign/RecordTheseHands.
  49. Open-domain sign language translation learned from online video, 2022.
  50. Popsign ASL v1.0: An isolated american sign language dataset collected via smartphones. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=yEf8NSqTPu.
  51. Reconsidering sentence-level sign language translation, 2024.
  52. Sign language translation from instructional videos, 2023.
  53. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus, 2023.
  54. Byt5: Towards a token-free future with pre-trained byte-to-byte models, 2022.
  55. Detection of major ASL sign types in continuous signing for ASL recognition. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3067–3073, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://www.aclweb.org/anthology/L16-1490.
  56. The word-gesture keyboard: reimagining keyboard interaction. Communications of the ACM, 55(9):91–101, 2012.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up