Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition (2304.05934v2)

Published 12 Apr 2023 in cs.CV and cs.CL

Abstract: Sign languages are used as a primary language by approximately 70 million D/deaf people world-wide. However, most communication technologies operate in spoken and written languages, creating inequities in access. To help tackle this problem, we release ASL Citizen, the first crowdsourced Isolated Sign Language Recognition (ISLR) dataset, collected with consent and containing 83,399 videos for 2,731 distinct signs filmed by 52 signers in a variety of environments. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their webcam to retrieve matching signs from a dictionary. We show that training supervised machine learning classifiers with our dataset advances the state-of-the-art on metrics relevant for dictionary retrieval, achieving 63% accuracy and a recall-at-10 of 91%, evaluated entirely on videos of users who are not present in the training or validation sets. An accessible PDF of this article is available at the following link: https://aashakadesai.github.io/research/ASLCitizen_arxiv_updated.pdf

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Machine learning methods for sign language recognition: A critical review and analysis. Intelligent Systems with Applications, 12:200056, 2021.
  2. Effect of automatic sign recognition performance on the usability of video-based search interfaces for sign language dictionaries. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pages 56–67, 2019.
  3. The american sign language lexicon video dataset. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8. IEEE, 2008.
  4. Sign pose-based transformer for word-level sign language recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 182–191, 2022.
  5. Carl Börstell. Ableist language teching over sign language research. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 1–10, Tórshavn, the Faroe Islands, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.resourceful-1.1.
  6. A user-powered american sign language dictionary. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pages 1837–1848, 2015.
  7. Sign language recognition, generation, and translation: An interdisciplinary perspective. In The 21st international ACM SIGACCESS conference on computers and accessibility, pages 16–31, 2019.
  8. Exploring collection of sign language datasets: Privacy, participation, and model performance. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pages 1–14, 2020.
  9. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing (TACCESS), 14(2):1–45, 2021.
  10. Exploring collection of sign language videos through crowdsourcing. CSCW 2022, 2022.
  11. Native and foreign vocabulary in american sign language: A lexicon with multiple origins. In Foreign vocabulary in sign languages, pages 87–119. Psychology Press, 2001.
  12. Diane Brentari et al. A prosodic model of sign language phonology. Mit Press, 1998.
  13. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  14. A performance evaluation of hmm and dtw for gesture recognition. In Iberoamerican Congress on Pattern Recognition, pages 236–243. Springer, 2012.
  15. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  16. Asl-lex: A lexical database of american sign language. Behavior research methods, 49(2):784–801, 2017.
  17. A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1):131–153, 2019.
  18. Sign language recognition. In Visual analysis of humans, pages 539–562. Springer, 2011.
  19. CVPR 2017 Workshop on Visual Understanding Across Modalities. Charades challenge: Recognize and locate activities taking place in a video, 2017. URL http://vuchallenge.org/charades.html.
  20. Sign language recognition with transformer networks. In 12th international conference on language resources and evaluation, pages 6018–6024. European Language Resources Association (ELRA), 2020.
  21. Maartje De Meulder. Is “good enough” good enough? ethical and responsible development of sign language technologies. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pages 12–22, 2021.
  22. Automatic sign language recognition: A survey. In 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pages 1–7. IEEE, 2017.
  23. Michael Erard. Why sign-language gloves don’t help deaf people, 2017. URL https://www.theatlantic.com/technology/archive/2017/11/why-sign-language-gloves-dont-help-deaf-people/545441/.
  24. Building bsl signbank: The lemma dilemma revisited. International Journal of Lexicography, 28(2):169–206, 2015.
  25. Susan D Fischer and Harry Van der Hulst. Sign language structures. Deaf studies language and education, pages 319–331, 2003.
  26. Modality combination techniques for continuous sign language recognition. In Iberian conference on pattern recognition and image analysis, pages 89–99. Springer, 2013.
  27. Hands and Voices. Deaf culture and community, 2022. https://www.handsandvoices.org/comcon/articles/deafculture.htm.
  28. Evaluation of an english word look-up tool for web-browsing with sign language video for deaf readers. In International Conference on Universal Access in Human-Computer Interaction, pages 205–215. Springer, 2018.
  29. Research ethics in sign language communities. Sign Language Studies, 9(2):104–131, 2009.
  30. Effect of sign-recognition performance on the usability of sign-language dictionary search. ACM Transactions on Accessible Computing (TACCESS), 14(4):1–33, 2021.
  31. Design and evaluation of hybrid search for american sign language to english dictionaries: Making the most of imperfect sign recognition. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA, 2022. Association for Computing Machinery. doi: 10.1145/3491102.3501986. URL https://doi.org/10.1145/3491102.3501986.
  32. Building the asl signbank. lemmatization principles for asl. 2018.
  33. Sign language in the interface: access for deaf signers. Universal Access Handbook. NJ: Erlbaum, 38:14, 2009.
  34. Technological solutions for sign language recognition: a scoping review of research trends, challenges, and opportunities. IEEE Access, 2022.
  35. Hamid Reza Vaezi Joze and Oscar Koller. Ms-asl: A large-scale data set and benchmark for understanding american sign language. arXiv preprint arXiv:1812.01053, 2018.
  36. Oscar Koller. Quantitative survey of the state of the art in sign language recognition. arXiv preprint arXiv:2008.09918, 2020.
  37. Innovations in deaf studies: The role of deaf scholars. Oxford University Press, 2017.
  38. Harlan Lane. When the Mind Hears: A History of the Deaf. Knopf Doubleday Publishing Group, London, 1989.
  39. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459–1469, 2020.
  40. Enrollments in languages other than english in united states institutions of higher education, summer 2016 and fall 2016. In Modern language association. ERIC, 2019.
  41. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019.
  42. The hidden treasure of Black ASL: Its history and structure. Gallaudet University Press Washington, DC, 2011.
  43. Detecting and identifying sign languages through visual features. In 2016 IEEE International Symposium on Multimedia (ISM), pages 287–290. IEEE, 2016.
  44. A boosted classifier tree for hand shape detection. In Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings., pages 889–894. IEEE, 2004.
  45. Deep convolutional neural networks for sign language recognition. In 2018 Conference on Signal Processing And Communication Engineering Systems (SPACES), pages 194–197. IEEE, 2018.
  46. Sign language recognition: A deep survey. Expert Systems with Applications, 164:113794, 2021.
  47. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
  48. The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language. The Journal of Deaf Studies and Deaf Education, 26(2):263–277, 2021.
  49. Openhands: Making sign language recognition accessible with pose-based pretrained models across languages. arXiv preprint arXiv:2110.05877, 2021.
  50. Signing Savvy. Signing savvy an ASL sign language video dictionary, 2022. URL https://www.signingsavvy.com/.
  51. MF Tolba and AS Elons. Recent developments in sign language recognition systems. In 2013 8th International Conference on Computer Engineering & Systems (ICCES), pages xxxvi–xlii. IEEE, 2013.
  52. Adapting hidden markov models for asl recognition by using three-dimensional computer vision methods. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, volume 1, pages 156–161. IEEE, 1997.
  53. Sign language recognition systems: A decade systematic literature review. Archives of Computational Methods in Engineering, 28(3):785–813, 2021.
  54. WFD. Legal recognition of national sign languages., 2022a. https://wfdeaf.org/news/the-legal-recognition-of-national-sign-languages/.
  55. WFD. World federation of the deaf, 2022b. URL http://wfdeaf.org/our-work.
  56. Purdue rvl-slll american sign language database. 2006.
  57. Ronnie B Wilbur. Phonological and prosodic layering of nonmanuals in american sign language. In The signs of language revisited, pages 196–220. Psychology Press, 2013.
  58. Automatic gloss dictionary for sign language learners. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 83–92, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-demo.8. URL https://aclanthology.org/2022.acl-demo.8.
  59. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
  60. Including signed languages in natural language processing. arXiv preprint arXiv:2105.05222, 2021.
  61. Combination of tangent distance and an image distortion model for appearance-based sign language recognition. In Joint Pattern Recognition Symposium, pages 401–408. Springer, 2005.
  62. Inge Zwitserlood. Classifiers. In Sign language: An international handbook. De Gruyter, 2012.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Aashaka Desai (3 papers)
  2. Lauren Berger (2 papers)
  3. Fyodor O. Minakov (2 papers)
  4. Vanessa Milan (2 papers)
  5. Chinmay Singh (3 papers)
  6. Kriston Pumphrey (1 paper)
  7. Richard E. Ladner (1 paper)
  8. Alex X. Lu (9 papers)
  9. Naomi Caselli (5 papers)
  10. Danielle Bragg (8 papers)
  11. Hal Daumé III (76 papers)
Citations (10)
View on Semantic Scholar