Papers
Topics
Authors
Recent
Search
2000 character limit reached

3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands

Published 3 Sep 2024 in cs.CV, cs.AI, and cs.CL | (2409.01901v1)

Abstract: In this work, we present an efficient approach for capturing sign language in 3D, introduce the 3D-LEX v1.0 dataset, and detail a method for semi-automatic annotation of phonetic properties. Our procedure integrates three motion capture techniques encompassing high-resolution 3D poses, 3D handshapes, and depth-aware facial features, and attains an average sampling rate of one sign every 10 seconds. This includes the time for presenting a sign example, performing and recording the sign, and archiving the capture. The 3D-LEX dataset includes 1,000 signs from American Sign Language and an additional 1,000 signs from the Sign Language of the Netherlands. We showcase the dataset utility by presenting a simple method for generating handshape annotations directly from 3D-LEX. We produce handshape labels for 1,000 signs from American Sign Language and evaluate the labels in a sign recognition task. The labels enhance gloss recognition accuracy by 5% over using no handshape annotations, and by 1% over expert annotations. Our motion capture data supports in-depth analysis of sign features and facilitates the generation of 2D projections from any viewpoint. The 3D-LEX collection has been aligned with existing sign language benchmarks and linguistic resources, to support studies in 3D-aware sign language processing.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Deep multi view spatio temporal spectral feature embedding on skeletal sign language videos for recognition. International Journal of Advanced Computer Science and Applications, 13(4).
  2. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’19, page 16–31, New York, NY, USA. Association for Computing Machinery.
  3. Exploring collection of sign language datasets: Privacy, participation, and model performance. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’20, New York, NY, USA. Association for Computing Machinery.
  4. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis; Machine Intelligence, 43:172–186.
  5. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3413–3423.
  6. Improving sign recognition with phonology. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2732–2737, Dubrovnik, Croatia. Association for Computational Linguistics.
  7. Oscar Koller. 2020. Quantitative survey of the state of the art in sign language recognition. ArXiv, abs/2008.09918.
  8. Signcollect - a ’touchless’ pipeline for constructing large-scale sign language repositories. In LREC-COLING 2024 - 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources, Turin, Italy.
  9. Sign language production: A review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3451–3461.
  10. OpenHands: Making sign language recognition accessible with pose-based pretrained models across languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2114–2133, Dublin, Ireland. Association for Computational Linguistics.
  11. William C. Stokoe. Sign language structure: An outline of the visual communication systems of the american deaf. The Journal of Deaf Studies and Deaf Education, 10(1):3–37.
  12. Viewing angle matters in british sign language processing. Scientific Reports, 14(1).
  13. Including signed languages in natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7347–7360, Online. Association for Computational Linguistics.
  14. The American Sign Language Lexicon Video Dataset. 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.
  15. Collecting and Analysing a Motion-Capture Corpus of French Sign Language. Workshop on the Representation and Processing of Sign Languages.
  16. Helen Cooper and Eng-Jon Ong and Nicolas Pugeault and Richard Bowden. 2012. Sign Language Recognition using Sub-Units. Journal of Machine Learning Research.
  17. NGT dataset in Global Signbank. Radboud University, Centre for Language Studies, ISLRN 976-021-358-388-6.
  18. Amanda Cardoso Duarte and Shruti Palaskar and Deepti Ghadiyaram and Kenneth DeHaan and Florian Metze and Jordi Torres and Xavier Giró-i-Nieto. 2020. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. ISLRN 583-408-694-292-6.
  19. Multi-View Fusion for Sign Language Recognition through Knowledge Transfer Learning. Association for Computing Machinery, VRCAI ’22.
  20. Gibet, Sylvie. 2018. Building French Sign Language Motion Capture Corpora for Signing Avatars.
  21. Captured Motion Data Processing for Real Time Synthesis of Sign Language. Springer Berlin Heidelberg.
  22. Video-based sign language recognition without temporal segmentation. AAAI Press, AAAI’18/IAAI’18/EAAI’18.
  23. Sign Language Motion Capture Dataset for Data-driven Synthesis. European Language Resources Association (ELRA).
  24. Hamid Reza Vaezi Joze and Oscar Koller. 2019. MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. BMVA Press.
  25. The Sem-Lex Benchmark: Modeling ASL Signs and their Phonemes. ACM, ASSETS ’23.
  26. An extension of the NGT dataset in Global Signbank. SignBank.
  27. The Sign Language Dataset Compendium: Creating an Overview of Digital Linguistic Resources. European Language Resources Association.
  28. Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. The IEEE Winter Conference on Applications of Computer Vision.
  29. Collecting a Motion-Capture Corpus of American Sign Language for Data-Driven Generation Research. Association for Computational Linguistics.
  30. Multiview meta-metric learning for sign language recognition using triplet loss embeddings. Pattern Analysis and Applications.
  31. Mariusz Oszust and Marian Wysocki. 2013. Polish sign language words recognition with Kinect. 2013 6th International Conference on Human System Interactions (HSI).
  32. Razieh Rastgoo and Kourosh Kiani and Sergio Escalera. 2020. Hand sign language recognition using multi-view hand skeleton. Expert Syst. Appl.
  33. Adam C. Schembri and Jordan B Fenlon and Ramas Rentelis and Sally Reynolds and Kearsy Cormier. 2013. Building the British Sign Language Corpus. University of Hawaii Press.
  34. The ASL-LEX 2.0 Project: A Database of Lexical and Phonological Properties for 2,723 Signs in American Sign Language. The Journal of Deaf Studies and Deaf Education.
  35. SIGNUM Database: Video Corpus for Signer-Independent Continuous Sign Language Recognition. European Language Resources Association (ELRA).
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up