Papers
Topics
Authors
Recent
Search
2000 character limit reached

Select and Reorder: A Novel Approach for Neural Sign Language Production

Published 17 Apr 2024 in cs.CL | (2404.11532v1)

Abstract: Sign languages, often categorised as low-resource languages, face significant challenges in achieving accurate translation due to the scarcity of parallel annotated datasets. This paper introduces Select and Reorder (S&R), a novel approach that addresses data scarcity by breaking down the translation process into two distinct steps: Gloss Selection (GS) and Gloss Reordering (GR). Our method leverages large spoken LLMs and the substantial lexical overlap between source spoken languages and target sign languages to establish an initial alignment. Both steps make use of Non-AutoRegressive (NAR) decoding for reduced computation and faster inference speeds. Through this disentanglement of tasks, we achieve state-of-the-art BLEU and Rouge scores on the Meine DGS Annotated (mDGS) dataset, demonstrating a substantial BLUE-1 improvement of 37.88% in Text to Gloss (T2G) Translation. This innovative approach paves the way for more effective translation models for sign languages, even in resource-constrained settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project. In IEE Seminar on speech and language processing for disabled and elderly people (Ref. No. 2000/025), pages 6–1. IET.
  2. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–480.
  3. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  4. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  5. First language acquisition differs from second language acquisition in prelingually deaf signers: Evidence from sensitivity to grammaticality judgement in british sign language. Cognition, 124(1):50–65.
  6. Tessa, a system to aid communication with deaf people. In Proceedings of the fifth international ACM conference on Assistive technologies, pages 205–212.
  7. Vincent J Della Pietra. 1994. The mathematics of statistical machine translation: Parameter estimation. Using Large Corpora, page 223.
  8. Jinhua Du and Andy Way. 2017. Pre-reordering for neural machine translation: Helpful or harmful? Prague Bulletin of Mathematical Linguistics, 108:171–181.
  9. The dicta-sign wiki: Enabling web communication for the deaf. In International Conference on Computers for Handicapped Persons, pages 205–212. Springer.
  10. Hosni Mostafa El-dali. 2011. Towards an understanding of the distinctive nature of translation studies. Journal of King Saud University-Languages and Translation, 23(1):29–45.
  11. Dmitriy Genzel. 2010. Automatically learning source-side reordering rules for large scale machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 376–384.
  12. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics.
  13. Thomas Hanke. 2004. Hamnosys-representing sign language data in language resources and language processing contexts. In LREC, volume 4, pages 1–6.
  14. Otedama: Fast rule-based pre-ordering for machine translation. Prague Bull. Math. Linguistics, 106:159–168.
  15. Towards fast and high-quality sign language production. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3172–3181.
  16. Non-autoregressive sign language production with gaussian space. In Proceedings of the British Machine Vision Conference (BMVC).
  17. Machine translation between spoken languages and signed languages represented in signwriting. arXiv preprint arXiv:2210.05404.
  18. Mihoko Kato. 2008. A study of notation and sign writing systems for the deaf. Intercultural Communication Studies, 17(4):97–114.
  19. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980.
  20. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872.
  21. Statistical phrase-based translation. Technical report, University of Southern California Marina Del Rey Information Sciences Inst.
  22. Joey nmt: A minimalist nmt toolkit for novices. arXiv:1907.12484.
  23. Gradient-based learning applied to document recognition. Proceedings of the IEEE.
  24. Barbara LeMaster and Leila Monaghan. 2005. Variation in Sign Languages, chapter 7. John Wiley & Sons, Ltd.
  25. Transcribing natural languages for the deaf via neural editing programs. arXiv preprint arXiv:2112.09600.
  26. Transcribing natural languages for the deaf via neural editing programs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11991–11999.
  27. Pre-training multilingual neural machine translation by leveraging alignment information. arXiv preprint arXiv:2010.03142.
  28. Ceil Lucas and Clayton Valli. 2014. American deaf community. The Sociolinguistics of the Deaf Community, page 11.
  29. Chloë R Marshall and Angela Hobsbaum. 2015. Sign-supported english: is it effective at teaching vocabulary to young children with english as an additional language? International journal of language & communication disorders, 50(5):616–628.
  30. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems.
  31. Data augmentation for sign language gloss translation. arXiv:2105.07476.
  32. Tetsuji Nakagawa. 2015. Efficient top-down BTG parsing for machine translation preordering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 208–218, Beijing, China. Association for Computational Linguistics.
  33. National Geographic Society. 2017. Sign language. https://education.nationalgeographic.org/resource/sign-language, Retrieved September 7, 2022.
  34. Inducing a discriminative parser to optimize machine translation reordering. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 843–853.
  35. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40th Annual meeting of the Association for Computational Linguistics, pages 295–302.
  36. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc.
  37. Sign language production: a review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3451–3461.
  38. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728.
  39. Neural systems for sign language production: mechanisms supporting lexical selection, phonological encoding, and articulation. Human brain mapping, 23(3):156–167.
  40. Adversarial training for multi-channel sign language production. arXiv preprint arXiv:2008.12405.
  41. Everybody sign now: Translating spoken language to photo realistic sign language video. arXiv preprint arXiv:2011.09846.
  42. Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision.
  43. Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  44. Jerome Daniel Schein and Marcus T. Delk. 1974. The deaf population of the united states.
  45. Machine translation using deep learning: An overview. In 2017 international conference on computer, communications and electronics (comptelix), pages 162–167. IEEE.
  46. The influence of language deprivation in early childhood on l2 processing: An erp comparison of deaf native signers and deaf signers with a delayed language acquisition. BMC neuroscience, 13:1–14.
  47. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research.
  48. William C Stokoe. 1980. Sign language structure. Annual Review of Anthropology.
  49. Sign language production using neural machine translation and generative adversarial networks. In Proceedings of the 29th British Machine Vision Conference.
  50. There and back again: 3d sign language generation from text using back-translation. Unpublished.
  51. Shinichi Tamura and Shingo Kawasaki. 1988. Recognition of sign language motion images. Pattern Recognition.
  52. Don Tuggener. 2016. Incremental coreference resolution for German. Ph.D. thesis, University of Zurich.
  53. Desmond Eustin Van Wyk. 2008. Virtual human modelling and animation for real-time sign language visualisation. Ph.D. thesis, University of the Western Cape.
  54. Attention is all you need. Advances in Neural Information Processing Systems.
  55. Hmm-based word alignment in statistical translation. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
  56. Changing the representation: Examining language representation for neural sign language production. arXiv preprint arXiv:2210.06312.
  57. WHO. 2021. Deafness and hearing loss.
  58. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational linguistics, 23(3):377–403.
  59. Exploiting pre-ordering for neural machine translation. In Proceedings of the eleventh international conference on language resources and evaluation (lrec 2018).
  60. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823.
  61. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201.
  62. Synthetic signing for the deaf: Esign. In Proceedings of the conference and workshop on assistive technologies for vision and hearing impairment (CVHI).

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up