Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Privacy-Aware Sign Language Translation at Scale (2402.09611v2)

Published 14 Feb 2024 in cs.CL, cs.AI, cs.CV, and cs.LG

Abstract: A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a two-stage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages self-supervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Bbc-oxford british sign language dataset. arXiv preprint.
  2. Charlotte Baker-Shenk. 1985. The facial behavior of deaf signers: Evidence of a complex language. American Annals of the Deaf, 130(4).
  3. Neural sign language translation. In CVPR.
  4. Sign language transformers: Joint end-to-end sign language recognition and translation. In CVPR.
  5. João Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR.
  6. A simple multi-modality transfer learning baseline for sign language translation. In CVPR.
  7. Two-stream network for sign language recognition and translation. In NeurIPS.
  8. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML.
  9. Randaugment: Practical automated data augmentation with a reduced search space. In NeurIPS.
  10. Mathieu De Coster and Joni Dambre. 2022. Leveraging frozen pretrained written language models for neural sign language translation. Information, 13(5).
  11. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  12. Sign language video retrieval with free-form textual queries. In CVPR.
  13. How2sign: A large-scale multimodal dataset for continuous american sign language. In CVPR.
  14. Cynthia Dwork. 2006. Differential privacy. In ICALP.
  15. Masked autoencoders as spatiotemporal learners. In NeurIPS.
  16. Large-scale privacy protection in google street view. In ICCV.
  17. Tight analysis of privacy and utility tradeoff in approximate differential privacy. In AISTATS.
  18. Masked autoencoders are scalable vision learners. In CVPR.
  19. Masked autoencoders that listen. In NeurIPS.
  20. Amy Isard. 2020. Approaches to the anonymisation of sign language corpora. In LREC Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives.
  21. The kinetics human action video dataset. arXiv preprint.
  22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  23. American sign language video anonymization to support online participation of deaf and hard of hearing users. In ASSETS.
  24. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
  25. Scaling language-image pre-training via masking. In CVPR.
  26. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
  27. Gloss-free end-to-end sign language translation. In ACL.
  28. Multilingual denoising pre-training for neural machine translation. TACL, 8.
  29. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
  30. Ross E Mitchell and Travas A Young. 2022. How Many People Use Sign Language? A National Health Survey-Based Estimate. The Journal of Deaf Studies and Deaf Education, 28(1).
  31. Evaluating the immediate applicability of pose estimation for sign language recognition. In CVPR Workshops.
  32. Findings of the second WMT shared task on sign language translation (WMT-SLT23). In WMT.
  33. Considerations for meaningful sign language machine translation based on glosses. In ACL.
  34. The Syntax of American Sign Language: Functional Categories and Hierarchical Structure. MIT Press.
  35. Bleu: a method for automatic evaluation of machine translation. In ACL.
  36. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS.
  37. Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT.
  38. Learning transferable visual models from natural language supervision. In ICML.
  39. Improving language understanding by generative pre-training. OpenAI Technical Report.
  40. Language models are unsupervised multitask learners. OpenAI Technical Report.
  41. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR (21).
  42. Language modelling with pixels. In ICLR.
  43. Hiera: A hierarchical vision transformer without the bells-and-whistles. In ICML.
  44. Self-supervised video transformers for isolated sign language recognition. arXiv preprint.
  45. Anonysign: Novel human appearance synthesis for sign language video anonymisation. In FG.
  46. BLEURT: Learning robust metrics for text generation. In ACL.
  47. Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint.
  48. Open-domain sign language translation learned from online video. In EMNLP.
  49. William C Stokoe. 1980. Sign language structure. Annual review of anthropology, 9(1).
  50. Sign language translation from instructional videos. In CVPR Workshops.
  51. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS.
  52. Cartoonized anonymization of sign language videos. In IVSMP.
  53. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. In NeurIPS.
  54. Clayton Valli and Ceil Lucas. 2000. Linguistics of American Sign Language: An Introduction. Gallaudet University Press.
  55. Attention is all you need. In NeurIPS.
  56. Videomae V2: scaling video masked autoencoders with dual masking. In CVPR.
  57. Sign language video anonymization. In LREC Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources.
  58. Diffslva: Harnessing diffusion models for sign language video anonymization. arXiv preprint.
  59. A study of face obfuscation in imagenet. In ICML.
  60. Gloss attention for gloss-free sign language translation. In CVPR.
  61. Including signed languages in natural language processing. In ACL.
  62. SLTUNET: A simple unified model for sign language translation. In ICLR.
  63. Gloss-free sign language translation: Improving from visual-language pretraining. In ICCV.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Phillip Rust (12 papers)
  2. Bowen Shi (82 papers)
  3. Skyler Wang (10 papers)
  4. Jean Maillard (17 papers)
  5. Necati Cihan Camgöz (5 papers)
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets