Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas (2403.02563v1)

Published 5 Mar 2024 in cs.CV and cs.CL

Abstract: Growing research in sign language recognition, generation, and translation AI has been accompanied by calls for ethical development of such technologies. While these works are crucial to helping individual researchers do better, there is a notable lack of discussion of systemic biases or analysis of rhetoric that shape the research questions and methods in the field, especially as it remains dominated by hearing non-signing researchers. Therefore, we conduct a systematic review of 101 papers in sign language AI. Our analysis identifies significant biases in the current state of sign language AI research, including an overfocus on addressing perceived communication barriers, a lack of use of representative datasets, use of annotations lacking linguistic foundations, and development of methods that build on flawed models. We take the position that the field lacks meaningful input from Deaf stakeholders, and is instead driven by what decisions are the most convenient or perceived as important to hearing researchers. We end with a call to action: the field must make space for Deaf researchers to lead the conversation in sign language AI.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Bbc-oxford british sign language dataset. arXiv preprint arXiv:2111.03635.
  2. Dhoest Alexander and Jorn Rijckaert. 2022. News ‘with’ or ‘in’ sign language? case study on the comprehensibility of sign language in news broadcasts. Perspectives, 30(4):627–642.
  3. Bridging the gap: Understanding the intersection of deaf and technical perspectives on signing avatars. A. Way, D. Shterionov, C. Rathmann & L. Leeson (eds.), Sign Language Machine Translation.
  4. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604.
  5. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
  6. Ten principles of disability justice. WSQ: Women’s Studies Quarterly, 46(1):227–230.
  7. Carl Börstell. 2023. Ableist language teching over sign language research. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 1–10.
  8. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing (TACCESS), 14(2):1–45.
  9. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299.
  10. Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308.
  11. 1001 small victories: Deaf academics and imposter syndrome. In The Palgrave handbook of imposter syndrome in higher education, pages 481–496. Springer.
  12. Leipzig glossing rules. conventions for interlinear morpheme-by-morpheme glosses. max planck institute for evolutionary anthropology, leizpig.
  13. Maartje De Meulder. 2021. Is “good enough” good enough? ethical and responsible development of sign language technologies. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pages 12–22, Virtual. Association for Machine Translation in the Americas.
  14. Maartje De Meulder and Annalies Kusters. 2021. Twitter thread.
  15. The legal recognition of sign languages: Advocacy and outcomes around the world. Multilingual Matters.
  16. Challenges with sign language datasets for sign language recognition and translation. In Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Odijk J, Piperidis S, editors. LREC 2022, 13th International Conference on Language Resources and Evaluation; 2022 June 20-25; Marseille, France. Paris: European Language Resources; 2022. 10 p. European Language Resources Association.
  17. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
  18. Asl citizen: A community-sourced dataset for advancing isolated sign language recognition. Advances in Neural Information Processing Systems, 36.
  19. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  20. Extensions of the sign language recognition and translation corpus rwth-phoenix-weather. In LREC, pages 1911–1916.
  21. Best practices for sign language technology research. Universal Access in the Information Society, pages 1–9.
  22. The design of eco-feedback technology. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 1999–2008.
  23. Moa Gärdenfors. 2021. The writing process and the written product in bimodal bilingual deaf and hard of hearing children. Languages, 6(2).
  24. Jon Henner and Octavian Robinson. 2023. Crip linguistics goes to school. Languages, 8(1):48.
  25. Joseph C. Hill. 2023. Overrepresentation of whiteness is in sign language as well: A commentary on “undoing competence: Coloniality, homogeneity, and the overrepresentation of whiteness in applied linguistics”. Language Learning, 73(S2):312–316.
  26. Julie Hochgesang. 2022a. Documenting signed language use while considering our spaces as a deaf* linguist.
  27. Julie Hochgesang. 2022b. Managing sign language acquisition video data: A personal journey in the organization and representation of signed data. The Open Handbook of Linguistic Data Management, pages 367–383.
  28. Julie A Hochgesang. 2014. Using design principles to consider representation of the hand in some notation systems. Sign Language Studies, 14(4):488–542.
  29. Julie A Hochgesang. 2019. Tyranny of glossing revisited: reconsidering representational practices of signed languages via best practices of data citation. In TISLR13, the 13th Conference of Theoretical Issues in Sign Language Research, Hamburg, Germany (September 26–28, 2019).
  30. W (h) ither the asl corpus?: Considering trends in signed corpus development. In Advances in Sign Language Corpus Linguistics, pages 287–308. John Benjamins.
  31. Gabrielle Hodge and Onno Crasborn. 2022. Good practices in annotation. In Signed language corpora, pages 46–89. Gallaudet University Press.
  32. Edgcon: Auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical gestures. In Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, pages 3–10.
  33. Lynn Hou. 2017. Negotiating language practices and language ideologies in fieldwork: A reflexive meta-documentation. Innovations in deaf studies: The role of deaf scholars, pages 339–360.
  34. Signbert+: Hand-model-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  35. Self-emphasizing network for continuous sign language recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 854–862.
  36. Signing outside the studio: Benchmarking background robustness for continuous sign language recognition. arXiv preprint arXiv:2211.00448.
  37. The sem-lex benchmark: Modeling asl signs and their phonemes. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, pages 1–10.
  38. Innovations in deaf studies: Critically mapping the field. Innovations in deaf studies: The role of deaf scholars, pages 1–53.
  39. Annelies Kusters and Ceil Lucas. 2022. Emergence and evolutions: Introducing sign language sociolinguistics. Journal of Sociolinguistics, 26(1):84–98.
  40. Beyond languages, beyond modalities: Transforming the study of semiotic repertoires. International Journal of multilingualism, 14(3):219–232.
  41. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459–1469.
  42. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172.
  43. What do we mean by “accessibility research”? a literature survey of accessibility papers in chi and assets from 1994 to 2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–18.
  44. Christopher D Mellinger. 2020. Positionality in public service interpreting research. FITISPos International Journal, 7(1):92–109.
  45. E. Morozov. 2013. To Save Everything, Click Here: Technology, Solutionism, and the Urge to Fix Problems that Don’t Exist. Penguin Books Limited.
  46. Evaluating the immediate applicability of pose estimation for sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3434–3440.
  47. Education and health of children with hearing loss: the necessity of signed languages. Bulletin of the World Health Organization, 97(10):711.
  48. Deaf professionals’ perceptions of trust in relationships with signed/spoken language interpreters. Translation & Interpreting, 15(2):25–42.
  49. The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language. The Journal of Deaf Studies and Deaf Education, 26(2):263–277.
  50. Openhands: Making sign language recognition accessible with pose-based pretrained models across languages. arXiv preprint arXiv:2110.05877.
  51. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205.12870.
  52. SignOn. 2022. Sign language technology: Do’s and don’ts.
  53. Adhd and technology research–investigated by neurodivergent readers. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–21.
  54. Phonology recognition in american sign language. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8452–8456. IEEE.
  55. Read and attend: Temporal localisation in sign language videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16857–16866.
  56. Gloss alignment using word embeddings. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pages 1–5. IEEE.
  57. Angelina Wang and Olga Russakovsky. 2023. Overwriting pretrained bias with finetuning data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3957–3968.
  58. Shuai Wang and Eric Nalisnick. 2023. Active learning for multilingual fingerspelling corpora. arXiv preprint arXiv:2309.12443.
  59. Including signed languages in natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7347–7360.
  60. Natural language-assisted sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14890–14900.
Citations (12)
View on Semantic Scholar

Summary

  • The paper systematically reviews 101 studies and reveals pervasive hearing-centric biases that marginalize authentic Deaf perspectives.
  • It identifies misalignments in dataset representation and gloss annotation that oversimplify the rich linguistic and cultural nuances of Deaf communities.
  • The study advocates for Deaf leadership in research to develop models that authentically capture sign language use and cultural context.

Systemic Biases in Sign Language AI Research: A Critical Appraisal

The paper "Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas" delivers a meticulous examination of the entrenched biases pervading current research agendas in sign language AI. Conducted by a team of researchers identifying as deaf or hard-of-hearing (DHH), this paper offers a valuable introspective into how hearing-dominated research communities have shaped the field's trajectory, often sidelining the perspectives and priorities of the primary stakeholders—Deaf communities.

Systematic Review and Key Findings

The authors conducted a systematic review of 101 papers from 2021 through 2023 to dissect prevailing research narratives. The analysis surfaces several troubling trends:

  1. Motivational Bias: A significant portion (64 out of 101) of the reviewed papers justify their research primarily through the lens of addressing communication barriers between deaf individuals and hearing society. This perspective frequently oversimplifies the rich linguistic and cultural tapestry of Deaf communities by reducing sign languages to mere tools for accessing spoken languages. Such framing inadvertently perpetuates communication as a deficit-model centered on Deaf people's need to "fit" into hearing norms.
  2. Dataset Representation: The team identifies a discord between dataset selection and the actual linguistic diversity found within Deaf communities. Twenty-nine of 43 datasets are either mixed or do not specify representation, and commonly-used benchmarks are frequently rooted in interpreter-derived data, which may not reflect authentic usage patterns within Deaf communities. This misalignment could lead to models that prioritize convenience over authenticity, prompting Deaf users to adapt to AI system limitations rather than the reverse.
  3. Annotation Practices: Glossing systems, employed as seen in 51 papers, are often used without clarity or consistency. Many studies treat glosses as direct translations instead of linguistic representations intertwined with numerous semiotic nuances. This approach potentially bottlenecks the rich expressiveness of sign languages and limits the potential utility of semantic machine learning applications.
  4. Modeling Decisions: A disconcerting dependency on pre-trained models trained on non-sign language datasets emerges. Such models, particularly those utilizing pose-based augmentation, may not capture the intricacies of sign language, leading to systemic perpetuation of biases entrenched in the source data. This practice casts doubt on the reliability and applicability of derived models for authentic sign language recognition and translation tasks.

Implications and Contributions

This paper culminates in a firm call for enhanced Deaf leadership in shaping the research agenda for sign language AI. Unlike previous calls merely advocating for Deaf collaboration, the authors urge that genuine leadership roles—especially in posing research questions and making methodological decisions—are essential for aligning research outcomes with the needs and realities of Deaf communities.

The authors propose a broader adoption of positionality statements to foreground the biases that researchers themselves bring into their work, a practice already established in the humanities but underutilized in technical fields like AI. They argue that fostering spaces for direct Deaf leadership and critical engagement will better ensure that technologies are developed that respect and augment Deaf linguistic identities, rather than subjugate them to the demands of convenience-driven AI models.

Conclusion and Prospect for Future Work

This work provides significant reflections for experienced researchers in AI and related fields. It argues convincingly that genuine inclusion begins when Deaf researchers guide the understanding and representation of their language and communities. Future work could focus on developing models free from inherited hearing-centric biases and increasing the participation of Deaf stakeholders in academic and development endeavors. Sign language AI can indeed flourish and be beneficial, provided it is grounded in the lived experiences and insights of those it purports to serve. This paper is an integral step toward such an equitable research paradigm.