Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages (2402.14277v1)

Published 22 Feb 2024 in cs.CL and cs.AI

Abstract: Neural Machine Translation (NMT) continues to improve in quality and adoption, yet the inadvertent perpetuation of gender bias remains a significant concern. Despite numerous studies on gender bias in translations into English from weakly gendered-languages, there are no benchmarks for evaluating this phenomenon or for assessing mitigation strategies. To address this gap, we introduce GATE X-E, an extension to the GATE (Rarrick et al., 2023) corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. Each translation is accompanied by feminine, masculine, and neutral variants. The dataset, which contains between 1250 and 1850 instances for each of the four language pairs, features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. Additionally, we present a translation gender rewriting solution built with GPT-4 and use GATE X-E to evaluate it. We open source our contributions to encourage further research on gender debiasing.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Gender in danger? evaluating speech translation technology on the MuST-SHE corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923–6933, Online. Association for Computational Linguistics.
  2. Examining covert gender bias: A case study in turkish and english machine translation models.
  3. MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4287–4299, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  4. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5960–5969, Online. Association for Computational Linguistics.
  5. Tira Nur Fitria. 2021. Gender bias in translation using google translate: Problems and solution. Language Circle: Journal of Language and Literature, 15(2).
  6. Sourojit Ghosh and Aylin Caliskan. 2023. Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages.
  7. Automatic gender identification and reinflection in Arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155–165, Florence, Italy. Association for Computational Linguistics.
  8. Melvin Johnson. 2020. A scalable approach to reducing gender bias in google translate.
  9. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand.
  10. James Kuczmarski. 2018. Reducing gender bias in google translate.
  11. QED: A Framework and Dataset for Explanations in Question Answering. Transactions of the Association for Computational Linguistics, 9:790–806.
  12. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  13. Maria Lopez-Medel. 2021. Gender bias in machine translation: an analysis of google translate in english and spanish. Academia Letters.
  14. OpenAI. 2022. Introducing chatgpt.
  15. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
  16. Good, but not always fair: An evaluation of gender bias for three commercial machine translation systems.
  17. Assessing gender bias in machine translation – a case study with google translate.
  18. Paul Raine. 2020. Talk corpus: A web-based corpus of ted talks for english language teachers and learners.
  19. Gate: A challenge set for gender-ambiguous translation examples. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’23, page 845–854, New York, NY, USA. Association for Computing Machinery.
  20. Gender bias amplification during speed-quality optimization in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 99–109, Online. Association for Computational Linguistics.
  21. A case study of natural gender phenomena in translation a comparison of google translate, bing microsoft translator and deepl for english to italian, french and spanish. In Computational Linguistics CLiC-it, page 359.
  22. Under the morphosyntactic lens: A multifaceted evaluation of gender bias in speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1807–1824, Dublin, Ireland. Association for Computational Linguistics.
  23. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1351–1361, Online. Association for Computational Linguistics.
  24. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  25. They, them, theirs: Rewriting with gender-neutral english.
  26. No language left behind: Scaling human-centered machine translation.
  27. NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8940–8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  28. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2717–2739, Toronto, Canada. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Spencer Rarrick (4 papers)
  2. Ranjita Naik (8 papers)
  3. Sundar Poudel (3 papers)
  4. Vishal Chowdhary (7 papers)