Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus (2310.05294v1)

Published 8 Oct 2023 in cs.CL

Abstract: Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Lauren Ackerman. 2019. Syntactic and cognitive issues in investigating gendered coreference. Glossa: a journal of general linguistics, 4(1).
  2. User-centric gender rewriting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 618–631, Seattle, United States. Association for Computational Linguistics.
  3. Exploiting biased models to de-bias text: A gender-fair rewriting model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4486–4506, Toronto, Canada. Association for Computational Linguistics.
  4. APA. 2020. Publication Manual of the American Psychological Association, 7th edition. American Psychological Association.
  5. E-mimic: Empowering multilingual inclusive communication. In 2021 IEEE International Conference on Big Data (Big Data), pages 4227–4234. IEEE.
  6. Remy Attig and Ártemis López. 2020. Queer Community Input in Gender-Inclusive Translations. Linguistic Society of America [Blog].
  7. Based on billions of words on the internet, people= men. Science Advances, 8(13):eabm2463.
  8. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  9. Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923–6933, Online. Association for Computational Linguistics.
  10. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
  11. How conservative are language models? adapting to the introduction of gender-neutral pronouns. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3624–3630, Seattle, United States. Association for Computational Linguistics.
  12. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics.
  13. On Measuring Gender bias in Translation of Gender-neutral Pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181, Florence, IT. Association for Computational Linguistics.
  14. Improving gender translation accuracy with filtered self-training. arXiv preprint arXiv:2104.07695.
  15. Gloria Comandini. 2021. Salve a tutt\textreve, tutt*, tuttu, tuttx e tutt@: l’uso delle strategie di neutralizzazione di genere nella comunità queer online. : Indagine su un corpus di italiano scritto informale sul web. Testo e Senso, 23:43–64.
  16. Marta R. Costa-jussà and Adrià de Jorge. 2020. Fine-tuning neural machine translation on gender-balanced datasets. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 26–34, Barcelona, Spain (Online). Association for Computational Linguistics.
  17. MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4287–4299, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  18. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1968–1994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  19. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  20. Neutralising linguistic sexism: Promising but cumbersome? Group Processes & Intergroup Relations, 21(5):844–858.
  21. Sourojit Ghosh and Aylin Caliskan. 2023. Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages.
  22. María Isabel Rivas Ginel and Sarah Theroine. 2022. Neutralising for equality: All-inclusive games machine translation. In Proceedings of New Trends in Translation and Technology, pages 125–133. NeTTT.
  23. Bleu meets comet: Combining lexical and neural metrics towards robust machine translation evaluation.
  24. Frida Höglund and Marie Flinkfeldt. 2023. De-gendering parents: Gender inclusion and standardised language in screen-level bureaucracy. International Journal of Social Welfare.
  25. Levi C. R. Hord. 2016. Bucking the linguistic binary: Gender neutral language in English, Swedish, French, and German. Western Papers in Linguistics/Cahiers linguistiques de Western, 3(1):4.
  26. Kris Aric Knisely. 2020. Le français non-binaire: Linguistic forms used by non-binary speakers of French. Foreign Language Annals, 53(4):850–876.
  27. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the tenth Machine Translation Summit, pages 79–86, Phuket, TH. AAMT.
  28. Jennifer Langston. 2020. New AI tools help writers be more clear, concise and inclusive in Office and across the Web. https://blogs.microsoft.com/ai/microsoft-365-ai-tools/. Accessed: 2021-02-25.
  29. Manuel Lardelli and Dagmar Gromann. 2022. Gender-fair (machine) translation. In Proceedings of New Trends in Translation and Technology, pages 166–177. NeTTT.
  30. Manuel Lardelli and Dagmar Gromann. 2023. Gender-fair post-editing: A case study beyond the binary. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 251–260, Tampere, Finland.
  31. Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1221–1232, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  32. What about em? how commercial machine translation fails to handle (neo-)pronouns.
  33. Michela Menegatti and Monica Rubini. 2017. Gender Bias and Sexism in Language. In Oxford Research Encyclopedia of Communication vol.1, pages 451–468. Oxford University Press, New York, USA.
  34. Parmy Olson. 2018. The Algorithm That Helped Google Translate Become Sexist. https://bit.ly/olson_google_sexist. Accessed: 2023-06-20.
  35. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprache.
  36. “i’m fully who i am”: Towards centering transgender and non-binary voices to measure biases in open language generation. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1246–1266.
  37. Dimitrios Papadimoulis. 2018. GENDER-NEUTRAL LANGUAGE in the European Parliament. European Parliament 2018.
  38. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  39. Gender neutralization for an inclusive machine translation: from theoretical foundations to open challenges.
  40. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  41. Gate: A challenge set for gender-ambiguous translation examples.
  42. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  43. Adithya Renduchintala and Adina Williams. 2022. Investigating failures of automatic translationin the case of unambiguous gender. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3454–3469, Dublin, Ireland. Association for Computational Linguistics.
  44. Danielle Saunders. 2022. Domain adaptation for neural machine translation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 9–10, Ghent, Belgium. European Association for Machine Translation.
  45. Danielle Saunders and Katrina Olsen. 2023. Gender, names and other mysteries: Towards the ambiguous for gender-inclusive translation.
  46. Neural machine translation doesn’t translate gender coreference right unless you make it. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 35–43, Barcelona, Spain (Online). Association for Computational Linguistics.
  47. First the worst: Finding better gender translations during beam search. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3814–3823, Dublin, Ireland. Association for Computational Linguistics.
  48. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874.
  49. Under the morphosyntactic lens: A multifaceted evaluation of gender bias in speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1807–1824, Dublin, Ireland. Association for Computational Linguistics.
  50. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
  51. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4275–4293, Online. Association for Computational Linguistics.
  52. Jeanette Silveira. 1980. Generic Masculine Words and Thinking. Women’s Studies International Quarterly, 3(2-3):165–178.
  53. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas,, pages 223–231, Cambridge. Association for Machine Translation in the Americas.
  54. Representation of the Sexes in Language. Social communication, pages 163–187.
  55. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  56. Adhering, steering, and queering: Treatment of gender in natural language generation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, page 1–14, New York, NY, USA. Association for Computing Machinery.
  57. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
  58. They, Them, Theirs: Rewriting with Gender-Neutral English. arXiv preprint arXiv:2102.06788.
  59. Fabio Tamburini. 2020. How "bertology" changed the state-of-the-art also for italian NLP. In Proceedings of the Seventh Italian Conference on Computational Linguistics, CLiC-it 2020, Bologna, Italy, March 1-3, 2021, volume 2769 of CEUR Workshop Proceedings. CEUR-WS.org.
  60. NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8940–8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  61. Eva Vanmassenhove and Johanna Monti. 2021. gENder-IT: An annotated English-Italian parallel challenge set for cross-linguistic natural gender phenomena. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 1–7, Online. Association for Computational Linguistics.
  62. Jonas Wagner and Sina Zarrieß. 2022. Do gender neutral affixes naturally reduce gender bias in static word embeddings? In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 88–97, Potsdam, Germany. KONVENS 2022 Organizers.
  63. Benjamin D Wasserman and Allyson J Weseley. 2009. ?‘ qué? quoi? do languages with grammatical gender promote sexist attitudes? Sex Roles, 61(9-10):634.
  64. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  65. Bertscore: Evaluating text generation with bert.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Andrea Piergentili (6 papers)
  2. Beatrice Savoldi (19 papers)
  3. Dennis Fucci (11 papers)
  4. Matteo Negri (93 papers)
  5. Luisa Bentivogli (38 papers)
Citations (11)
View on Semantic Scholar