Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns (2405.00134v1)

Published 30 Apr 2024 in cs.CL and cs.AI

Abstract: Gender-neutral pronouns are increasingly being introduced across Western languages. Recent evaluations have however demonstrated that English NLP systems are unable to correctly process gender-neutral pronouns, with the risk of erasing and misgendering non-binary individuals. This paper examines a Dutch coreference resolution system's performance on gender-neutral pronouns, specifically hen and die. In Dutch, these pronouns were only introduced in 2016, compared to the longstanding existence of singular they in English. We additionally compare two debiasing techniques for coreference resolution systems in non-binary contexts: Counterfactual Data Augmentation (CDA) and delexicalisation. Moreover, because pronoun performance can be hard to interpret from a general evaluation metric like LEA, we introduce an innovative evaluation metric, the pronoun score, which directly represents the portion of correctly processed pronouns. Our results reveal diminished performance on gender-neutral pronouns compared to gendered counterparts. Nevertheless, although delexicalisation fails to yield improvements, CDA substantially reduces the performance gap between gendered and gender-neutral pronouns. We further show that CDA remains effective in low-resource settings, in which a limited set of debiasing documents is used. This efficacy extends to previously unseen neopronouns, which are currently infrequently used but may gain popularity in the future, underscoring the viability of effective debiasing with minimal resources and low computational costs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Y Gavriel Ansara and Peter Hegarty. 2014. Methodologies of misgendering: Recommendations for reducing cisgenderism in psychological research. Feminism & Psychology 24, 2 (2014), 259–270.
  2. Connor Baumler and Rachel Rudinger. 2022. Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 3426–3432. https://doi.org/10.18653/v1/2022.naacl-main.250
  3. Sander Becker. 2020. Is het Nederlands klaar voor het genderneutrale ‘Hen loopt’? Trouw (8 Oct. 2020). https://www.trouw.nl/cultuur-media/is-het-nederlands-klaar-voor-het-genderneutrale-hen-loopt~b5fb2e1b/
  4. Henrik Björklund and Hannah Devinney. 2023. Computer, enhence: POS-tagging improvements for nonbinary pronoun use in Swedish. In Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion, Bharathi R. Chakravarthi, B. Bharathi, Joephine Griffith, Kalika Bali, and Paul Buitelaar (Eds.). INCOMA Ltd., Shoumen, Bulgaria, Varna, Bulgaria, 54–61. https://aclanthology.org/2023.ltedi-1.8
  5. Coreference Resolution through a seq2seq Transition-Based System. Transactions of the Association for Computational Linguistics 11 (2023), 212–226. https://doi.org/10.1162/tacl_a_00543
  6. How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 3624–3630. https://doi.org/10.18653/v1/2022.naacl-main.265
  7. Yang Trista Cao and Hal Daumé III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4568–4595. https://doi.org/10.18653/v1/2020.acl-main.418
  8. Yang Trista Cao and Hal Daumé III. 2021. Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*. Computational Linguistics 47, 3 (Nov. 2021), 615–661. https://doi.org/10.1162/coli_a_00413
  9. Guiding Principles for Participatory Design-inspired Natural Language Processing. In Proceedings of the 1st Workshop on NLP for Positive Impact. Association for Computational Linguistics, Online, 27–35. https://doi.org/10.18653/v1/2021.nlp4posimpact-1.4
  10. On Measuring Gender Bias in Translation of Gender-neutral Pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Florence, Italy, 173–181. https://doi.org/10.18653/v1/W19-3824
  11. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 8440–8451. https://doi.org/10.18653/v1/2020.acl-main.747
  12. RobBERT: a Dutch RoBERTa-based Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 3255–3265. https://doi.org/10.18653/v1/2020.findings-emnlp.292
  13. Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1968–1994. https://doi.org/10.18653/v1/2021.emnlp-main.150
  14. Theories of “Gender” in NLP Bias Research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 2083–2102. https://doi.org/10.1145/3531146.3534627
  15. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423
  16. Vladimir Dobrovolskii. 2021. Word-Level Coreference Resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7670–7675. https://doi.org/10.18653/v1/2021.emnlp-main.605
  17. EditieNL. 2021. Lij of vij: is er een nieuw non-binair persoonlijk voornaamwoord nodig? RTLnieuws (25 May 2021). https://www.rtlnieuws.nl/editienl/artikel/5232738/nieuw-persoonlijk-voornaamwoord-non-binaire-personen-hen-hun-die
  18. Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Trans. Inf. Syst. 14, 3 (jul 1996), 330–347. https://doi.org/10.1145/230538.230561
  19. Marinel Gerritsen. 2002. Language and gender in Netherlands Dutch: Towards a more gender-fair usage. Gender Across Languages 2 (2002).
  20. Sourojit Ghosh and Aylin Caliskan. 2023. ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five Other Low-Resource Languages. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (, Montréal, QC, Canada,) (AIES ’23). Association for Computing Machinery, New York, NY, USA, 901–912. https://doi.org/10.1145/3600211.3604672
  21. Introducing a gender-neutral pronoun in a natural gender language: the influence of time on attitudes and behavior. Frontiers in Psychology 6 (2015). https://doi.org/10.3389/fpsyg.2015.00893
  22. A Coreference Corpus and Resolution System for Dutch. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). European Language Resources Association (ELRA), Marrakech, Morocco. http://www.lrec-conf.org/proceedings/lrec2008/pdf/49_paper.pdf
  23. Het Neutrale Taal collectief. [n. d.]. Lijst van populaire voornaamwoorden. https://nl.pronouns.page/voornaamwoorden. Accessed: 2022-10-14.
  24. MISGENDERED: Limits of Large Language Models in Understanding Pronouns. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5352–5367. https://doi.org/10.18653/v1/2023.acl-long.293
  25. Robin Mattias Hurkens. 2021. Genderneutraal: geen ‘hij’, ‘zij’, ‘hen’, ‘dij’, maar ‘ij’. De Volkskrant (26 May 2021). https://www.volkskrant.nl/columns-opinie/genderneutraal-geen-hij-zij-hen-dij-maar-ij~b3c890b9/
  26. The Report of the 2015 U.S. Transgender Survey. National Center for Transgender Equality (2016). https://transequality.org/sites/default/files/docs/usts/USTS-Full-Report-Dec17.pdf
  27. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics 8 (2020), 64–77. https://doi.org/10.1162/tacl_a_00300
  28. Measuring Bias in Contextualized Word Representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Florence, Italy, 166–172. https://doi.org/10.18653/v1/W19-3823
  29. Detecting intersectionality in NER models: A data-driven approach. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, Stefania Degaetano-Ortlieb, Anna Kazantseva, Nils Reiter, and Stan Szpakowicz (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 116–127. https://doi.org/10.18653/v1/2023.latechclfl-1.13
  30. Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender. In Proceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 1221–1232. https://aclanthology.org/2022.coling-1.105
  31. What about “em”? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 377–392. https://doi.org/10.18653/v1/2023.acl-long.23
  32. End-to-end Neural Coreference Resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 188–197. https://doi.org/10.18653/v1/D17-1018
  33. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, 687–692. https://doi.org/10.18653/v1/N18-2108
  34. Autoregressive Structured Prediction with Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 993–1005. https://doi.org/10.18653/v1/2022.findings-emnlp.70
  35. Measuring Gender Bias in West Slavic Language Models. In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), Jakub Piskorski, Michał Marcińczuk, Preslav Nakov, Maciej Ogrodniczuk, Senja Pollak, Pavel Přibáň, Piotr Rybak, Josef Steinberger, and Roman Yangarber (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 146–154. https://doi.org/10.18653/v1/2023.bsnlp-1.17
  36. Nafise Sadat Moosavi and Michael Strube. 2016. Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 632–642. https://doi.org/10.18653/v1/P16-1060
  37. SoNaR User Documentation. 1, 4 (2013). https://www.ivdnt.org/images/stories/producten/documentatie/sonar_documentatie.pdf
  38. “I’m fully who I am”: Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (, Chicago, IL, USA,) (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 1246–1266. https://doi.org/10.1145/3593013.3594078
  39. Corbèn Poot and Andreas van Cranenburgh. 2020. A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference. Association for Computational Linguistics, Barcelona, Spain (online), 79–90. https://aclanthology.org/2020.crac-1.9
  40. Micah Rajunov and A Scott Duane. 2019. Nonbinary: Memoirs of gender and identity. Columbia University Press.
  41. Balancing SoNaR: IPR versus Processing Issues in a 500-Million-Word Written Dutch Reference Corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, Malta. http://www.lrec-conf.org/proceedings/lrec2010/pdf/549_Paper.pdf
  42. Gender Bias in Coreference Resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, 8–14. https://doi.org/10.18653/v1/N18-2002
  43. Measuring Gender Bias in Natural Language Processing: Incorporating Gender-Neutral Linguistic Forms for Non-Binary Gender Identities in Abusive Speech Detection. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing. 1121–1131.
  44. Trans, intersekse en non-binaire mensen aan het werk in Nederland: Een nationaal rapport. Transgender Netwerk Nederland (TNN) (2021). https://www.transgendernetwerk.nl/wp-content/uploads/Inclusion4All-National-report-Netherlands_NL.pdf
  45. Transgender Netwerk Nederland. 2016. ZO MAAK JE NA TOILETTEN OOK TAAL GENDERNEUTRAAL. https://www.transgendernetwerk.nl/non-binair-voornaamwoord-uitslag/. Transgender Netwerk Nederland Nieuws (June 2016). https://www.transgendernetwerk.nl/non-binair-voornaamwoord-uitslag/
  46. Andreas van Cranenburgh. 2019. A Dutch coreference resolution system with an evaluation on literary fiction. Computational Linguistics in the Netherlands Journal 9 (2019), 27–54.
  47. A Hybrid Rule-Based and Neural Coreference Resolution System with an Evaluation on Dutch Literature. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference. Association for Computational Linguistics, Punta Cana, Dominican Republic, 47–56. https://doi.org/10.18653/v1/2021.crac-1.5
  48. What social attitudes about gender does BERT encode? Leveraging insights from psycholinguistics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 6790–6809. https://doi.org/10.18653/v1/2023.acl-long.375
  49. Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns. Transactions of the Association for Computational Linguistics 6 (2018), 605–617. https://doi.org/10.1162/tacl_a_00240
  50. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6
  51. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 629–634. https://doi.org/10.18653/v1/N19-1064
  52. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, 15–20. https://doi.org/10.18653/v1/N18-2003
  53. Lal Zimman. 2018. Transgender Language, Transgender Moment: Toward a Trans Linguistics. In The Oxford Handbook of Language and Sexuality. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190212926.013.45 arXiv:https://academic.oup.com/book/0/chapter/358161844/chapter-ag-pdf/50001811/book_42645_section_358161844.ag.pdf
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Goya van Boven (3 papers)
  2. Yupei Du (11 papers)
  3. Dong Nguyen (28 papers)